SimGrid-3.14.159/000755 001750 001750 00000000000 13031031310 015061 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/NEWS000644 001750 001750 00000037177 13031031306 015604 0ustar00mquinsonmquinson000000 000000 _ _____ _ _ _ __ _____ _ __ ___(_) ___ _ __ |___ / / | || | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ | | || |_ \ V / __/ | \__ \ | (_) | | | | ___) || |__ _| \_/ \___|_| |___/_|\___/|_| |_| |____(_)_| |_| Dec 24 2016 The Christmas Pi Release. * Documentation reorganized and improved * S4U interface further rising, toward SimGrid 4 - Routing code rewritten for readability - Virtual Machines almost turned into a plugin - MSG, SimDag, MPI interfaces mostly unchanged * The model-checker now works on FreeBSD too. _ _____ _ _____ __ _____ _ __ ___(_) ___ _ __ |___ / / |___ / \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ | | |_ \ \ V / __/ | \__ \ | (_) | | | | ___) || |___) | \_/ \___|_| |___/_|\___/|_| |_| |____(_)_|____/ Apr 27 2016 The Half Release, a.k.a. the Zealous Easter Trim. * Half of the lines of code are gone. - v3.12: 286k lines; v3.13: 142k lines (+ comments) - Experimental untested unused "features" removed - We rewrote several parts in C++ instead of C * Introducing v4 of the XML platform format - Many long-due cleanups (details in the Changelog) * MSG examples fully reorganized (in C and Java) * The S4U interface is rising, toward SimGrid 4 - All host manipulations now done in S4U - SimDag was mostly rewritten on top of S4U - MSG & SimDag interfaces mostly unchanged _ _____ _ ____ __ _____ _ __ ___(_) ___ _ __ |___ / / |___ \ \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ | | __) | \ V / __/ | \__ \ | (_) | | | | ___) || |/ __/ \_/ \___|_| |___/_|\___/|_| |_| |____(_)_|_____| Oct 12 2015 The Facelift Release. Major changes: * Many interface glitches addressed, everywhere. - Require g++-4.7, Java 7 and boost 1.48 - Builds on Linux, OS X, Windows and FreeBSD - See ChangeLog for renamed functions and options. * Energy plugin: major cleanups/rewrites. http://simgrid.org/tutorials/simgrid-energy-101.pdf * Model-Checker progresses toward production use: - More robust (now runs in a separate process). - More efficient (with snapshots' memory compaction). - More usable (execution paths replayed out of MC) http://simgrid.org/tutorials/simgrid-mc-101.pdf * SMPI improvements: - Better standard coverage - SMP-aware collectives - InfiniBand model - Early support for RMA - Replay: Dynamic selector - MVAPICH2 selector http://simgrid.org/tutorials/simgrid-smpi-101.pdf Upcoming v3.13 will introduce new interfaces, toward SimGrid 4. _ _____ _ _ __ _____ _ __ ___(_) ___ _ __ |___ / / / | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ | | | \ V / __/ | \__ \ | (_) | | | | ___) || | | \_/ \___|_| |___/_|\___/|_| |_| |____(_)_|_| May 31 2014 The Class Release. Major changes: * Surf is now in C++ (and documented!) * Virtual Machine model for Cloud Simulation * Surf callbacks: plug your code directly in Surf! * Simcalls are script-generated to clean the mess _ _____ _ ___ __ _____ _ __ ___(_) ___ _ __ |___ / / |/ _ \ \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ | | | | | \ V / __/ | \__ \ | (_) | | | | ___) || | |_| | \_/ \___|_| |___/_|\___/|_| |_| |____(_)_|\___/ Nov 17 2013 The Clean Diaper Release, a.k.a. SimGrid is leak-free. Major changes: * Preliminary DVFS support to track the energy consumption * Java is back in the main package (and is very stable). * The storage sub-modules is now believed to be usable. * SMPI is now very stable (we pass most MPICH tests). * Lots of memory leaks were corrected in this release. * Verification and model checking further improved: liveness+SMPI works; reduction through state equality detection Plus the usual load of bug fixes and small improvements _ _____ ___ __ _____ _ __ ___(_) ___ _ __ |___ // _ \ \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ (_) | \ V / __/ | \__ \ | (_) | | | | ___) \__, | \_/ \___|_| |___/_|\___/|_| |_| |____(_)/_/ Feb 5 2013 The "Grasgory" release. Major changes: * Gras was completely removed from this version. * Documentation reorganization to ease browsing it. * New default value for the TCP_gamma parameter: 4MiB _ _____ ___ _ __ _____ _ __ ___(_) ___ _ __ |___ / ( _ ) / | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ / _ \ | | \ V / __/ | \__ \ | (_) | | | | ___) | (_) || | \_/ \___|_| |___/_|\___/|_| |_| |____(_)___(_)_| Oct 27 2012 The "we are told that some people want to also *install* the simgrid framework" release. * Add missing manpage to the archive. _ _____ ___ __ _____ _ __ ___(_) ___ _ __ |___ / ( _ ) \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ / _ \ \ V / __/ | \__ \ | (_) | | | | ___) | (_) | \_/ \___|_| |___/_|\___/|_| |_| |____(_)___/ Oct 25 2012 The Psssshiiiit release: SimGrid jumps into the Cloud. Major changes: * Experimental interface to manipulate VMs, EC2-style. * Fixes around process restart and stochastic workloads * platf: New C interface to create fixed or random platforms * SimDag: Many fixes and improvements of typed tasks * SMPI now covers more of the MPI interface. More datatypes, more functions, more robust. * Model-checking: mmalloc is more robust to user errors. _ _____ _____ _ __ _____ _ __ ___(_) ___ _ __ |___ /|___ / | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ / /| | \ V / __/ | \__ \ | (_) | | | | ___) | / /_| | \_/ \___|_| |___/_|\___/|_| |_| |____(_)_/(_)_| June 7 2012 The "cleaning the cleanup" release. Major changes: * Portability fixups for Mac OSX and Windows * Some other bug fixing and various polishing. _ ____ _____ __ _____ _ __ ___(_) ___ _ __ |___ /|___ | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ / / \ V / __/ | \__ \ | (_) | | | | ___) | / / \_/ \___|_| |___/_|\___/|_| |_| |____(_)_/ May 15 2012 The "spring cleanups (before the next Big Project kicks in)" release. Major changes: * Major cleanups all around (doc, user options, MSG, Lua, internals) Small backward compatibility glitches may have been introduced * Parallel execution of user code should be stable and efficient now * SMPI is now considered stable * Add temporals to Model-Checking (highly experimental right now) _ _____ __ ____ __ _____ _ __ ___(_) ___ _ __ |___ / / /_ |___ \ \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \| '_ \ __) | \ V / __/ | \__ \ | (_) | | | | ___) | (_) | / __/ \_/ \___|_| |___/_|\___/|_| |_| |____(_)___(_)_____| Oct 5 2011 The "Not coding new stuff allows to polish old things" release. * Portability to Mac and Windows improved. * Possible misconfigurations (contexts, libPCRE) made impossible by removing the option or providing sane default value. * Experimental support to NS3 simulator as a back-end. _ _____ __ _ __ _____ _ __ ___(_) ___ _ __ |___ / / /_ / | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \| '_ \ | | \ V / __/ | \__ \ | (_) | | | | ___) | (_) || | \_/ \___|_| |___/_|\___/|_| |_| |____(_)___(_)_| Jun 27 2011 The "Oops, we broke Macs too" release. Bug fix to an issue preventing SimGrid from working on Mac OSX. _ _____ __ __ _____ _ __ ___(_) ___ _ __ |___ / / /_ \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \| '_ \ \ V / __/ | \__ \ | (_) | | | | ___) | (_) | \_/ \___|_| |___/_|\___/|_| |_| |____(_)___/ Jun 21 2011 The "OMG! They Killed Kenny!" version. Major changes: * Java and Ruby bindings were removed from the main archive - They are now distributed separately * GRAS is not considered as stable anymore, but experimental. Sorry. * First support for parallel simulation: - Can run the user code of any simulation in parallel. - Basic support for multi-cores hosts in platform files * Improved P2P support: - Peer tag in platform files for DSL connected nodes - Vivaldi routing scheme for lightweigted yet accurate models * Improved SMPI: Faster Fortran, automatic privatization of C globals * Tracing: trace contains the full platform hierarchy exactly as declared using the ASes of the platform file Plus numerous other goodies (check the ChangeLog for details) _ _____ ____ __ _____ _ __ ___(_) ___ _ __ |___ / | ___| \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ |___ \ \ V / __/ | \__ \ | (_) | | | | ___) | ___) | \_/ \___|_| |___/_|\___/|_| |_| |____(_)____/ 01 Dec 2010 The "Winter in Frejus" release. Also known as "ANR/ADT funding helps" Major changes are: * New feature: Model check any simgrid simulation * SMPI is now very usable. * Visualization: - now covers the whole framework - major usability improvements * SURF: - scalable platform management through hierarchical description - new efficient way to bypass the XML parser * MSG: at last asynchronous functions are available * SIMDAG: many usability improvements (dotloader, ...) * GRAS: finally catch up with latest internal evolutions * Build chain: - Windows port: should be usable now but still considered experimental - Autotools have now been completely removed _ _____ _ _ _ __ _____ _ __ ___(_) ___ _ __ |___ /| || | / | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \| || |_ | | \ V / __/ | \__ \ | (_) | | | | ___) |__ _|| | \_/ \___|_| |___/_|\___/|_| |_| |____(_) |_|(_)_| 04 May 2010 The "Polishing easter eggs is probably a good idea" release. This is a bug fixes release only. _ _____ _ _ __ _____ _ __ ___(_) ___ _ __ |___ /| || | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \| || |_ \ V / __/ | \__ \ | (_) | | | | ___) |__ _| \_/ \___|_| |___/_|\___/|_| |_| |____(_) |_| 28 Apr 2010 The "Easter in Cargese" release. Also known as (major changes): * the "se habla Java, Ruby 話せます, fala-se Lua (and deaf-friendly)" ~> bindings were greatly improved ~> new tracing infrastructure for better visualization introduced * the "Welcome to configury modernity" release. ~> we switched from autotools to cmake, and improved our cdash _ _____ _____ _ _ __ _____ _ __ ___(_) ___ _ __ |___ / |___ /| || | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ |_ \| || |_ \ V / __/ | \__ \ | (_) | | | | ___) | ___) |__ _| \_/ \___|_| |___/_|\___/|_| |_| |____(_)____(_) |_| 24 Dec 2009 The "Desktop Grid needs love too" release (also called "Xmas release"). Most important changes: * Big speedup through lazy evaluation of the linear models * Supernovae mode: Compile everything in one unit to improve inlining * Simix network module for internal cleanups * Load DAX of applications into SimDag * Lot of small cleanups and other bug fixes _ _____ _____ _____ __ _____ _ __ ___(_) ___ _ __ |___ / |___ / |___ / \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ |_ \ |_ \ \ V / __/ | \__ \ | (_) | | | | ___) | ___) | ___) | \_/ \___|_| |___/_|\___/|_| |_| |____(_)____(_)____/ 20 Aug 2009 The "Need for Speed" release. Big speedup through some function inlining. _ _____ _____ ____ __ _____ _ __ ___(_) ___ _ __ |___ / |___ / |___ \ \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ |_ \ __) | \ V / __/ | \__ \ | (_) | | | | ___) | ___) | / __/ \_/ \___|_| |___/_|\___/|_| |_| |____(_)____(_)_____| 19 Aug 2009 The "Simplicity does not preceed complexity, but follows it" release. Most important changes: * surf and simix modules reworked for simplification. It lays the ground for future extensions and improvements of SimGrid * SMPI was further improved, although not ready for production use yet. On the way, we gained a bit more than 5% on the classical master/slave example. More to come on this in future releases. _ _____ _____ _ __ _____ _ __ ___(_) ___ _ __ |___ / |___ / / | \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ |_ \ | | \ V / __/ | \__ \ | (_) | | | | ___) | ___) || | \_/ \___|_| |___/_|\___/|_| |_| |____(_)____(_)_| 27 Jun 2009 This dot release is mainly a maintenance one. Most important changes: * We fixed a large amount of bugs all around * We sanitized the way configuration is handled internally. Try passing --cfg-help to binaries compiled against this version. * SMPI is in better shape: lot of bugs fixing & usability improvements It may be worth trying it (even if all bugs are not gone yet) This version may have a bit more of memleaks than 3.3. This will be fixed in a latter release. ____ _ ____ _ _ / ___|(_)_ __ ___ / ___|_ __(_) __| | \___ \| | '_ ` _ \| | _| '__| |/ _` | ___) | | | | | | | |_| | | | | (_| | |____/|_|_| |_| |_|\____|_| |_|\__,_| _ _____ _____ __ _____ _ __ ___(_) ___ _ __ |___ / |___ / \ \ / / _ \ '__/ __| |/ _ \| '_ \ |_ \ |_ \ \ V / __/ | \__ \ | (_) | | | | ___) | ___) | \_/ \___|_| |___/_|\___/|_| |_| |____(_)____/ _ _ _ _ _ _ _ | |___ __ _ __| |_ ___ __| | | || | ' \| / -_) _` (_-< ' \/ -_) _` | \_,_|_||_|_\___\__,_/__/_||_\___\__,_| Mar 16 2007 homepage: http://simgrid.gforge.inria.fr/ download: http://gforge.inria.fr/frs/?group_id=12 At least, after 2 years of hard work, we managed to release the 3.3 version of the SimGrid framework. There is so many changes that the changelog only lists the most important ones, leaving alone the small improvements, bug fixing and new gadgets. In short: * Java bindings * New simulation models, and improvement of the GTNetS wrapper * Large memory savings (mainly in parser) * Faster (twice faster is not uncommon, and from 20 hours to 2 minutes on very large scenarios) * Much better scalability (tested up to 250,000 processes) * Complete regression testing framework so that you can trust the tool * Lot of neat new modules in the XBT toolbox This version was thoroughly tested on linux 32bits and 64bits (debian), as well as Mac OSX (leopard). Unfortunately, our windows-guy left, and we cannot release the windows version at the same time than the other archs. Any help would be really welcomed here. Some of the 96 included test suites are known to fail, but everything should work anyway (don't panic): * The amok module does not work in real deployment ATM, but I don't see this as release critical since I'm not aware of anyone needing this right now * Some tests about the ability of GRAS to receive messages from exotic platforms fail because I've lost access to these platforms (such as AIX) * the example/gras/pmm sometimes fails (about 1/10 of cases). I'm quite puzzled, but I suspect a bug in the test, not in the lib. * the tesh auto-tests "fail" on Mac OSX. This is because "rm -rf" is sometimes too verbose (when the OS creates hidden files, I suspect), but tesh definitely work as the rest on this arch. We hope to manage to do more timely releases in the future, even if that may turn out difficult since big stuff is coming (I don't say much here for the suspense ;) We hope you'll enjoy this new version, and please report any feedback on the list. Martin Quinson (for Da SimGrid Team) SimGrid-3.14.159/CMakeLists.txt000644 001750 001750 00000130364 13031031273 017640 0ustar00mquinsonmquinson000000 000000 cmake_minimum_required(VERSION 2.8.8) message(STATUS "Cmake version ${CMAKE_MAJOR_VERSION}.${CMAKE_MINOR_VERSION}.${CMAKE_PATCH_VERSION}") set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} ${CMAKE_HOME_DIRECTORY}/tools/cmake/Modules) project(SimGrid C CXX) #-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-# # Check for the compiler # #-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-# ### Need to set rc ccompiler before enable language if(WIN32) SET(CMAKE_RC_COMPILER "windres") endif() ## ## Check the C/C++ standard that we need ## See also tools/cmake/GCCFlags.cmake that sets our paranoid warning flags INCLUDE(CheckCCompilerFlag) CHECK_C_COMPILER_FLAG(-fstack-cleaner HAVE_C_STACK_CLEANER) ## Request full debugging flags set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -g3") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g3") set(CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} -g") if (CMAKE_COMPILER_IS_GNUCC) if (CMAKE_CXX_COMPILER_VERSION VERSION_LESS "4.7") message(FATAL_ERROR "SimGrid needs at least g++ version 4.7 to compile but you have ${CMAKE_CXX_COMPILER_VERSION}." "You need a sufficient support of c++11 to compile SimGrid.") endif() endif() ## We need a decent support of the c++11 standard include(CheckCXXCompilerFlag) CHECK_CXX_COMPILER_FLAG("-std=gnu++11" COMPILER_SUPPORTS_CXX11) if(COMPILER_SUPPORTS_CXX11) set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=gnu++11") else() message(FATAL_ERROR "The compiler ${CMAKE_CXX_COMPILER} (v${CMAKE_CXX_COMPILER_VERSION}) has no C++11 support. " "Please use a decent C++ compiler.") endif() ### And we need C11 standard, too include(CheckCCompilerFlag) CHECK_C_COMPILER_FLAG("-std=gnu11" COMPILER_SUPPORTS_C11) if(COMPILER_SUPPORTS_C11) set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -std=gnu11") else() message(FATAL_ERROR "The compiler ${CMAKE_C_COMPILER} (v${CMAKE_C_COMPILER_VERSION}) has no C11 support. " "Please use a decent C compiler " "(note that c++11 support of ${CMAKE_CXX_COMPILER} seems ok).") endif() if(APPLE AND (CMAKE_C_COMPILER_VERSION VERSION_LESS "4.6")) ### gcc 4.[1-5] cannot compile ucontext on OSX message(STATUS "Ucontext can't be used with this version of gcc (must be greater than 4.5)") set(HAVE_UCONTEXT_H 0) endif() ### Setup Options include(${CMAKE_HOME_DIRECTORY}/tools/cmake/Option.cmake) ### SMPI vs. Fortran if ((NOT DEFINED enable_smpi) OR enable_smpi) # First unset the compiler in case we're re-running cmake over a previous # configuration where it was saved as smpiff unset(CMAKE_Fortran_COMPILER) SET(SMPI_FORTRAN 0) if(enable_fortran) enable_language(Fortran OPTIONAL) endif() if(CMAKE_Fortran_COMPILER) # Fortran compiler detected: save it, then replace by smpiff set(SAVED_Fortran_COMPILER "${CMAKE_Fortran_COMPILER}" CACHE FILEPATH "The real Fortran compiler") set(CMAKE_Fortran_COMPILER smpiff) set(SMPI_FORTRAN 1) endif(CMAKE_Fortran_COMPILER) endif() #-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-# # Build the version number # #-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-# set(SIMGRID_VERSION_MAJOR "3") set(SIMGRID_VERSION_MINOR "14") set(SIMGRID_VERSION_PATCH "159") # set(SIMGRID_VERSION_EXTRA "-alpha") # Extra words to add to version string (e.g. -rc1) set(SIMGRID_VERSION_DATE "2016") # Year for copyright information if(${SIMGRID_VERSION_PATCH} EQUAL "0") set(release_version "${SIMGRID_VERSION_MAJOR}.${SIMGRID_VERSION_MINOR}") else() set(release_version "${SIMGRID_VERSION_MAJOR}.${SIMGRID_VERSION_MINOR}.${SIMGRID_VERSION_PATCH}") endif() set(SIMGRID_VERSION_STRING "SimGrid version ${release_version}${SIMGRID_VERSION_EXTRA}") set(libsimgrid_version "${release_version}") set(libsimgrid-java_version "${release_version}") ### SET THE LIBRARY EXTENSION if(APPLE) set(LIB_EXE "dylib") elseif(WIN32) set(LIB_EXE "a") set(BIN_EXE ".exe") else() set(LIB_EXE "so") endif() execute_process(COMMAND ${CMAKE_LINKER} -version OUTPUT_VARIABLE LINKER_VERSION) string(REGEX MATCH "[0-9].[0-9]*" LINKER_VERSION "${LINKER_VERSION}") ### Set the library providing dlopen if("${CMAKE_SYSTEM}" MATCHES "Linux") find_library(DL_LIBRARY dl) endif("${CMAKE_SYSTEM}" MATCHES "Linux") ### Find programs and paths FIND_PROGRAM(GCOV_PATH gcov) include(FindPerl) if(NOT PERL_FOUND) message(FATAL_ERROR "Please install Perl to compile SimGrid.") endif() # tesh.py needs python 3 (or the module python-subprocess32 on python2.8+) set(PythonInterp_FIND_VERSION 3) set(PythonInterp_FIND_VERSION_COUNT 1) set(PythonInterp_FIND_VERSION_MAJOR 3) include(FindPythonInterp) if(NOT PYTHONINTERP_FOUND) message(FATAL_ERROR "Please install Python (version 3 or higher).") endif() if (APPLE) set(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/lib") set(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE) endif() SET(LIBRARY_OUTPUT_PATH ${CMAKE_BINARY_DIR}/lib) ### Compute the include paths set(INCLUDES ${CMAKE_HOME_DIRECTORY} ${CMAKE_HOME_DIRECTORY}/include ${CMAKE_HOME_DIRECTORY}/src/include ${CMAKE_BINARY_DIR} ${CMAKE_BINARY_DIR}/include ) if(NOT CMAKE_CROSSCOMPILING AND EXISTS /usr/include/) set(INCLUDES ${INCLUDES} /usr/include/) endif() if(WIN32) set(CMAKE_INCLUDE_WIN "${CMAKE_C_COMPILER}") set(CMAKE_LIB_WIN "${CMAKE_C_COMPILER}") string(REGEX REPLACE "/bin/gcc.*" "/include" CMAKE_INCLUDE_WIN "${CMAKE_INCLUDE_WIN}") string(REGEX REPLACE "/bin/gcc.*" "/lib" CMAKE_LIB_WIN "${CMAKE_LIB_WIN}") set(INCLUDES ${INCLUDES} ${CMAKE_INCLUDE_WIN}) unset(CMAKE_INCLUDE_WIN) endif() include_directories(${INCLUDES}) # library dependency cannot start with a space (CMP0004), so initialize it with something that is never desactivated. set(SIMGRID_DEP "-lm") ### Determine the assembly flavor that we need today set(HAVE_RAW_CONTEXTS 0) include(CMakeDetermineSystem) IF(CMAKE_SYSTEM_PROCESSOR MATCHES ".86|AMD64|amd64") IF(CMAKE_SIZEOF_VOID_P EQUAL 4) # 32 bits message(STATUS "System processor: i686 (${CMAKE_SYSTEM_PROCESSOR}, 32 bits)") set(SIMGRID_PROCESSOR_i686 1) set(SIMGRID_PROCESSOR_x86_64 0) ELSE() message(STATUS "System processor: x86_64 (${CMAKE_SYSTEM_PROCESSOR}, 64 bits)") set(SIMGRID_PROCESSOR_i686 0) set(SIMGRID_PROCESSOR_x86_64 1) ENDIF() if (WIN32) message(STATUS "Disable fast raw contexts on Windows.") else() set(HAVE_RAW_CONTEXTS 1) endif() ELSE() set(SIMGRID_PROCESSOR_i686 0) set(SIMGRID_PROCESSOR_x86_64 0) ENDIF() include(CheckFunctionExists) include(CheckTypeSize) include(CheckIncludeFile) include(CheckIncludeFiles) include(CheckLibraryExists) include(CheckSymbolExists) set(HAVE_GRAPHVIZ 0) include(FindGraphviz) set(HAVE_LUA 0) if(enable_lua) include(FindLuaSimgrid) endif() set(HAVE_NS3 0) if(enable_ns3) include(FindNS3) if (NOT HAVE_NS3) message(FATAL_ERROR "Cannot find NS3. Please install it (apt-get install ns3 libns3-dev) or disable that cmake option") endif() endif() if(WIN32) set(Boost_USE_STATIC_LIBS 1) endif() set(HAVE_PAPI 0) if(enable_smpi_papi) include(FindPAPI) if (NOT HAVE_PAPI) message(FATAL_ERROR "Cannot find PAPI. Please install it (apt-get install papi-tools libpapi-dev) or disable PAPI bindings.") endif() endif() # Not finding this is perfectly OK find_package(Boost COMPONENTS unit_test_framework) if (Boost_UNIT_TEST_FRAMEWORK_FOUND) message("-- (enabling the Boost-based unit tests)") else() message("-- (disabling the Boost-based unit tests -- please install libboost-test-dev)") endif() find_package(Boost 1.48) if(Boost_FOUND) include_directories(${Boost_INCLUDE_DIRS}) else() if(APPLE) message(FATAL_ERROR "Boost libraries not found. Try to install them with 'sudo fink install boost1.53.nopython'") else() message(FATAL_ERROR "Boost libraries not found. Install libboost-dev (>= 1.48.0).") endif() endif() #Boost context 1.61 will break compatibility with our implementation. #Warn users about that find_package(Boost 1.61 COMPONENTS context) if(Boost_FOUND AND Boost_CONTEXT_FOUND) message(" WARNING : our implementation of Boost context factory is not compatible with Boost >=1.61 yet.") set(Boost_FOUND 1) set(HAVE_BOOST_CONTEXTS 0) else() # Try again to see if we have libboost-context find_package(Boost 1.42 COMPONENTS context) set(Boost_FOUND 1) # We don't care of whether this component is missing if(Boost_FOUND AND Boost_CONTEXT_FOUND) # We should use feature detection for this instead: if (Boost_VERSION LESS 105600) message("Found Boost.Context API v1") set(HAVE_BOOST_CONTEXTS 1) else() message("Found Boost.Context API v2") set(HAVE_BOOST_CONTEXTS 2) endif() else() message (" boost : found.") message (" boost-context: missing. Install libboost-context-dev for this optional feature.") set(HAVE_BOOST_CONTEXTS 0) endif() endif() # Checks for header libraries functions. CHECK_LIBRARY_EXISTS(rt clock_gettime "" HAVE_POSIX_GETTIME) CHECK_LIBRARY_EXISTS(pthread pthread_create "" HAVE_PTHREAD) if(NOT APPLE) # OS X El Capitan deprecates this function CHECK_LIBRARY_EXISTS(pthread sem_init "" HAVE_SEM_INIT_LIB) endif() CHECK_LIBRARY_EXISTS(pthread sem_open "" HAVE_SEM_OPEN_LIB) set(HAVE_PTHREAD_SETAFFINITY 0) CHECK_LIBRARY_EXISTS(pthread pthread_setaffinity_np "" HAVE_PTHREAD_SETAFFINITY) if(CMAKE_SYSTEM_NAME MATCHES "Darwin") set(CMAKE_REQUIRED_DEFINITIONS "-D_XOPEN_SOURCE=700 -D_DARWIN_C_SOURCE") elseif(MINGW) # Use the GNU version of unusual modifiers like PRIx64 add_definitions(-D__USE_MINGW_ANSI_STDIO=1) set(CMAKE_REQUIRED_DEFINITIONS "-D__USE_MINGW_ANSI_STDIO=1") else() set(CMAKE_REQUIRED_DEFINITIONS "-D_GNU_SOURCE") endif() CHECK_INCLUDE_FILE("valgrind/valgrind.h" HAVE_VALGRIND_H) CHECK_INCLUDE_FILE("unistd.h" HAVE_UNISTD_H) CHECK_INCLUDE_FILE("execinfo.h" HAVE_EXECINFO_H) CHECK_INCLUDE_FILE("signal.h" HAVE_SIGNAL_H) CHECK_INCLUDE_FILE("sys/param.h" HAVE_SYS_PARAM_H) CHECK_INCLUDE_FILE("sys/sysctl.h" HAVE_SYS_SYSCTL_H) CHECK_INCLUDE_FILE("ucontext.h" HAVE_UCONTEXT_H) CHECK_INCLUDE_FILE("linux/futex.h" HAVE_FUTEX_H) CHECK_FUNCTION_EXISTS(backtrace HAVE_BACKTRACE) CHECK_FUNCTION_EXISTS(dlfunc HAVE_DLFUNC) CHECK_FUNCTION_EXISTS(gettimeofday HAVE_GETTIMEOFDAY) CHECK_FUNCTION_EXISTS(nanosleep HAVE_NANOSLEEP) CHECK_FUNCTION_EXISTS(getdtablesize HAVE_GETDTABLESIZE) CHECK_FUNCTION_EXISTS(sysconf HAVE_SYSCONF) CHECK_FUNCTION_EXISTS(popen HAVE_POPEN) CHECK_FUNCTION_EXISTS(process_vm_readv HAVE_PROCESS_VM_READV) CHECK_FUNCTION_EXISTS(mmap HAVE_MMAP) CHECK_FUNCTION_EXISTS(mremap HAVE_MREMAP) CHECK_SYMBOL_EXISTS(vasprintf stdio.h HAVE_VASPRINTF) if(MINGW) # The detection of vasprintf fails on MinGW, assumingly because it's # defined as an inline function in stdio.h instead of a regular # function. So force the result to be 1 despite of the test. set(HAVE_VASPRINTF 1) endif() #Check if __thread is defined execute_process( COMMAND "${CMAKE_C_COMPILER} ${CMAKE_HOME_DIRECTORY}/tools/cmake/test_prog/prog_thread_storage.c -o testprog" WORKING_DIRECTORY ${CMAKE_BINARY_DIR} RESULT_VARIABLE HAVE_thread_storage_run) file(REMOVE testprog) if(HAVE_thread_storage_run) set(HAVE_THREAD_LOCAL_STORAGE 1) else() set(HAVE_THREAD_LOCAL_STORAGE 0) endif() if(enable_model-checking AND NOT "${CMAKE_SYSTEM}" MATCHES "Linux|FreeBSD") message(WARNING "Support for model-checking has not been enabled on ${CMAKE_SYSTEM}: disabling it") set(enable_model-checking FALSE) endif() if(HAVE_MMAP AND HAVE_THREAD_LOCAL_STORAGE) SET(HAVE_MMALLOC 1) else() SET(HAVE_MMALLOC 0) if(enable_model-checking) message(STATUS "Warning: support for model-checking has been disabled because you are missing either mmap or __thread.") endif() SET(enable_model-checking 0) endif() if(enable_jedule) set(HAVE_JEDULE 1) else() set(HAVE_JEDULE 0) endif() if(enable_mallocators) SET(HAVE_MALLOCATOR 1) else() SET(HAVE_MALLOCATOR 0) endif() include(FindLibunwind) if(HAVE_LIBUNWIND) if(NOT APPLE) SET(SIMGRID_DEP "${SIMGRID_DEP} -lunwind") else() # Apple forbids to link directly against its libunwind implementation # So let's comply and link against the System framework SET(SIMGRID_DEP "${SIMGRID_DEP} -lSystem") endif() if("${CMAKE_SYSTEM}" MATCHES "Linux|FreeBSD") set(SIMGRID_DEP "${SIMGRID_DEP} -lunwind-ptrace") # This supposes that the host machine is either an AMD or a X86. # This is deeply wrong, and should be fixed by manually loading -lunwind-PLAT (FIXME) if(SIMGRID_PROCESSOR_x86_64) SET(SIMGRID_DEP "${SIMGRID_DEP} -lunwind-x86_64") else() SET(SIMGRID_DEP "${SIMGRID_DEP} -lunwind-x86") endif() endif() else() if(enable_model-checking) message(FATAL_ERROR "Please either install the libunwind7-dev package (or equivalent) or turn off the model-checking option of SimGrid.") endif() endif() if(enable_model-checking) find_package(Libdw REQUIRED) find_package(Libevent REQUIRED) include_directories(${LIBDW_INCLUDE_DIRS} ${LIBEVENT_INCLUDE_DIRS}) set(SIMGRID_DEP "${SIMGRID_DEP} ${LIBEVENT_LIBRARIES} ${LIBDW_LIBRARIES}") set(HAVE_MC 1) if("${CMAKE_SYSTEM}" MATCHES "FreeBSD" AND enable_java) message(WARNING "FreeBSD + Model-Checking + Java = too much for now. Disabling java") set(enable_java FALSE) endif() else() SET(HAVE_MC 0) set(HAVE_MMALLOC 0) endif() if (enable_model-checking AND enable_ns3) message(FATAL_ERROR "Cannot activate both model-checking and NS3 bindings: NS3 pull too much dependencies for the MC to work") endif() if(enable_smpi) SET(HAVE_SMPI 1) if("${CMAKE_SYSTEM}" MATCHES "Linux|FreeBSD") SET(USE_LIBUTIL 0) SET(HAVE_PRIVATIZATION 1) else() message (STATUS "Warning: no support for SMPI automatic privatization on this platform") SET(HAVE_PRIVATIZATION 0) endif() else() SET(HAVE_SMPI 0) endif() #-------------------------------------------------------------------------------------------------- ### Check for GNU dynamic linker CHECK_INCLUDE_FILE("dlfcn.h" HAVE_DLFCN_H) if (HAVE_DLFCN_H) execute_process(COMMAND ${CMAKE_C_COMPILER} ${CMAKE_HOME_DIRECTORY}/tools/cmake/test_prog/prog_gnu_dynlinker.c ${DL_LIBRARY} -o test_gnu_ld WORKING_DIRECTORY ${CMAKE_BINARY_DIR} OUTPUT_VARIABLE HAVE_GNU_LD_compil ) if(HAVE_GNU_LD_compil) set(HAVE_GNU_LD 0) message(STATUS "Warning: test program toward GNU ld failed to compile:") message(STATUS "${HAVE_GNU_LD_comp_output}") else() execute_process(COMMAND ./test_gnu_ld WORKING_DIRECTORY ${CMAKE_BINARY_DIR} RESULT_VARIABLE HAVE_GNU_LD_run OUTPUT_VARIABLE var_exec ) if(NOT HAVE_GNU_LD_run) set(HAVE_GNU_LD 1) message(STATUS "We are using GNU dynamic linker") else() set(HAVE_GNU_LD 0) message(STATUS "Warning: error while checking for GNU ld:") message(STATUS "Test output: '${var_exec}'") message(STATUS "Exit status: ${HAVE_GNU_LD_run}") endif() file(REMOVE test_gnu_ld) endif() endif() #-------------------------------------------------------------------------------------------------- ### Initialize of CONTEXT THREADS set(HAVE_THREAD_CONTEXTS 0) if(HAVE_PTHREAD) ### Test that we have a way to create semaphores if(HAVE_SEM_OPEN_LIB) execute_process(COMMAND ${CMAKE_C_COMPILER} ${CMAKE_HOME_DIRECTORY}/tools/cmake/test_prog/prog_sem_open.c -lpthread -o sem_open WORKING_DIRECTORY ${CMAKE_BINARY_DIR} OUTPUT_VARIABLE HAVE_SEM_OPEN_compil ) # Test sem_open by compiling: if(HAVE_SEM_OPEN_compil) set(HAVE_SEM_OPEN 0) message(STATUS "Warning: sem_open not compilable") message(STATUS "HAVE_SEM_OPEN_comp_output: ${HAVE_SEM_OPEN_comp_output}") else() set(HAVE_SEM_OPEN 1) message(STATUS "sem_open is compilable") endif() # If we're not crosscompiling, we check by executing the program: if (HAVE_SEM_OPEN AND NOT CMAKE_CROSSCOMPILING) execute_process(COMMAND ./sem_open WORKING_DIRECTORY ${CMAKE_BINARY_DIR} RESULT_VARIABLE HAVE_SEM_OPEN_run OUTPUT_VARIABLE var_compil ) if (NOT HAVE_SEM_OPEN_run) set(HAVE_SEM_OPEN 1) message(STATUS "sem_open is executable") else() set(HAVE_SEM_OPEN 0) if(EXISTS "${CMAKE_BINARY_DIR}/sem_open") message(STATUS "Bin ${CMAKE_BINARY_DIR}/sem_open exists!") else() message(STATUS "Bin ${CMAKE_BINARY_DIR}/sem_open not exists!") endif() message(STATUS "Warning: sem_open not executable") message(STATUS "Compilation output: '${var_compil}'") message(STATUS "Exit result of sem_open: ${HAVE_SEM_OPEN_run}") endif() endif() file(REMOVE sem_open) else() set(HAVE_SEM_OPEN 0) endif() if(HAVE_SEM_INIT_LIB) execute_process(COMMAND ${CMAKE_C_COMPILER} ${CMAKE_HOME_DIRECTORY}/tools/cmake/test_prog/prog_sem_init.c -lpthread -o sem_init WORKING_DIRECTORY ${CMAKE_BINARY_DIR} RESULT_VARIABLE HAVE_SEM_INIT_run OUTPUT_VARIABLE HAVE_SEM_INIT_compil) # Test sem_init by compiling: if(HAVE_SEM_INIT_compil) set(HAVE_SEM_INIT 0) message(STATUS "Warning: sem_init not compilable") message(STATUS "HAVE_SEM_INIT_comp_output: ${HAVE_SEM_OPEN_comp_output}") else() set(HAVE_SEM_INIT 1) message(STATUS "sem_init is compilable") endif() # If we're not crosscompiling, we check by executing the program: if (HAVE_SEM_INIT AND NOT CMAKE_CROSSCOMPILING) execute_process(COMMAND ./sem_init WORKING_DIRECTORY ${CMAKE_BINARY_DIR} RESULT_VARIABLE HAVE_SEM_INIT_run OUTPUT_VARIABLE var_compil ) if (NOT HAVE_SEM_INIT_run) set(HAVE_SEM_INIT 1) message(STATUS "sem_init is executable") else() set(HAVE_SEM_INIT 0) if(EXISTS "${CMAKE_BINARY_DIR}/sem_init") message(STATUS "Bin ${CMAKE_BINARY_DIR}/sem_init exists!") else() message(STATUS "Bin ${CMAKE_BINARY_DIR}/sem_init not exists!") endif() message(STATUS "Warning: sem_init not executable") message(STATUS "Compilation output: '${var_compil}'") message(STATUS "Exit result of sem_init: ${HAVE_SEM_INIT_run}") endif() endif() file(REMOVE sem_init) else() set(HAVE_SEM_INIT 0) endif() if(NOT HAVE_SEM_OPEN AND NOT HAVE_SEM_INIT) message(FATAL_ERROR "Semaphores are not usable (neither sem_open nor sem_init is both compilable and executable), but they are mandatory to threads (you may need to mount /dev).") endif() set(HAVE_THREAD_CONTEXTS 1) message("-- Support for thread context factory ok.") endif() set(HAVE_UCONTEXT_CONTEXTS 0) if(NOT HAVE_UCONTEXT_H) message("-- No ucontext factory: not found.") elseif(APPLE) message("-- No ucontext factory: Apple don't want us to use them.") set(HAVE_UCONTEXT_H 0) else() try_compile(compile_makecontext ${CMAKE_BINARY_DIR} ${CMAKE_HOME_DIRECTORY}/tools/cmake/test_prog/prog_makecontext.c OUTPUT_VARIABLE compile_makecontext_output) #If can have both context if(compile_makecontext) set(HAVE_UCONTEXT_CONTEXTS 1) message("-- Support for ucontext factory ok.") else() message("-- Error: exists, but makecontext is not compilable. Compilation output:\n ${compile_makecontext_output}") message("-- No ucontext factory: makecontext() is not compilable.") endif() # Stack setup (size and address) try_run(RUN_makecontext_VAR COMPILE_makecontext_VAR ${CMAKE_BINARY_DIR} ${CMAKE_HOME_DIRECTORY}/tools/cmake/test_prog/prog_stacksetup.c RUN_OUTPUT_VARIABLE stack_setup) LIST(LENGTH stack_setup stack_setup_len) if("${stack_setup_len}" STREQUAL "2") LIST(GET stack_setup 0 makecontext_addr) LIST(GET stack_setup 1 makecontext_size) set(sg_makecontext_stack_addr "#define sg_makecontext_stack_addr(skaddr) (${makecontext_addr})") set(sg_makecontext_stack_size "#define sg_makecontext_stack_size(sksize) (${makecontext_size})") else() message(FATAL_ERROR "Could not figure out the stack setup. Compil: ${RUN_makecontext_VAR}. Exec: ${COMPILE_makecontext_VAR}. Output: ${stack_setup}") endif() endif() # Stack growth direction (upward or downward). Used for the following contexts: SysV, raw, Boost try_run(RUN_stackgrowth_VAR COMPILE_stackgrowth_VAR ${CMAKE_BINARY_DIR} ${CMAKE_HOME_DIRECTORY}/tools/cmake/test_prog/prog_stackgrowth.c RUN_OUTPUT_VARIABLE stack COPY_FILE test_stackgrowth) if("${stack}" STREQUAL "down") set(PTH_STACKGROWTH "-1") elseif("${stack}" STREQUAL "up") set(PTH_STACKGROWTH "1") else() if("${CMAKE_SYSTEM_PROCESSOR}" STREQUAL "x86_64") set(PTH_STACKGROWTH "-1") elseif("${CMAKE_SYSTEM_PROCESSOR}" STREQUAL "i686") set(PTH_STACKGROWTH "-1") else() message(FATAL_ERROR "Could not figure out the stack direction. Test prog returned: ${stack}; CMAKE_SYSTEM_PROCESSOR: ${CMAKE_SYSTEM_PROCESSOR}.") endif() endif() # If the test ran well, remove the test binary file(REMOVE test_stackgrowth) #-------------------------------------------------------------------------------------------------- ### check for addr2line find_path(ADDR2LINE NAMES addr2line PATHS NO_DEFAULT_PATHS) if(ADDR2LINE) set(ADDR2LINE "${ADDR2LINE}/addr2line") endif() ############### ## GIT version check ## if(EXISTS ${CMAKE_HOME_DIRECTORY}/.git/) execute_process( COMMAND git remote COMMAND head -n 1 WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY}/.git/ OUTPUT_VARIABLE remote OUTPUT_STRIP_TRAILING_WHITESPACE) #message(STATUS "Git remote: ${remote}") execute_process(COMMAND git config --get remote.${remote}.url WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY}/.git/ OUTPUT_VARIABLE url OUTPUT_STRIP_TRAILING_WHITESPACE) #message(STATUS "Git url: ${url}") if(url) execute_process(COMMAND git --git-dir=${CMAKE_HOME_DIRECTORY}/.git log --pretty=oneline --abbrev-commit -1 WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY}/.git/ OUTPUT_VARIABLE GIT_VERSION OUTPUT_STRIP_TRAILING_WHITESPACE) message(STATUS "Git version: ${GIT_VERSION}") execute_process(COMMAND git --git-dir=${CMAKE_HOME_DIRECTORY}/.git log -n 1 --pretty=format:%ai . WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY}/.git/ OUTPUT_VARIABLE GIT_DATE OUTPUT_STRIP_TRAILING_WHITESPACE) message(STATUS "Git date: ${GIT_DATE}") string(REGEX REPLACE " .*" "" GIT_VERSION "${GIT_VERSION}") execute_process(COMMAND git --git-dir=${CMAKE_HOME_DIRECTORY}/.git log --pretty=format:%H -1 WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY}/.git/ OUTPUT_VARIABLE SIMGRID_GITHASH OUTPUT_STRIP_TRAILING_WHITESPACE) endif() elseif(EXISTS ${CMAKE_HOME_DIRECTORY}/.gitversion) FILE(STRINGS ${CMAKE_HOME_DIRECTORY}/.gitversion GIT_VERSION) endif() ### Generate the required headers and scripts ############################################# # Avoid triggering a (full) rebuild by touching the files if they did not really change configure_file("${CMAKE_HOME_DIRECTORY}/tools/cmake/src/internal_config.h.in" "${CMAKE_BINARY_DIR}/src/internal_config.h.generated" @ONLY IMMEDIATE) configure_file("${CMAKE_HOME_DIRECTORY}/include/simgrid_config.h.in" "${CMAKE_BINARY_DIR}/include/simgrid_config.h.generated" @ONLY IMMEDIATE) execute_process(COMMAND ${CMAKE_COMMAND} -E copy_if_different ${CMAKE_BINARY_DIR}/src/internal_config.h.generated ${CMAKE_BINARY_DIR}/src/internal_config.h) execute_process(COMMAND ${CMAKE_COMMAND} -E copy_if_different ${CMAKE_BINARY_DIR}/include/simgrid_config.h.generated ${CMAKE_BINARY_DIR}/include/simgrid_config.h) file(REMOVE ${CMAKE_BINARY_DIR}/src/internal_config.h.generated) file(REMOVE ${CMAKE_BINARY_DIR}/include/simgrid_config.h.generated) # We need two versions of the SMPI scripts because they contain the path to the library # so, it depends of whether SimGrid is installed, or run from the sources (during the build) file(READ ${CMAKE_HOME_DIRECTORY}/src/smpi/smpitools.sh SMPITOOLS_SH) # Definitions shared amongst all SMPI scripts, inlined in each of them ### SMPI script used when simgrid is installed set(exec_prefix ${CMAKE_INSTALL_PREFIX}) set(includeflag "-I${CMAKE_INSTALL_PREFIX}/include -I${CMAKE_INSTALL_PREFIX}/include/smpi") set(includedir "${CMAKE_INSTALL_PREFIX}/include") set(libdir ${exec_prefix}/lib) set(CMAKE_SMPI_COMMAND "export LD_LIBRARY_PATH=\"${CMAKE_INSTALL_PREFIX}/lib") if(NS3_LIBRARY_PATH) set(CMAKE_SMPI_COMMAND "${CMAKE_SMPI_COMMAND}:${NS3_LIBRARY_PATH}") endif() set(CMAKE_SMPI_COMMAND "${CMAKE_SMPI_COMMAND}:\${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}\"") configure_file(${CMAKE_HOME_DIRECTORY}/include/smpi/mpif.h.in ${CMAKE_BINARY_DIR}/include/smpi/mpif.h @ONLY) foreach(script cc cxx ff f90 run) configure_file(${CMAKE_HOME_DIRECTORY}/src/smpi/smpi${script}.in ${CMAKE_BINARY_DIR}/bin/smpi${script} @ONLY) endforeach() ### SMPI scripts used when compiling simgrid set(exec_prefix "${CMAKE_BINARY_DIR}/smpi_script/") set(includeflag "-I${CMAKE_HOME_DIRECTORY}/include -I${CMAKE_HOME_DIRECTORY}/include/smpi") set(includeflag "${includeflag} -I${CMAKE_BINARY_DIR}/include -I${CMAKE_BINARY_DIR}/include/smpi") set(includedir "${CMAKE_HOME_DIRECTORY}/include") set(libdir "${CMAKE_BINARY_DIR}/lib") set(CMAKE_SMPI_COMMAND "export LD_LIBRARY_PATH=\"${CMAKE_BINARY_DIR}/lib") if(NS3_LIBRARY_PATH) set(CMAKE_SMPI_COMMAND "${CMAKE_SMPI_COMMAND}:${NS3_LIBRARY_PATH}") endif() set(CMAKE_SMPI_COMMAND "${CMAKE_SMPI_COMMAND}:\${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}\"") foreach(script cc cxx ff f90 run) configure_file(${CMAKE_HOME_DIRECTORY}/src/smpi/smpi${script}.in ${CMAKE_BINARY_DIR}/smpi_script/bin/smpi${script} @ONLY) endforeach() if(NOT WIN32) foreach(script cc cxx ff f90 run) execute_process(COMMAND chmod a=rwx ${CMAKE_BINARY_DIR}/bin/smpi${script}) execute_process(COMMAND chmod a=rwx ${CMAKE_BINARY_DIR}/smpi_script/bin/smpi${script}) endforeach() endif() set(generated_headers_to_install ${CMAKE_CURRENT_BINARY_DIR}/include/smpi/mpif.h ${CMAKE_CURRENT_BINARY_DIR}/include/simgrid_config.h ) set(generated_headers ${CMAKE_CURRENT_BINARY_DIR}/src/internal_config.h ) set(generated_files_to_clean ${generated_headers} ${generated_headers_to_install} ${CMAKE_BINARY_DIR}/bin/smpicc ${CMAKE_BINARY_DIR}/bin/smpicxx ${CMAKE_BINARY_DIR}/bin/smpiff ${CMAKE_BINARY_DIR}/bin/smpif90 ${CMAKE_BINARY_DIR}/bin/smpirun ${CMAKE_BINARY_DIR}/bin/colorize ${CMAKE_BINARY_DIR}/bin/simgrid_update_xml ${CMAKE_BINARY_DIR}/examples/smpi/tracing/smpi_traced.trace ) if(NOT "${CMAKE_BINARY_DIR}" STREQUAL "${CMAKE_HOME_DIRECTORY}") configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions0.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions0.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions1.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions1.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions_allReduce.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_allReduce.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions_barrier.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_barrier.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions_bcast.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_bcast.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions_with_isend.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_with_isend.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions_alltoall.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_alltoall.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions_alltoallv.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_alltoallv.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions_waitall.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_waitall.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions_reducescatter.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_reducescatter.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions_gather.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_gather.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay/actions_allgatherv.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_allgatherv.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/teshsuite/smpi/hostfile ${CMAKE_BINARY_DIR}/teshsuite/smpi/hostfile COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/description_file ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/description_file COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/README ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/README COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/smpi_replay.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/smpi_replay.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace0.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace0.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace1.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace1.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace2.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace2.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace3.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace3.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace4.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace4.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace5.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace5.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace6.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace6.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace7.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace7.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace8.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace8.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace9.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace9.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace10.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace10.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace11.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace11.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace12.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace12.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace13.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace13.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace14.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace14.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace15.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace15.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace16.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace16.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace17.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace17.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace18.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace18.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace19.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace19.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace20.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace20.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace21.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace21.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace22.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace22.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace23.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace23.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace24.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace24.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace25.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace25.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace26.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace26.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace27.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace27.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace28.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace28.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace29.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace29.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace30.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace30.txt COPYONLY) configure_file(${CMAKE_HOME_DIRECTORY}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace31.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace31.txt COPYONLY) set(generated_files_to_clean ${generated_files_to_clean} ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions0.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions1.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_allReduce.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_barrier.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_bcast.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_with_isend.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_alltoall.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_alltoallv.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_waitall.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_gather.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_allgatherv.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay/actions_reducescatter.txt ${CMAKE_BINARY_DIR}/teshsuite/smpi/hostfile ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/description_file ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/README ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/smpi_replay.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace0.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace1.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace2.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace3.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace4.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace5.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace6.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace7.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace8.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace9.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace10.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace11.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace12.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace13.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace14.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace15.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace16.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace17.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace18.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace19.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace20.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace21.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace22.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace23.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace24.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace25.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace26.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace27.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace28.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace29.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace30.txt ${CMAKE_BINARY_DIR}/examples/smpi/replay_multiple/ti_traces_32_1/ti_trace31.txt ) endif() SET_DIRECTORY_PROPERTIES(PROPERTIES ADDITIONAL_MAKE_CLEAN_FILES "${generated_files_to_clean}") ### Define source packages for Libs include(${CMAKE_HOME_DIRECTORY}/tools/cmake/DefinePackages.cmake) ### Build some Maintainer files include(${CMAKE_HOME_DIRECTORY}/tools/cmake/MaintainerMode.cmake) include(${CMAKE_HOME_DIRECTORY}/tools/cmake/UnitTesting.cmake) ### Setup gcc & clang flags if (NOT MSVC) include(${CMAKE_HOME_DIRECTORY}/tools/cmake/GCCFlags.cmake) endif() ### Make Libs if(NOT WIN32) include(${CMAKE_HOME_DIRECTORY}/tools/cmake/MakeLib.cmake) else() include(${CMAKE_HOME_DIRECTORY}/tools/cmake/MakeLibWin.cmake) endif() if(enable_java) include(${CMAKE_HOME_DIRECTORY}/tools/cmake/Java.cmake) endif() ### Make tests if(enable_memcheck_xml) set(enable_memcheck true) endif() INCLUDE(CTest) ENABLE_TESTING() include(${CMAKE_HOME_DIRECTORY}/tools/cmake/Tests.cmake) include(${CMAKE_HOME_DIRECTORY}/tools/cmake/CTestConfig.cmake) ### Define subdirectories foreach(cmakefile ${CMAKEFILES_TXT}) string(REPLACE "/CMakeLists.txt" "" repository ${cmakefile}) add_subdirectory("${CMAKE_HOME_DIRECTORY}/${repository}") endforeach() ### Setup the distrib include(${CMAKE_HOME_DIRECTORY}/tools/cmake/Distrib.cmake) ### Build the docs if asked to include(${CMAKE_HOME_DIRECTORY}/tools/cmake/Documentation.cmake) ### Print the result of configuration message("") message("##########################################") message("#### Content of src/internal_config.h ####") message("##########################################") file(STRINGS ${CMAKE_CURRENT_BINARY_DIR}/src/internal_config.h config_output) LIST(REMOVE_AT config_output 0 1 2 3 4 5 6 7 8 9 10) # Pass the file header foreach(line ${config_output}) message(" ${line}") endforeach() message("##########################################") message("#### Content of simgrid_config.h ####") message("##########################################") file(STRINGS ${CMAKE_CURRENT_BINARY_DIR}/include/simgrid_config.h config_output) LIST(REMOVE_AT config_output 0 1 2 3 4 5 6 7 8 9 -1) # Pass the file header foreach(line ${config_output}) message(" ${line}") endforeach() message("##########################################") message("#### End of configuration headers ####") message("##########################################") message("\nConfiguration of package `simgrid':") message(" Home directory ..............: ${CMAKE_HOME_DIRECTORY}") message(" Build Name ..................: ${BUILDNAME}") message(" Cmake Generator .............: ${CMAKE_GENERATOR}") message(" Site ........................: ${SITE}") message(" Install prefix ..............: ${CMAKE_INSTALL_PREFIX}") if(release) message(" Release .....................: simgrid-${release_version}${SIMGRID_VERSION_EXTRA} (release build)") else() message(" Release .....................: simgrid-${release_version}${SIMGRID_VERSION_EXTRA} (development build)") endif() message("") message(" Compiler: C .................: ${CMAKE_C_COMPILER} (id: ${CMAKE_C_COMPILER_ID})") message(" version .............: ${CMAKE_C_COMPILER_VERSION}") message(" is gnu ..............: ${CMAKE_COMPILER_IS_GNUCC}") message(" Compiler: C++ ...............: ${CMAKE_CXX_COMPILER} (id: ${CMAKE_CXX_COMPILER_ID})") message(" version .............: ${CMAKE_CXX_COMPILER_VERSION}") if(${Java_FOUND}) message(" Compiler: Javac .............: ${Java_JAVAC_EXECUTABLE}") message(" version .............: ${Java_VERSION_STRING}") if(${SWIG_FOUND}) message(" SWIG ................: Version ${SWIG_VERSION} found.") else() message(" SWIG ................: Missing.") endif() endif() if(CMAKE_Fortran_COMPILER) message(" Compiler: Fortran ...........: ${SAVED_Fortran_COMPILER} (id: ${CMAKE_Fortran_COMPILER_ID})") message(" version .............: ${CMAKE_Fortran_COMPILER_VERSION}") endif() message(" Linker: .....................: ${CMAKE_LINKER}") message(" version .............: ${LINKER_VERSION}") message(" Make program: ...............: ${CMAKE_MAKE_PROGRAM}") message("") message(" CFlags ......................: ${CMAKE_C_FLAGS}") message(" CXXFlags ....................: ${CMAKE_CXX_FLAGS}") message(" LDFlags .....................: ${CMAKE_C_LINK_FLAGS}") message(" with LTO ....................: ${enable_lto}") message("") if (HAVE_NS3) message(" Compile NS-3 ................: yes (path: ${NS3_PATH})") else() message(" Compile NS-3 ................: NO (hint: ${NS3_HINT})") endif() if (${Java_FOUND}) message(" Compile Java ................: yes") message(" Native lib in jar .........: ${enable_lib_in_jar}") else() message(" Compile Java ................: NO") endif() message(" Compile Lua .................: ${HAVE_LUA}") message(" Compile Smpi ................: ${HAVE_SMPI}") message(" Smpi fortran ..............: ${SMPI_FORTRAN}") message(" MPICH3 testsuite ..........: ${enable_smpi_MPICH3_testsuite}") message(" Privatization .............: ${HAVE_PRIVATIZATION}") message(" PAPI support...............: ${HAVE_PAPI}") message(" Compile Boost.Context support: ${HAVE_BOOST_CONTEXTS}") message("") message(" Maintainer mode .............: ${enable_maintainer_mode}") message(" Documentation................: ${enable_documentation}") message(" Model checking ..............: ${HAVE_MC}") message(" Jedule mode ................: ${HAVE_JEDULE}") message(" Graphviz mode ...............: ${HAVE_GRAPHVIZ}") message(" Mallocators .................: ${enable_mallocators}") message("") message(" Simgrid dependencies ........: ${SIMGRID_DEP}") message("") execute_process(COMMAND ${CMAKE_COMMAND} -E make_directory ${PROJECT_BINARY_DIR}/Testing/Notes/) file(WRITE ${PROJECT_BINARY_DIR}/Testing/Notes/Build "GIT version : ${GIT_VERSION}\n") file(APPEND ${PROJECT_BINARY_DIR}/Testing/Notes/Build "Release : simgrid-${release_version}\n") INCLUDE(Dart) SimGrid-3.14.159/examples/000755 001750 001750 00000000000 13031031275 016711 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/000755 001750 001750 00000000000 13031031277 017634 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/energy/000755 001750 001750 00000000000 13031031276 021124 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/energy/consumption/000755 001750 001750 00000000000 13031031303 023471 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/energy/consumption/Main.java000644 001750 001750 00000002217 13031031276 025233 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package energy.consumption; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; public class Main { public static final double task_comp_size = 10; public static final double task_comm_size = 10; public static final int hostNB = 2 ; private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws MsgException { Msg.energyInit(); Msg.init(args); if (args.length < 1) { Msg.info("Usage : Energy platform_file"); Msg.info("Usage : Energy ../platforms/energy_platform.xml"); System.exit(1); } /* Construct the platform */ Msg.createEnvironment(args[0]); /* Instanciate a process */ new EnergyConsumer("MyHost1","energyConsumer").start(); /* Execute the simulation */ Msg.run(); Msg.info("Total simulation time: "+ Msg.getClock()); } } SimGrid-3.14.159/examples/java/energy/consumption/EnergyConsumer.java000644 001750 001750 00000002256 13031031276 027317 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package energy.consumption; import org.simgrid.msg.Msg; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; import org.simgrid.msg.HostNotFoundException; public class EnergyConsumer extends Process { public EnergyConsumer(String hostname, String name) throws HostNotFoundException{ super(hostname,name); } public void main(String[] args) throws MsgException { Msg.info("Energetic profile: " + getHost().getProperty("watt_per_state")); Msg.info("Initial peak speed= " + getHost().getSpeed() + " flop/s; Energy dissipated = " + getHost().getConsumedEnergy() + " J"); this.waitFor(10); Msg.info("Currently consumed energy after sleeping 10 sec: "+getHost().getConsumedEnergy()); new Task(null, 1E9, 0).execute(); Msg.info("Currently consumed energy after executing 1E9 flops: "+getHost().getConsumedEnergy()); } } SimGrid-3.14.159/examples/java/energy/consumption/energy_consumption.tesh000644 001750 001750 00000001767 13031031303 030320 0ustar00mquinsonmquinson000000 000000 #! tesh ! timeout 15 $ java -classpath ${classpath:=.} energy/consumption/Main ${srcdir:=.}/../platforms/energy_platform.xml > [0.000000] [jmsg/INFO] Using regular java threads. > [MyHost1:energyConsumer:(1) 0.000000] [jmsg/INFO] Energetic profile: 100.0:120.0:200.0, 93.0:110.0:170.0, 90.0:105.0:150.0 > [MyHost1:energyConsumer:(1) 0.000000] [jmsg/INFO] Initial peak speed= 1.0E8 flop/s; Energy dissipated = 0.0 J > [MyHost1:energyConsumer:(1) 10.000000] [jmsg/INFO] Currently consumed energy after sleeping 10 sec: 1000.0 > [MyHost1:energyConsumer:(1) 20.000000] [jmsg/INFO] Currently consumed energy after executing 1E9 flops: 2200.0 > [20.000000] [jmsg/INFO] MSG_main finished; Cleaning up the simulation... > [20.000000] [jmsg/INFO] Total simulation time: 20.0 > [20.000000] [surf_energy/INFO] Total energy of host MyHost1: 2200.000000 Joules > [20.000000] [surf_energy/INFO] Total energy of host MyHost2: 2000.000000 Joules > [20.000000] [surf_energy/INFO] Total energy of host MyHost3: 2000.000000 Joules SimGrid-3.14.159/examples/java/energy/vm/000755 001750 001750 00000000000 13031031303 021535 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/energy/vm/Main.java000644 001750 001750 00000001702 13031031276 023275 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2016. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package energy.vm; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.HostNotFoundException; import org.simgrid.msg.NativeException; class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws NativeException, HostNotFoundException { Msg.energyInit(); Msg.init(args); if (args.length < 1) { Msg.info("Usage: Main ../platforms/energy_platform_file.xml"); System.exit(1); } /* construct the platform */ Msg.createEnvironment(args[0]); /* Create and start a runner for the experiment */ new EnergyVMRunner(Host.all()[0],"energy VM runner",null).start(); Msg.run(); } } SimGrid-3.14.159/examples/java/energy/vm/EnergyVMRunner.java000644 001750 001750 00000005102 13031031276 025275 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2016. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package energy.vm; import org.simgrid.msg.Msg; import org.simgrid.msg.VM; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.HostFailureException; import org.simgrid.msg.HostNotFoundException; import org.simgrid.msg.TaskCancelledException; import org.simgrid.msg.NativeException; /* This class is a process in charge of running the test. It creates and starts the VMs, and fork processes within VMs */ public class EnergyVMRunner extends Process { public class DummyProcess extends Process { public DummyProcess (Host host, String name) { super(host, name); } @Override public void main(String[] strings) { Task task = new Task(this.getHost().getName()+"-task", 300E6 , 0); try { task.execute(); } catch (HostFailureException | TaskCancelledException e) { Msg.error(e.getMessage()); e.printStackTrace(); } Msg.info("This worker is done."); } } EnergyVMRunner(Host host, String name, String[] args) throws HostNotFoundException, NativeException { super(host, name, args); } @Override public void main(String[] strings) throws HostNotFoundException, HostFailureException { /* get hosts */ Host host1 = Host.getByName("MyHost1"); Host host2 = Host.getByName("MyHost2"); Host host3 = Host.getByName("MyHost3"); Msg.info("Creating and starting two VMs"); VM vmHost1 = new VM(host1, "vmHost1", 2048, 10, 50); vmHost1.start(); VM vmHost3 = new VM(host3, "vmHost3", 2048, 10, 50); vmHost3.start(); Msg.info("Create two tasks on Host1: one inside a VM, the other directly on the host"); new DummyProcess (vmHost1, "p11"); new DummyProcess (host1, "p12"); Msg.info("Create two tasks on Host2: both directly on the host"); new DummyProcess (host2, "p21"); new DummyProcess (host2, "p22"); Msg.info("Create two tasks on Host3: both inside a VM"); new DummyProcess (vmHost3, "p31"); new DummyProcess (vmHost3, "p312"); Msg.info("Wait 5 seconds. The tasks are still running (they run for 3 seconds, but 2 tasks are co-located, " + "so they run for 6 seconds)"); waitFor(5); Msg.info("Wait another 5 seconds. The tasks stop at some point in between"); waitFor(5); vmHost1.destroy(); vmHost3.destroy(); } } SimGrid-3.14.159/examples/java/energy/vm/energy_vm.tesh000644 001750 001750 00000002250 13031031303 024414 0ustar00mquinsonmquinson000000 000000 #! tesh $ java -classpath ${classpath:=.} energy/vm/Main ${srcdir:=.}/../platforms/energy_platform.xml > [0.000000] [jmsg/INFO] Using regular java threads. > [MyHost1:energy VM runner:(1) 0.000000] [jmsg/INFO] Creating and starting two VMs > [MyHost1:energy VM runner:(1) 0.000000] [jmsg/INFO] Create two tasks on Host1: one inside a VM, the other directly on the host > [MyHost1:energy VM runner:(1) 0.000000] [jmsg/INFO] Create two tasks on Host2: both directly on the host > [MyHost1:energy VM runner:(1) 0.000000] [jmsg/INFO] Create two tasks on Host3: both inside a VM > [MyHost1:energy VM runner:(1) 0.000000] [jmsg/INFO] Wait 5 seconds. The tasks are still running (they run for 3 seconds, but 2 tasks are co-located, so they run for 6 seconds) > [MyHost1:energy VM runner:(1) 5.000000] [jmsg/INFO] Wait another 5 seconds. The tasks stop at some point in between > [10.000000] [jmsg/INFO] MSG_main finished; Cleaning up the simulation... > [10.000000] [surf_energy/INFO] Total energy of host MyHost1: 1000.000000 Joules > [10.000000] [surf_energy/INFO] Total energy of host MyHost2: 1000.000000 Joules > [10.000000] [surf_energy/INFO] Total energy of host MyHost3: 1000.000000 Joules SimGrid-3.14.159/examples/java/.classpath000644 001750 001750 00000000612 13031031273 021612 0ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/CMakeLists.txt000644 001750 001750 00000016371 13031031274 022401 0ustar00mquinsonmquinson000000 000000 set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/app/bittorrent) set(app_bittorrent_sources ${srcdir}/Main.java ${srcdir}/Common.java ${srcdir}/Connection.java ${srcdir}/MessageTask.java ${srcdir}/Peer.java ${srcdir}/Tracker.java ${srcdir}/TrackerTask.java) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/app/centralizedmutex) set(app_centralizedmutex_sources ${srcdir}/Main.java ${srcdir}/Coordinator.java ${srcdir}/GrantTask.java ${srcdir}/Node.java ${srcdir}/ReleaseTask.java ${srcdir}/RequestTask.java) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/app/masterworker) set(app_masterworker_sources ${srcdir}/Main.java ${srcdir}/Master.java ${srcdir}/Worker.java) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/app/pingpong) set(app_pingpong_sources ${srcdir}/Main.java ${srcdir}/PingPongTask.java ${srcdir}/Receiver.java ${srcdir}/Sender.java) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/app/tokenring) set(app_tokenring_sources ${srcdir}/Main.java ${srcdir}/RelayRunner.java) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/async/waitall) set(async_waitall_sources ${srcdir}/Main.java ${srcdir}/Receiver.java ${srcdir}/Sender.java) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/async/dsend) set(async_dsend_sources ${srcdir}/Main.java ${srcdir}/Receiver.java ${srcdir}/Sender.java) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/cloud/masterworker) set(cloud_masterworker_sources ${srcdir}/Main.java ${srcdir}/Master.java ${srcdir}/Worker.java) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/cloud/migration) set(cloud_migration_sources ${srcdir}/Main.java ${srcdir}/Daemon.java ${srcdir}/Test.java ${srcdir}/TestHostOnOff.java ${srcdir}/XVM.java) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/dht/chord) set(dht_chord_sources ${srcdir}/Main.java ${srcdir}/ChordTask.java ${srcdir}/Common.java ${srcdir}/FindSuccessorAnswerTask.java ${srcdir}/FindSuccessorTask.java ${srcdir}/GetPredecessorAnswerTask.java ${srcdir}/GetPredecessorTask.java ${srcdir}/Node.java ${srcdir}/NotifyTask.java) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/dht/kademlia) set(dht_kademlia_sources ${srcdir}/Main.java ${srcdir}/Answer.java ${srcdir}/Bucket.java ${srcdir}/Common.java ${srcdir}/Contact.java ${srcdir}/FindNodeAnswerTask.java ${srcdir}/FindNodeTask.java ${srcdir}/KademliaTask.java ${srcdir}/Node.java ${srcdir}/PingAnswerTask.java ${srcdir}/PingTask.java ${srcdir}/RoutingTable.java) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/trace/pingpong) set(trace_pingpong_sources ${srcdir}/Main.java ${srcdir}/PingPongTask.java ${srcdir}/Receiver.java ${srcdir}/Sender.java) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/energy/consumption) set(energy_consumption_sources ${srcdir}/Main.java ${srcdir}/EnergyConsumer.java) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/energy/vm) set(energy_vm_sources ${srcdir}/Main.java ${srcdir}/EnergyVMRunner.java) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/io/file) set(io_file_sources ${srcdir}/Main.java ${srcdir}/Node.java) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/io/storage) set(io_storage_sources ${srcdir}/Main.java ${srcdir}/Client.java) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/process/kill) set(process_kill_sources ${srcdir}/Main.java ${srcdir}/Killer.java ${srcdir}/Victim.java) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/process/migration) set(process_migration_sources ${srcdir}/Main.java ${srcdir}/Emigrant.java ${srcdir}/Policeman.java) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/process/startkilltime) set(process_startkilltime_sources ${srcdir}/Main.java ${srcdir}/Sleeper.java) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/process/suspend) set(process_suspend_sources ${srcdir}/Main.java ${srcdir}/DreamMaster.java ${srcdir}/LazyGuy.java) set (srcdir ${CMAKE_CURRENT_SOURCE_DIR}/task/priority) set(task_priority_sources ${srcdir}/Main.java ${srcdir}/Test.java) foreach (example app_bittorrent app_centralizedmutex app_masterworker app_pingpong app_tokenring async_waitall async_dsend cloud_migration cloud_masterworker dht_chord dht_kademlia energy_consumption energy_vm io_file io_storage process_kill process_migration process_startkilltime process_suspend task_priority trace_pingpong) string (REPLACE "_" "/" example_dir ${example}) if(enable_java) add_custom_command( COMMENT "Building java_${example}..." OUTPUT ${example_dir}/java_${example}_compiled DEPENDS ${example_sources} simgrid-java_jar ${SIMGRID_JAR} COMMAND ${CMAKE_COMMAND} -E make_directory ${example_dir} COMMAND ${JAVA_COMPILE} -classpath ${SIMGRID_JAR} -d ${CMAKE_CURRENT_BINARY_DIR} ${${example}_sources} COMMAND ${CMAKE_COMMAND} -E remove ${example_dir}/java_${example}_compiled COMMAND ${CMAKE_COMMAND} -E touch ${example_dir}/java_${example}_compiled ) add_custom_target(${example} ALL DEPENDS ${example_dir}/java_${example}_compiled) endif() set(examples_src ${examples_src} ${${example}_sources}) set(tesh_files ${tesh_files} ${CMAKE_CURRENT_SOURCE_DIR}/${example_dir}/${example}.tesh) endforeach() set(examples_src ${examples_src} PARENT_SCOPE) set(tesh_files ${tesh_files} PARENT_SCOPE) set(bin_files ${bin_files} ${CMAKE_CURRENT_SOURCE_DIR}/app/bittorrent/generate.py PARENT_SCOPE) set(txt_files ${txt_files} ${CMAKE_CURRENT_SOURCE_DIR}/app/masterworker/README ${CMAKE_CURRENT_SOURCE_DIR}/cloud/migration/README PARENT_SCOPE) set(xml_files ${xml_files} ${CMAKE_CURRENT_SOURCE_DIR}/app/bittorrent/bittorrent.xml ${CMAKE_CURRENT_SOURCE_DIR}/app/centralizedmutex/centralizedmutex.xml ${CMAKE_CURRENT_SOURCE_DIR}/app/masterworker/masterworker.xml ${CMAKE_CURRENT_SOURCE_DIR}/dht/chord/chord.xml ${CMAKE_CURRENT_SOURCE_DIR}/dht/kademlia/kademlia.xml ${CMAKE_CURRENT_SOURCE_DIR}/process/startkilltime/startkilltime.xml ${CMAKE_CURRENT_SOURCE_DIR}/task/priority/priority.xml PARENT_SCOPE) if(enable_java) foreach (example app_bittorrent app_centralizedmutex app_masterworker app_pingpong app_tokenring async_waitall async_dsend cloud_migration cloud_masterworker dht_chord dht_kademlia energy_consumption energy_vm io_file io_storage process_kill process_migration process_startkilltime process_suspend task_priority trace_pingpong) string (REPLACE "_" "/" example_dir ${example}) ADD_TESH(java-${example} --setenv srcdir=${CMAKE_HOME_DIRECTORY}/examples/java --setenv LD_LIBRARY_PATH=${CMAKE_BINARY_DIR}/lib --setenv classpath=${TESH_CLASSPATH} --cd ${CMAKE_BINARY_DIR}/examples/java ${CMAKE_HOME_DIRECTORY}/examples/java/${example_dir}/${example}.tesh) endforeach() endif() SimGrid-3.14.159/examples/java/.project000644 001750 001750 00000003701 13031031273 021300 0ustar00mquinsonmquinson000000 000000 SimGrid Examples org.eclipse.jdt.core.javabuilder org.eclipse.jdt.core.javanature 1471870571873 26 org.eclipse.ui.ide.multiFilter 1.0-name-matches-false-false-CMakeFiles 1471870571874 22 org.eclipse.ui.ide.multiFilter 1.0-projectRelativePath-matches-false-false-**/*.cmake 1471870571876 22 org.eclipse.ui.ide.multiFilter 1.0-name-matches-false-false-*~ 1471870571880 22 org.eclipse.ui.ide.multiFilter 1.0-name-matches-false-false-*_compiled 1471870571882 22 org.eclipse.ui.ide.multiFilter 1.0-name-matches-false-false-*.cmake 1471870571887 22 org.eclipse.ui.ide.multiFilter 1.0-name-matches-false-false-CMakeLists.txt 1471870571889 22 org.eclipse.ui.ide.multiFilter 1.0-name-matches-false-false-Makefile SimGrid-3.14.159/examples/java/cloud/000755 001750 001750 00000000000 13031031276 020741 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/cloud/masterworker/000755 001750 001750 00000000000 13031031303 023455 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/cloud/masterworker/Main.java000644 001750 001750 00000002512 13031031276 025215 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package cloud.masterworker; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.MsgException; class Main { public static final double TASK_COMP_SIZE = 10; public static final double TASK_COMM_SIZE = 10; public static final int NHOSTS = 2 ; private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws MsgException { Msg.init(args); if (args.length < 1) { Msg.info("Usage : Main platform_file"); Msg.info("Usage : Main ../platforms/platform.xml"); System.exit(1); } /* Construct the platform */ Msg.createEnvironment(args[0]); Host[] hosts = Host.all(); if (hosts.length < NHOSTS+1) { Msg.info("I need at least "+ (NHOSTS+1) +" hosts in the platform file, but " + args[0] + " contains only " + hosts.length + " hosts"); System.exit(42); } Msg.info("Start "+ NHOSTS +" hosts"); new Master(hosts[0],"Master",hosts).start(); /* Execute the simulation */ Msg.run(); } } SimGrid-3.14.159/examples/java/cloud/masterworker/Worker.java000644 001750 001750 00000002237 13031031276 025606 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package cloud.masterworker; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; public class Worker extends Process { private int number; public Worker(Host host, int number) { super(host,"WRK0" + number,null); this.number = number; } public void main(String[] args) throws MsgException { Msg.info(this.getName() +" is listening on MBOX:WRK0"+ number); while(true) { Task task =null; try { task = Task.receive("MBOX:WRK0"+number); if (task == null) break; } catch (MsgException e) { Msg.debug("Received failed. I'm done. See you!"); } Msg.info("Received \"" + task.getName() + "\". Processing it."); task.execute(); Msg.info(this.getName() +" executed task (" + task.getName()+")"); } } } SimGrid-3.14.159/examples/java/cloud/masterworker/cloud_masterworker.tesh000644 001750 001750 00000003430 13031031303 030255 0ustar00mquinsonmquinson000000 000000 #! tesh ! output sort 19 $ java -classpath ${classpath:=.} cloud/masterworker/Main ${srcdir:=.}/../platforms/small_platform.xml "--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n" > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (0:maestro@) Start 2 hosts > [ 0.000000] (1:Master@Boivin) create VM00 > [ 0.000000] (1:Master@Boivin) Put Worker WRK00 on VM00 > [ 0.000000] (1:Master@Boivin) create VM01 > [ 0.000000] (1:Master@Boivin) Put Worker WRK01 on VM01 > [ 0.000000] (1:Master@Boivin) Launched 2 VMs > [ 0.000000] (1:Master@Boivin) Send a first batch of work to everyone > [ 0.000000] (1:Master@Boivin) Sending to WRK00 > [ 0.000000] (2:WRK00@VM00) WRK00 is listening on MBOX:WRK00 > [ 0.000000] (3:WRK01@VM01) WRK01 is listening on MBOX:WRK01 > [ 0.090194] (1:Master@Boivin) Sending to WRK01 > [ 0.090194] (2:WRK00@VM00) Received "Task00". Processing it. > [ 0.090194] (2:WRK00@VM00) WRK00 executed task (Task00) > [ 0.754233] (1:Master@Boivin) Suspend all VMs > [ 0.754233] (1:Master@Boivin) Suspend VM00 > [ 0.754233] (1:Master@Boivin) Suspend VM01 > [ 0.754233] (1:Master@Boivin) Wait a while > [ 0.754233] (3:WRK01@VM01) Received "Task01". Processing it. > [ 2.754233] (1:Master@Boivin) Resume all VMs. > [ 2.754233] (1:Master@Boivin) Sleep long enough for everyone to be done with previous batch of work > [ 2.754234] (3:WRK01@VM01) WRK01 executed task (Task01) > [1000.000000] (1:Master@Boivin) Migrate everyone to Ginette > [1000.000000] (1:Master@Boivin) Migrate VM00fromBourassato Ginette > [1100.157755] (1:Master@Boivin) Migrate VM01fromFafardto Ginette > [1242.754111] (0:maestro@) MSG_main finished; Cleaning up the simulation... > [1242.754111] (1:Master@Boivin) Let's shut down the simulation and kill everyone. > [1242.754111] (1:Master@Boivin) Master done. SimGrid-3.14.159/examples/java/cloud/masterworker/Master.java000644 001750 001750 00000005226 13031031276 025571 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package cloud.masterworker; import java.util.ArrayList; import org.simgrid.msg.Msg; import org.simgrid.msg.VM; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; public class Master extends Process { private Host[] hosts; public Master(Host host, String name, Host[] hosts) { super(host,name,null); this.hosts = hosts; } public void main(String[] args) throws MsgException { int workersCount = Main.NHOSTS; ArrayList vms = new ArrayList<>(); // Create one VM per host and bind a process inside each one. for (int i = 0; i < workersCount; i++) { Msg.info("create VM0"+i); VM vm = new VM(hosts[i+1],"VM0"+i); vm.start(); vms.add(vm); Worker worker= new Worker(vm,i); Msg.info("Put Worker "+worker.getName()+ " on "+vm.getName()); worker.start(); } Msg.info("Launched " + vms.size() + " VMs"); Msg.info("Send a first batch of work to everyone"); workBatch(workersCount); Msg.info("Suspend all VMs"); for (int i = 0; i < vms.size(); i++) { Msg.info("Suspend "+vms.get(i).getName()); vms.get(i).suspend(); } Msg.info("Wait a while"); waitFor(2); Msg.info("Resume all VMs."); for (int i = 0; i < vms.size(); i++) { vms.get(i).resume(); } Msg.info("Sleep long enough for everyone to be done with previous batch of work"); waitFor(1000 - Msg.getClock()); /* Msg.info("Add one more process per VM."); for (int i = 0; i < vms.size(); i++) { VM vm = vms.get(i); Worker worker = new Worker(vm,i + vms.size()); worker.start(); } workBatch(workersCount * 2); */ Msg.info("Migrate everyone to "+hosts[3].getName()); for (int i = 0; i < vms.size(); i++) { Msg.info("Migrate "+vms.get(i).getName()+"from"+hosts[i+1].getName()+"to "+hosts[3].getName()); vms.get(i).migrate(hosts[3]); } Msg.info("Let's shut down the simulation and kill everyone."); for (int i = 0; i < vms.size(); i++) { vms.get(i).destroy(); } Msg.info("Master done."); } public void workBatch(int workersCount) throws MsgException { for (int i = 0; i < workersCount; i++) { Task task = new Task("Task0" + i, Main.TASK_COMP_SIZE, Main.TASK_COMM_SIZE); Msg.info("Sending to WRK0" + i); task.send("MBOX:WRK0" + i); } } } SimGrid-3.14.159/examples/java/cloud/migration/000755 001750 001750 00000000000 13031031306 022724 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/cloud/migration/Main.java000644 001750 001750 00000001744 13031031276 024467 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package cloud.migration; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; public class Main { private static boolean endOfTest = false; private Main() { throw new IllegalAccessError("Utility class"); } public static void setEndOfTest(){ endOfTest=true; } public static boolean isEndOfTest(){ return endOfTest; } public static void main(String[] args) throws MsgException { Msg.init(args); if (args.length < 1) { Msg.info("Usage : Main platform_file.xml"); System.exit(1); } /* construct the platform and deploy the application */ Msg.createEnvironment(args[0]); new cloud.migration.Test("PM0","Test").start(); Msg.run(); } } SimGrid-3.14.159/examples/java/cloud/migration/XVM.java000644 001750 001750 00000004311 13031031276 024246 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package cloud.migration; import org.simgrid.msg.Msg; import org.simgrid.msg.VM; import org.simgrid.msg.Host; import org.simgrid.msg.HostNotFoundException; import org.simgrid.msg.HostFailureException; public class XVM extends VM { private int dpIntensity; private int ramsize; private int currentLoad; private Daemon daemon; public XVM(Host host, String name, int ramsize, int migNetBW, int dpIntensity){ super(host, name, ramsize, (int)(migNetBW*0.9), dpIntensity); this.currentLoad = 0; this.dpIntensity = dpIntensity ; this.ramsize= ramsize; this.daemon = new Daemon(this, 100); } public void setLoad(int load){ if (load >0) { this.setBound(this.getSpeed()*load/100); // this.getDaemon().setLoad(load); daemon.resume(); } else{ daemon.suspend(); } currentLoad = load ; } public void start(){ super.start(); try { daemon.start(); } catch (HostNotFoundException e) { e.printStackTrace(); } this.setLoad(0); } public Daemon getDaemon(){ return this.daemon; } public int getLoad(){ System.out.println("Remaining comp:" + this.daemon.getRemaining()); return this.currentLoad; } public void migrate(Host host) throws HostFailureException { Msg.info("Start migration of VM " + this.getName() + " to " + host.getName()); Msg.info(" currentLoad:" + this.currentLoad + "/ramSize:" + this.ramsize + "/dpIntensity:" + this.dpIntensity + "/remaining:" + String.format(java.util.Locale.US, "%.2E",this.daemon.getRemaining())); try{ super.migrate(host); } catch (Exception e){ Msg.info("Something wrong during the live migration of VM "+this.getName()); throw new HostFailureException(); } this.setLoad(this.currentLoad); //Fixed the fact that setBound is not propagated to the new node. Msg.info("End of migration of VM " + this.getName() + " to node " + host.getName()); } } SimGrid-3.14.159/examples/java/cloud/migration/cloud_migration.tesh000644 001750 001750 00000005444 13031031303 026774 0ustar00mquinsonmquinson000000 000000 #! tesh $ java -classpath ${classpath:=.} cloud/migration/Main ${srcdir:=.}/../platforms/three_multicore_hosts.xml > [0.000000] [jmsg/INFO] Using regular java threads. > [PM0:Test:(1) 0.000000] [jmsg/INFO] This example evaluates the migration time of a VM in presence of collocated VMs on the source and the dest nodes > [PM0:Test:(1) 0.000000] [jmsg/INFO] The migrated VM has a memory intensity rate of 70% of the network BW and a cpu load of 90% " (see cloudcom 2013 paper "Adding a Live Migration Model Into SimGrid" for further information) > [PM0:Test:(1) 0.000000] [jmsg/INFO] Load of collocated VMs fluctuate between 0 and 90% in order to create a starvation issue and see whether it impacts or not the migration time > [PM0:Test:(1) 0.000000] [jmsg/INFO] Round trip of VM1 (load 90%) > [PM0:Test:(1) 0.000000] [jmsg/INFO] - Launch migration from PM0 to PM1 > [PM0:Test:(1) 0.000000] [jmsg/INFO] Start migration of VM vm0 to PM1 > [PM0:Test:(1) 0.000000] [jmsg/INFO] currentLoad:90/ramSize:2048/dpIntensity:70/remaining:8.10E+11 > [PM0:Test:(1) 35.084170] [jmsg/INFO] End of migration of VM vm0 to node PM1 > [PM0:Test:(1) 35.084170] [jmsg/INFO] - End of Migration from PM0 to PM1 (duration:35.0841702956701) > [PM0:Test:(1) 35.084170] [jmsg/INFO] - Launch migration from PM1 to PM0 > [PM0:Test:(1) 35.084170] [jmsg/INFO] Start migration of VM vm0 to PM0 > [PM0:Test:(1) 35.084170] [jmsg/INFO] currentLoad:90/ramSize:2048/dpIntensity:70/remaining:6.22E+11 > [PM0:Test:(1) 67.551019] [jmsg/INFO] End of migration of VM vm0 to node PM0 > [PM0:Test:(1) 67.551019] [jmsg/INFO] - End of Migration from PM1 to PM0 (duration:32.46684874546391) > [PM0:Test:(1) 67.551019] [jmsg/INFO] > [PM0:Test:(1) 67.551019] [jmsg/INFO] > [PM0:Test:(1) 67.551019] [jmsg/INFO] Round trip of VM1 (load 80%) > [PM0:Test:(1) 67.551019] [jmsg/INFO] - Launch migration from PM0 to PM1 > [PM0:Test:(1) 67.551019] [jmsg/INFO] Start migration of VM vm0 to PM1 > [PM0:Test:(1) 67.551019] [jmsg/INFO] currentLoad:80/ramSize:2048/dpIntensity:70/remaining:4.65E+11 > [PM0:Test:(1) 102.635189] [jmsg/INFO] End of migration of VM vm0 to node PM1 > [PM0:Test:(1) 102.635189] [jmsg/INFO] - End of Migration from PM0 to PM1 (duration:35.08417029567006) > [PM0:Test:(1) 102.635189] [jmsg/INFO] - Launch migration from PM1 to PM0 > [PM0:Test:(1) 102.635189] [jmsg/INFO] Start migration of VM vm0 to PM0 > [PM0:Test:(1) 102.635189] [jmsg/INFO] currentLoad:80/ramSize:2048/dpIntensity:70/remaining:2.77E+11 > [PM0:Test:(1) 135.102038] [jmsg/INFO] End of migration of VM vm0 to node PM0 > [PM0:Test:(1) 135.102038] [jmsg/INFO] - End of Migration from PM1 to PM0 (duration:32.46684874546395) > [PM0:Test:(1) 135.102038] [jmsg/INFO] Forcefully destroy VMs > [135.102038] [jmsg/INFO] MSG_main finished; Cleaning up the simulation... SimGrid-3.14.159/examples/java/cloud/migration/Daemon.java000644 001750 001750 00000002233 13031031276 025000 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package cloud.migration; import org.simgrid.msg.*; import org.simgrid.msg.Process; public class Daemon extends Process { private Task currentTask; public Daemon(VM vm, int load) { super((Host)vm,"Daemon"); currentTask = new Task(this.getHost().getName()+"-daemon-0", this.getHost().getSpeed()*100, 0); } public void main(String[] args) throws MsgException { int i = 1; while(!Main.isEndOfTest()) { // TODO the binding is not yet available try { currentTask.execute(); } catch (HostFailureException e) { e.printStackTrace(); } catch (TaskCancelledException e) { System.out.println("task cancelled"); suspend(); // Suspend the process } currentTask = new Task(this.getHost().getName()+"-daemon-"+(i++), this.getHost().getSpeed()*100, 0); } } public double getRemaining(){ return this.currentTask.getFlopsAmount(); } } SimGrid-3.14.159/examples/java/cloud/migration/README000644 001750 001750 00000000617 13031031306 023610 0ustar00mquinsonmquinson000000 000000 There are two tests: - The first one aims at validating the migration in presence of consolidated VMs - The second one aims at validating the robustness of the migration process (the SRC node or the DST node are turned off during the migration process) To switch between the first and the second tests, you should instrument Main.java (sorry for that :( ) Adsein - Thu Oct 9 18:25:39 CEST 2014 SimGrid-3.14.159/examples/java/cloud/migration/Test.java000644 001750 001750 00000006542 13031031276 024523 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package cloud.migration; import java.util.ArrayList; import java.util.List; import org.simgrid.msg.*; import org.simgrid.msg.Process; public class Test extends Process{ Test(String hostname, String name) throws HostNotFoundException, NativeException { super(hostname, name); } public void doMigration(VM vm, Host src, Host dst) throws HostFailureException{ Msg.info(" - Launch migration from "+ src.getName() +" to " + dst.getName()); double startTime = Msg.getClock(); vm.migrate(dst); double endTime = Msg.getClock(); Msg.info(" - End of Migration from "+ src.getName() +" to " + dst.getName()+ " (duration:" + (endTime-startTime)+")"); } public void main(String[] strings) throws MsgException { Host host0 = null; Host host1 = null; try { /* get hosts 1 and 2*/ host0 = Host.getByName("PM0"); host1 = Host.getByName("PM1"); }catch (HostNotFoundException e) { e.printStackTrace(); } List vms = new ArrayList<>(); /* Create VM1 */ int dpRate = 70; int load1 = 90; int load2 = 80; Msg.info("This example evaluates the migration time of a VM in presence of collocated VMs on the source and " + "the dest nodes"); Msg.info("The migrated VM has a memory intensity rate of 70% of the network BW and a cpu load of 90% \" " +"(see cloudcom 2013 paper \"Adding a Live Migration Model Into SimGrid\" for further information) "); Msg.info("Load of collocated VMs fluctuate between 0 and 90% in order to create a starvation issue and see " + "whether it impacts or not the migration time"); XVM vm1 = new XVM(host0, "vm0", 2048, // Ramsize, 125, // Net bandwidth, dpRate // Memory intensity ); vms.add(vm1); vm1.start(); /* Collocated VMs */ int collocatedSrc = 6; int[] vmSrcLoad = { 80, 0, 90, 40, 30, 90, }; XVM tmp; for (int i=1 ; i<= collocatedSrc ; i++){ tmp = new XVM(host0, "vm"+i, 2048, // Ramsize, 125, // Net bandwidth, dpRate // Memory intensity ); vms.add(tmp); tmp.start(); tmp.setLoad(vmSrcLoad[i-1]); } int collocatedDst = 6; int[] vmDstLoad = { 0, 40, 90, 100, 0, 80, }; for (int i=1 ; i <= collocatedDst ; i++){ tmp = new XVM(host1, "vm"+(i+collocatedSrc), 2048, // Ramsize, 125, // Net bandwidth, dpRate // Memory intensity ); vms.add(tmp); tmp.start(); tmp.setLoad(vmDstLoad[i-1]); } Msg.info("Round trip of VM1 (load "+load1+"%)"); vm1.setLoad(load1); doMigration(vm1, host0, host1); doMigration(vm1, host1, host0); Msg.info(""); Msg.info(""); Msg.info("Round trip of VM1 (load "+load2+"%)"); vm1.setLoad(load2); doMigration(vm1, host0, host1); doMigration(vm1, host1, host0); Main.setEndOfTest(); Msg.info("Forcefully destroy VMs"); for (VM vm: vms) vm.destroy(); } } SimGrid-3.14.159/examples/java/cloud/migration/TestHostOnOff.java000644 001750 001750 00000010454 13031031276 026306 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package cloud.migration; import org.simgrid.msg.*; import org.simgrid.msg.Process; // This test aims at validating that the migration process is robust in face of host turning off either on the SRC // node or on the DST node. public class TestHostOnOff extends Process{ protected Host host0 = null; protected Host host1 = null; protected Host host2 = null; TestHostOnOff(String hostname, String name) throws HostNotFoundException, NativeException { super(hostname, name); } public void main(String[] strings) throws MsgException { /* get hosts 1 and 2*/ try { host0 = Host.getByName("PM0"); host1 = Host.getByName("PM1"); host2 = Host.getByName("PM2"); }catch (HostNotFoundException e) { Msg.error("You are trying to use a non existing node:" + e.getMessage()); } // Robustness on the SRC node for (int i =100 ; i < 55000 ; i+=100) testVMMigrate(host1, i); // Robustness on the DST node for (int i =0 ; i < 55000 ; i++) testVMMigrate(host2, i); /* End of Tests */ Msg.info("Nor more tests, Bye Bye !"); Main.setEndOfTest(); } public void testVMMigrate (Host hostToKill, long killAt) throws MsgException { Msg.info("**** **** **** ***** ***** Test Migrate with host shutdown ***** ***** **** **** ****"); Msg.info("Turn on one host, assign a VM on this host, launch a process inside the VM, migrate the VM and " + "turn off either the SRC or DST"); host1.off(); host2.off(); host1.on(); host2.on(); // Create VM0 int dpRate = 70; XVM vm0 = null; vm0 = new XVM(host1, "vm0", 2048, // Ramsize, 125, // Net bandwidth, dpRate // Memory intensity ); vm0.start(); vm0.setLoad(90); String[] args = new String[3]; args[0] = "vm0"; args[1] = "PM1"; args[2] = "PM2"; new Process(host1, "Migrate-" + killAt, args) { public void main(String[] args) { Host destHost = null; Host sourceHost = null; try { sourceHost = Host.getByName(args[1]); destHost = Host.getByName(args[2]); } catch (HostNotFoundException e) { Msg.error("You are trying to migrate from/to a non existing node: " + e.getMessage()); e.printStackTrace(); } if (destHost != null && sourceHost.isOn() && destHost.isOn()) { try { Msg.info("Migrate vm "+args[0]+" to node "+destHost.getName()); VM.getVMByName(args[0]).migrate(destHost); } catch (HostFailureException e) { Msg.error("Something occurs during the migration that cannot validate the operation: " + e.getMessage()); e.printStackTrace(); } } } }.start(); // Wait killAt ms before killing thehost Process.sleep(killAt); Msg.info("The migration process should be stopped and we should catch an exception"); hostToKill.off(); Process.sleep(50000); Msg.info("Destroy VMs"); vm0.shutdown(); Process.sleep(20000); } public void testVMShutdownDestroy () throws HostFailureException { Msg.info("**** **** **** ***** ***** Test shutdown a VM ***** ***** **** **** ****"); Msg.info("Turn on host1, assign a VM on host1, launch a process inside the VM, and turn off the vm, " + "and check whether you can reallocate the same VM"); // Create VM0 int dpRate = 70; XVM vm0 = new XVM(host1, "vm0", 2048, // Ramsize, 125, // Net bandwidth, dpRate // Memory intensity ); Msg.info("Start VM0"); vm0.start(); vm0.setLoad(90); Process.sleep(5000); Msg.info("Shutdown VM0"); vm0.shutdown(); Process.sleep(5000); Msg.info("Restart VM0"); vm0 = new XVM(host1, "vm0", 2048, // Ramsize, 125, // Net bandwidth, dpRate // Memory intensity ); vm0.start(); vm0.setLoad(90); Msg.info("You suceed to recreate and restart a VM without generating any exception ! Great the Test is ok"); Process.sleep(5000); vm0.shutdown(); } } SimGrid-3.14.159/examples/java/task/000755 001750 001750 00000000000 13031031277 020576 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/task/priority/000755 001750 001750 00000000000 13031031310 022443 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/task/priority/Main.java000644 001750 001750 00000002034 13031031277 024205 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package task.priority; import org.simgrid.msg.Msg; import org.simgrid.msg.NativeException; /* Demonstrates the use of Task.setPriority to change the computation priority of a task */ public class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws NativeException { Msg.init(args); if(args.length < 2) { Msg.info("Usage : Priority platform_file deployment_file"); Msg.info("example : Priority ../platforms/platform.xml priorityDeployment.xml"); System.exit(1); } /* construct the platform and deploy the application */ Msg.createEnvironment(args[0]); Msg.deployApplication(args[1]); /* execute the simulation. */ Msg.run(); } } SimGrid-3.14.159/examples/java/task/priority/priority.xml000644 001750 001750 00000000615 13031031310 025050 0ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/task/priority/Test.java000644 001750 001750 00000001754 13031031277 024250 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package task.priority; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; public class Test extends Process { public Test(Host host, String name, String[]args) { super(host,name,args); } public void main(String[] args) throws MsgException { double computationAmount = Double.parseDouble(args[0]); double priority = Double.parseDouble(args[1]); Msg.info("Hello! Running a task of size " + computationAmount + " with priority " + priority); Task task = new Task("Task", computationAmount, 0); task.setPriority(priority); task.execute(); Msg.info("Goodbye now!"); } } SimGrid-3.14.159/examples/java/task/priority/task_priority.tesh000644 001750 001750 00000001226 13031031303 026236 0ustar00mquinsonmquinson000000 000000 #! tesh ! output sort 19 $ java -classpath ${classpath:=.} task/priority/Main ${srcdir:=.}/../platforms/small_platform.xml ${srcdir:=.}/task/priority/priority.xml "--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n" > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (1:task.priority.Test@Fafard) Hello! Running a task of size 7.6296E7 with priority 1.0 > [ 0.000000] (2:task.priority.Test@Fafard) Hello! Running a task of size 7.6296E7 with priority 2.0 > [ 1.500000] (2:task.priority.Test@Fafard) Goodbye now! > [ 2.000000] (0:maestro@) MSG_main finished; Cleaning up the simulation... > [ 2.000000] (1:task.priority.Test@Fafard) Goodbye now! SimGrid-3.14.159/examples/java/dht/000755 001750 001750 00000000000 13031031276 020412 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/dht/chord/000755 001750 001750 00000000000 13031031310 021476 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/dht/chord/ChordTask.java000644 001750 001750 00000001474 13031031276 024244 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.chord; import dht.chord.Common; import org.simgrid.msg.Task; public class ChordTask extends Task { private String issuerHostName; private String answerTo; public ChordTask() { this(null,null); } public ChordTask(String issuerHostName, String answerTo) { super(null, Common.COMP_SIZE, Common.COMM_SIZE); this.issuerHostName = issuerHostName; this.answerTo = answerTo; } public String getIssuerHostName(){ return this.issuerHostName; } public String getAnswerTo(){ return this.answerTo; } } SimGrid-3.14.159/examples/java/dht/chord/GetPredecessorAnswerTask.java000644 001750 001750 00000001135 13031031276 027275 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.chord; public class GetPredecessorAnswerTask extends ChordTask { private int answerId; public GetPredecessorAnswerTask(String issuerHostname, String answerTo, int answerId) { super(issuerHostname,answerTo); this.answerId = answerId; } public int getAnswerId(){ return this.answerId; } } SimGrid-3.14.159/examples/java/dht/chord/Main.java000644 001750 001750 00000001534 13031031276 023243 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.chord; import org.simgrid.msg.Msg; class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) { Msg.init(args); if(args.length < 2) { Msg.info("Usage : Chord platform_file deployment_file"); Msg.info("example : Chord ../platforms/platform.xml chord.xml"); System.exit(1); } /* construct the platform and deploy the application */ Msg.createEnvironment(args[0]); Msg.deployApplication(args[1]); /* execute the simulation. */ Msg.run(); } } SimGrid-3.14.159/examples/java/dht/chord/FindSuccessorAnswerTask.java000644 001750 001750 00000001133 13031031276 027127 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.chord; public class FindSuccessorAnswerTask extends ChordTask { private int answerId; public FindSuccessorAnswerTask(String issuerHostname, String answerTo, int answerId) { super(issuerHostname,answerTo); this.answerId = answerId; } public int getAnswerId(){ return this.answerId; } } SimGrid-3.14.159/examples/java/dht/chord/NotifyTask.java000644 001750 001750 00000001110 13031031276 024440 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.chord; public class NotifyTask extends ChordTask { private int requestId; public NotifyTask(String issuerHostname, String answerTo, int requestId) { super(issuerHostname, answerTo); this.requestId = requestId; } public int getRequestId(){ return this.requestId; } } SimGrid-3.14.159/examples/java/dht/chord/FindSuccessorTask.java000644 001750 001750 00000001127 13031031276 025752 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.chord; public class FindSuccessorTask extends ChordTask { private int requestId; public FindSuccessorTask(String issuerHostname, String answerTo, int requestId) { super(issuerHostname, answerTo); this.requestId = requestId; } public int getRequestId(){ return this.requestId; } } SimGrid-3.14.159/examples/java/dht/chord/dht_chord.tesh000644 001750 001750 00000002411 13031031303 024321 0ustar00mquinsonmquinson000000 000000 #! tesh ! output sort 19 $ java -classpath ${classpath:=.} dht/chord/Main ${srcdir:=.}/../platforms/cluster.xml ${srcdir:=.}/dht/chord/chord.xml > [0.000000] [jmsg/INFO] Using regular java threads. > [1046.787586] [jmsg/INFO] MSG_main finished; Cleaning up the simulation... > [node-1.acme.org:dht.chord.Node:(2) 0.000000] [jmsg/INFO] Joining the ring with id 366680 knowing node 42 > [node-2.acme.org:dht.chord.Node:(3) 0.000000] [jmsg/INFO] Joining the ring with id 533744 knowing node 366680 > [node-3.acme.org:dht.chord.Node:(4) 0.000000] [jmsg/INFO] Joining the ring with id 1319738 knowing node 42 > [node-4.acme.org:dht.chord.Node:(5) 0.000000] [jmsg/INFO] Joining the ring with id 16509405 knowing node 366680 > [node-5.acme.org:dht.chord.Node:(6) 0.000000] [jmsg/INFO] Joining the ring with id 10874876 knowing node 533744 > [node-6.acme.org:dht.chord.Node:(7) 0.000000] [jmsg/INFO] Joining the ring with id 16728096 knowing node 1319738 > [node-7.acme.org:dht.chord.Node:(8) 0.000000] [jmsg/INFO] Joining the ring with id 10004760 knowing node 16509405 > [node-8.acme.org:dht.chord.Node:(9) 0.000000] [jmsg/INFO] Joining the ring with id 6518808 knowing node 42 > [node-9.acme.org:dht.chord.Node:(10) 0.000000] [jmsg/INFO] Joining the ring with id 2015253 knowing node 1319738 SimGrid-3.14.159/examples/java/dht/chord/Node.java000644 001750 001750 00000032356 13031031276 023252 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.chord; import org.simgrid.msg.Msg; import org.simgrid.msg.Comm; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; import org.simgrid.msg.TimeoutException; public class Node extends Process { protected int id; protected String mailbox; protected int predId; protected String predMailbox; protected int nextFingerToFix; protected Comm commReceive; ///Last time I changed a finger or my predecessor protected double lastChangeDate; int[] fingers; public Node(Host host, String name, String[] args) { super(host,name,args); } @Override public void main(String[] args) throws MsgException { if (args.length != 2 && args.length != 4) { Msg.info("You need to provide 2 or 4 arguments."); return; } double initTime = Msg.getClock(); int i; boolean joinSuccess; double deadline; double nextStabilizeDate = initTime + Common.PERIODIC_STABILIZE_DELAY; double nextFixFingersDate = initTime + Common.PERIODIC_FIX_FINGERS_DELAY; double nextCheckPredecessorDate = initTime + Common.PERIODIC_CHECK_PREDECESSOR_DELAY; double nextLookupDate = initTime + Common.PERIODIC_LOOKUP_DELAY; mailbox = args[0]; id = Integer.parseInt(args[0]); fingers = new int[Common.NB_BITS]; for (i = 0; i < Common.NB_BITS; i++) { fingers[i] = -1; setFinger(i,this.id); } //First node if (args.length == 2) { deadline = Integer.parseInt(args[1]); create(); joinSuccess = true; } else { int knownId = Integer.valueOf(args[1]); deadline = Integer.valueOf(args[3]); Msg.debug("Hey! Let's join the system with the id " + id + "."); joinSuccess = join(knownId); } if (joinSuccess) { double currentClock = Msg.getClock(); while (currentClock < (initTime + deadline) && currentClock < Common.MAX_SIMULATION_TIME) { if (commReceive == null) { commReceive = Task.irecv(this.mailbox); } try { if (!commReceive.test()) { if (currentClock >= nextStabilizeDate) { stabilize(); nextStabilizeDate = Msg.getClock() + Common.PERIODIC_STABILIZE_DELAY; } else if (currentClock >= nextFixFingersDate) { fixFingers(); nextFixFingersDate = Msg.getClock() + Common.PERIODIC_FIX_FINGERS_DELAY; } else if (currentClock >= nextCheckPredecessorDate) { this.checkPredecessor(); nextCheckPredecessorDate = Msg.getClock() + Common.PERIODIC_CHECK_PREDECESSOR_DELAY; } else if (currentClock >= nextLookupDate) { this.randomLookup(); nextLookupDate = Msg.getClock() + Common.PERIODIC_LOOKUP_DELAY; } else { waitFor(5); } currentClock = Msg.getClock(); } else { handleTask(commReceive.getTask()); currentClock = Msg.getClock(); commReceive = null; } } catch (Exception e) { currentClock = Msg.getClock(); commReceive = null; } } leave(); if (commReceive != null) { commReceive = null; } } else { Msg.info("I couldn't join the ring"); } } void handleTask(Task task) { if (task instanceof FindSuccessorTask) { FindSuccessorTask fTask = (FindSuccessorTask)task; Msg.debug("Receiving a 'Find Successor' request from " + fTask.getIssuerHostName() + " for id " + fTask.getRequestId()); // is my successor the successor? if (isInInterval(fTask.getRequestId(), this.id + 1, fingers[0])) { Msg.debug("Send the request to " + fTask.getAnswerTo() + " with answer " + fingers[0]); FindSuccessorAnswerTask answer = new FindSuccessorAnswerTask(getHost().getName(), mailbox, fingers[0]); answer.dsend(fTask.getAnswerTo()); } else { // otherwise, forward the request to the closest preceding finger in my table int closest = closestPrecedingNode(fTask.getRequestId()); Msg.debug("Forward the request to " + closest); fTask.dsend(Integer.toString(closest)); } } else if (task instanceof GetPredecessorTask) { GetPredecessorTask gTask = (GetPredecessorTask)(task); Msg.debug("Receiving a 'Get Predecessor' request from " + gTask.getIssuerHostName()); GetPredecessorAnswerTask answer = new GetPredecessorAnswerTask(getHost().getName(), mailbox, predId); answer.dsend(gTask.getAnswerTo()); } else if (task instanceof NotifyTask) { NotifyTask nTask = (NotifyTask)task; notify(nTask.getRequestId()); } else { Msg.debug("Ignoring unexpected task of type:" + task); } } void leave() { Msg.debug("Well Guys! I Think it's time for me to quit ;)"); quitNotify(1); //Notify my successor quitNotify(-1); //Notify my predecessor. } /** * @brief Notifies the successor or the predecessor of the current node of the departure * @param to 1 to notify the successor, -1 to notify the predecessor */ static void quitNotify( int to) { //TODO } /** * @brief Initializes the current node as the first one of the system. */ void create() { Msg.debug("Create a new Chord ring..."); setPredecessor(-1); } // Makes the current node join the ring, knowing the id of a node already in the ring boolean join(int knownId) { Msg.info("Joining the ring with id " + this.id + " knowing node " + knownId); setPredecessor(-1); int successorId = remoteFindSuccessor(knownId, this.id); if (successorId == -1) { Msg.info("Cannot join the ring."); } else { setFinger(0, successorId); } return successorId != -1; } void setPredecessor(int predecessorId) { if (predecessorId != predId) { predId = predecessorId; if (predecessorId != -1) { predMailbox = Integer.toString(predId); } lastChangeDate = Msg.getClock(); } } /** * @brief Asks another node its predecessor. * @param askTo the node to ask to * @return the id of its predecessor node, or -1 if the request failed(or if the node does not know its predecessor) */ int remoteGetPredecessor(int askTo) { int predecessorId = -1; boolean stop = false; Msg.debug("Sending a 'Get Predecessor' request to " + askTo); String mailboxTo = Integer.toString(askTo); GetPredecessorTask sendTask = new GetPredecessorTask(getHost().getName(), this.mailbox); try { sendTask.send(mailboxTo, Common.TIMEOUT); try { do { if (commReceive == null) { commReceive = Task.irecv(this.mailbox); } commReceive.waitCompletion(Common.TIMEOUT); Task taskReceived = commReceive.getTask(); if (taskReceived instanceof GetPredecessorAnswerTask) { predecessorId = ((GetPredecessorAnswerTask) taskReceived).getAnswerId(); stop = true; } else { handleTask(taskReceived); } commReceive = null; } while (!stop); } catch (MsgException e) { commReceive = null; stop = true; } } catch (MsgException e) { Msg.debug("Failed to send the Get Predecessor request"); } return predecessorId; } /** * @brief Makes the current node find the successor node of an id. * @param node the current node * @param id the id to find * @return the id of the successor node, or -1 if the request failed */ int findSuccessor(int id) { if (isInInterval(id, this.id + 1, fingers[0])) { return fingers[0]; } int closest = this.closestPrecedingNode(id); return remoteFindSuccessor(closest, id); } // Asks another node the successor node of an id. int remoteFindSuccessor(int askTo, int id) { int successor = -1; boolean stop = false; String askToMailbox = Integer.toString(askTo); Task sendTask = new FindSuccessorTask(getHost().getName(), this.mailbox, id); Msg.debug("Sending a 'Find Successor' request to " + askToMailbox + " for id " + id); try { sendTask.send(askToMailbox, Common.TIMEOUT); do { if (commReceive == null) { commReceive = Task.irecv(this.mailbox); } try { commReceive.waitCompletion(Common.TIMEOUT); Task task = commReceive.getTask(); if (task instanceof FindSuccessorAnswerTask) { //TODO: Check if this this our answer. FindSuccessorAnswerTask fTask = (FindSuccessorAnswerTask) task; stop = true; successor = fTask.getAnswerId(); } else { handleTask(task); } commReceive = null; } catch (TimeoutException e) { stop = true; commReceive = null; } } while (!stop); } catch (TimeoutException e) { Msg.debug("Failed to send the 'Find Successor' request"); } catch (MsgException e) { Msg.debug("Failed to receive Find Successor"); } return successor; } // This function is called periodically. It checks the immediate successor of the current node. void stabilize() { Msg.debug("Stabilizing node"); int candidateId; int successorId = fingers[0]; if (successorId != this.id){ candidateId = remoteGetPredecessor(successorId); } else { candidateId = predId; } //This node is a candidate to become my new successor if (candidateId != -1 && isInInterval(candidateId, this.id + 1, successorId - 1)) { setFinger(0, candidateId); } if (successorId != this.id) { remoteNotify(successorId, this.id); } } /** * @brief Notifies the current node that its predecessor may have changed. * @param candidate_id the possible new predecessor */ void notify(int predecessorCandidateId) { if (predId == -1 || isInInterval(predecessorCandidateId, predId + 1, this.id - 1 )) { setPredecessor(predecessorCandidateId); } } /** * @brief Notifies a remote node that its predecessor may have changed. * @param notify_id id of the node to notify * @param candidate_id the possible new predecessor */ void remoteNotify(int notifyId, int predecessorCandidateId) { Msg.debug("Sending a 'Notify' request to " + notifyId); Task sentTask = new NotifyTask(getHost().getName(), this.mailbox, predecessorCandidateId); sentTask.dsend(Integer.toString(notifyId)); } // This function is called periodically. // It refreshes the finger table of the current node. void fixFingers() { Msg.debug("Fixing fingers"); int i = this.nextFingerToFix; int successorId = this.findSuccessor(this.id + (int)Math.pow(2,i)); //FIXME: SLOW if (successorId != -1) { if (successorId != fingers[i]) { setFinger(i, successorId); } nextFingerToFix = (i + 1) % Common.NB_BITS; } } // This function is called periodically. // It checks whether the predecessor has failed void checkPredecessor() { //TODO } // Performs a find successor request to a random id. void randomLookup() { int id = 1337; //Msg.info("Making a lookup request for id " + id); findSuccessor(id); } /** * @brief Returns the closest preceding finger of an id with respect to the finger table of the current node. * @param id the id to find * @return the closest preceding finger of that id */ int closestPrecedingNode(int id) { for (int i = Common.NB_BITS - 1; i >= 0; i--) { if (isInInterval(fingers[i], this.id + 1, id - 1)) { return fingers[i]; } } return this.id; } /** * @brief Returns whether an id belongs to the interval [start, end]. * * The parameters are noramlized to make sure they are between 0 and nb_keys - 1). * 1 belongs to [62, 3] * 1 does not belong to [3, 62] * 63 belongs to [62, 3] * 63 does not belong to [3, 62] * 24 belongs to [21, 29] * 24 does not belong to [29, 21] * * @param id id to check * @param start lower bound * @param end upper bound * @return a non-zero value if id in in [start, end] */ static boolean isInInterval(int id, int start, int end) { int normId = normalize(id); int normStart = normalize(start); int normEnd = normalize(end); // make sure end >= start and id >= start if (normEnd < normStart) { normEnd += Common.NB_KEYS; } if (normId < normStart) { normId += Common.NB_KEYS; } return (normId <= normEnd); } /** * @brief Turns an id into an equivalent id in [0, nb_keys). * @param id an id * @return the corresponding normalized id */ static int normalize(int id) { return id & (Common.NB_KEYS - 1); } /** * @brief Sets a finger of the current node. * @param finger_index index of the finger to set (0 to nb_bits - 1) * @param id the id to set for this finger */ void setFinger(int fingerIndex, int id) { if (id != fingers[fingerIndex]) { fingers[fingerIndex] = id; lastChangeDate = Msg.getClock(); } } } SimGrid-3.14.159/examples/java/dht/chord/GetPredecessorTask.java000644 001750 001750 00000000722 13031031276 026116 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.chord; public class GetPredecessorTask extends ChordTask { public GetPredecessorTask(String issuerHostName, String answerTo) { super(issuerHostName, answerTo); } } SimGrid-3.14.159/examples/java/dht/chord/chord.xml000644 001750 001750 00000004020 13031031310 023313 0ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/dht/chord/Common.java000644 001750 001750 00000001610 13031031276 023602 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.chord; public class Common { public static final int COMM_SIZE = 10; public static final int COMP_SIZE = 0; public static final int NB_BITS = 24; public static final int NB_KEYS = 16777216; public static final int TIMEOUT = 50; public static final int MAX_SIMULATION_TIME = 1000; public static final int PERIODIC_STABILIZE_DELAY = 20; public static final int PERIODIC_FIX_FINGERS_DELAY = 120; public static final int PERIODIC_CHECK_PREDECESSOR_DELAY = 120; public static final int PERIODIC_LOOKUP_DELAY = 10; private Common() { throw new IllegalAccessError("Utility class"); } } SimGrid-3.14.159/examples/java/dht/kademlia/000755 001750 001750 00000000000 13031031310 022146 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/dht/kademlia/RoutingTable.java000644 001750 001750 00000006710 13031031276 025427 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; import java.util.Collections; import java.util.Vector; import org.simgrid.msg.Msg; public class RoutingTable { /* Bucket list */ private Vector buckets; /* Id of the routing table owner */ private int id; public RoutingTable(int id) { this.id = id; buckets = new Vector(); for (int i = 0; i < Common.IDENTIFIER_SIZE + 1; i++) { buckets.add(new Bucket(i)); } } /** * @brief Returns an identifier which is in a specific bucket of a routing table * @param id id of the routing table owner * @param prefix id of the bucket where we want that identifier to be */ public int getIdInPrefix(int id, int prefix) { if (prefix == 0) { return 0; } int identifier = 1; identifier = identifier << (prefix - 1); identifier = identifier ^ id; return identifier; } /* Returns the corresponding node prefix for a given id */ public int getNodePrefix(int id) { for (int j = 0; j < 32; j++) { if ((id >> (32 - 1 - j) & 0x1) != 0) { return 32 - j; } } return 0; } /* Finds the corresponding bucket in a routing table for a given identifier */ public Bucket findBucket(int id) { int xorNumber = id ^ this.id; int prefix = this.getNodePrefix(xorNumber); return buckets.get(prefix); } /* Updates the routing table with a new value. */ public void update(int id) { Bucket bucket = this.findBucket(id); if (bucket.contains(id)) { Msg.debug("Updating " + Integer.toString(id) + " in my routing table"); //If the element is already in the bucket, we update it. bucket.pushToFront(id); } else { Msg.debug("Adding " + id + " to my routing table"); bucket.add(id); if (bucket.size() > Common.BUCKET_SIZE) { //TODO: Ping the least seen guy and remove him if he is offline. } } } /* Returns the closest notes we know to a given id */ public Answer findClosest(int destinationId) { Answer answer = new Answer(destinationId); Bucket bucket = this.findBucket(destinationId); bucket.addToAnswer(answer,destinationId); for (int i = 1; answer.size() < Common.BUCKET_SIZE && ((bucket.getId() - i) >= 0 || (bucket.getId() + i) <= Common.IDENTIFIER_SIZE); i++) { //Check the previous buckets if (bucket.getId() - i >= 0) { Bucket bucketP = this.buckets.get(bucket.getId() - i); bucketP.addToAnswer(answer,destinationId); } //Check the next buckets if (bucket.getId() + i <= Common.IDENTIFIER_SIZE) { Bucket bucketN = this.buckets.get(bucket.getId() + i); bucketN.addToAnswer(answer, destinationId); } } //We sort the list Collections.sort(answer.getNodes()); //We trim the list while (answer.size() > Common.BUCKET_SIZE) { answer.remove(answer.size() - 1); //TODO: Not the best thing. } return answer; } @Override public String toString() { String string = "RoutingTable [ id=" + id + " " ; for (int i = 0; i < buckets.size(); i++) { if (buckets.get(i).size() > 0) { string += buckets.get(i) + " "; } } return string; } } SimGrid-3.14.159/examples/java/dht/kademlia/FindNodeTask.java000644 001750 001750 00000001423 13031031276 025335 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; /** * @brief Find node tasks sent by a node to another "Find Node" task sent by a node to another. Ask him for its closest * nodes from a destination. */ public class FindNodeTask extends KademliaTask { /* Id of the node we are trying to find: the destination */ private int destination; public FindNodeTask(int senderId, int destination) { super(senderId); this.destination = destination; } public int getDestination() { return destination; } } SimGrid-3.14.159/examples/java/dht/kademlia/KademliaTask.java000644 001750 001750 00000001162 13031031276 025356 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; import dht.kademlia.Common; import org.simgrid.msg.Task; public class KademliaTask extends Task { protected int senderId; public KademliaTask(int senderId) { super("kademliatask",Common.COMP_SIZE,Common.COMM_SIZE); this.senderId = senderId; } public int getSenderId() { return senderId; } } SimGrid-3.14.159/examples/java/dht/kademlia/Main.java000644 001750 001750 00000001647 13031031276 023720 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws MsgException { Msg.init(args); if(args.length < 2) { Msg.info("Usage : Kademlia platform_file deployment_file"); Msg.info("example : Kademlia ../platforms/platform.xml kademlia.xml"); System.exit(1); } /* construct the platform and deploy the application */ Msg.createEnvironment(args[0]); Msg.deployApplication(args[1]); /* execute the simulation. */ Msg.run(); } } SimGrid-3.14.159/examples/java/dht/kademlia/PingTask.java000644 001750 001750 00000000640 13031031276 024544 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; public class PingTask extends KademliaTask { public PingTask(int senderId) { super(senderId); } } SimGrid-3.14.159/examples/java/dht/kademlia/Answer.java000644 001750 001750 00000003316 13031031276 024266 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; import java.util.ArrayList; import java.util.Collections; /* Answer to a "FIND_NODE" query. Contains the nodes closest to an id given */ public class Answer { private int destinationId; /* Closest nodes in the answer. */ private ArrayList nodes; public Answer(int destinationId) { this.destinationId = destinationId; nodes = new ArrayList(); } int getDestinationId() { return destinationId; } ArrayList getNodes() { return nodes; } int size() { return nodes.size(); } public void remove(int index) { nodes.remove(index); } public void add(Contact contact) { nodes.add(contact); } /* Merge the contents of this answer with another answer */ public int merge(Answer answer) { int nbAdded = 0; for (Contact c: answer.getNodes()) { if (!nodes.contains(c)) { nbAdded++; nodes.add(c); } } Collections.sort(nodes); //Trim the list while (answer.size() > Common.BUCKET_SIZE) { answer.remove(answer.size() - 1); } return nbAdded; } /* Returns if the destination has been found */ public boolean destinationFound() { if (nodes.size() < 1) { return false; } Contact tail = nodes.get(0); return tail.getDistance() == 0; } @Override public String toString() { return "Answer [destinationId=" + destinationId + ", nodes=" + nodes + "]"; } } SimGrid-3.14.159/examples/java/dht/kademlia/FindNodeAnswerTask.java000644 001750 001750 00000001344 13031031276 026517 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; import dht.kademlia.Answer; public class FindNodeAnswerTask extends KademliaTask { protected int destinationId; protected Answer answer; public FindNodeAnswerTask(int senderId, int destinationId, Answer answer) { super(senderId); this.destinationId = destinationId; this.answer = answer; } public int getDestinationId() { return destinationId; } public Answer getAnswer() { return answer; } } SimGrid-3.14.159/examples/java/dht/kademlia/dht_kademlia.tesh000644 001750 001750 00000002133 13031031303 025442 0ustar00mquinsonmquinson000000 000000 #! tesh ! output sort 19 $ java -classpath ${classpath:=.} dht/kademlia/Main ${srcdir:=.}/../platforms/cluster.xml ${srcdir:=.}/dht/kademlia/kademlia.xml > [0.000000] [jmsg/INFO] Using regular java threads. > [900.000000] [jmsg/INFO] MSG_main finished; Cleaning up the simulation... > [node-0.acme.org:dht.kademlia.Node:(1) 0.000000] [jmsg/INFO] Hi, I'm going to create the network with the id 0! > [node-0.acme.org:dht.kademlia.Node:(1) 900.000000] [jmsg/INFO] 8/8 FIND_NODE have succedded. > [node-1.acme.org:dht.kademlia.Node:(2) 0.000000] [jmsg/INFO] Hi, I'm going to join the network with the id 1! > [node-1.acme.org:dht.kademlia.Node:(2) 900.000000] [jmsg/INFO] 8/8 FIND_NODE have succedded. > [node-2.acme.org:dht.kademlia.Node:(3) 0.000000] [jmsg/INFO] Hi, I'm going to join the network with the id 2! > [node-2.acme.org:dht.kademlia.Node:(3) 900.000000] [jmsg/INFO] 8/8 FIND_NODE have succedded. > [node-3.acme.org:dht.kademlia.Node:(4) 0.000000] [jmsg/INFO] Hi, I'm going to join the network with the id 4! > [node-3.acme.org:dht.kademlia.Node:(4) 900.000000] [jmsg/INFO] 8/8 FIND_NODE have succedded. SimGrid-3.14.159/examples/java/dht/kademlia/PingAnswerTask.java000644 001750 001750 00000000654 13031031276 025731 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; public class PingAnswerTask extends KademliaTask { public PingAnswerTask(int senderId) { super(senderId); } } SimGrid-3.14.159/examples/java/dht/kademlia/kademlia.xml000644 001750 001750 00000001746 13031031310 024447 0ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/dht/kademlia/Bucket.java000644 001750 001750 00000002471 13031031276 024245 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; import java.util.ArrayList; public class Bucket { private ArrayList nodes; private int id; public Bucket(int id) { this.nodes = new ArrayList<>(); this.id = id; } public int getId() { return this.id; } public int size() { return nodes.size(); } public boolean contains(int id) { return nodes.contains(id); } /* Add a node to the front of the bucket */ public void add(int id) { nodes.add(0,id); } /* Push a node to the front of a bucket */ public void pushToFront(int id) { int i = nodes.indexOf(id); nodes.remove(i); nodes.add(0, id); } public int getNode(int id) { return nodes.get(id); } /* Add the content of the bucket into a answer object. */ public void addToAnswer(Answer answer, int destination) { for (int nodeId : this.nodes) { answer.getNodes().add(new Contact(nodeId,nodeId ^ destination)); } } @Override public String toString() { return "Bucket [id= " + id + " nodes=" + nodes + "]"; } } SimGrid-3.14.159/examples/java/dht/kademlia/Node.java000644 001750 001750 00000024343 13031031276 023717 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; import org.simgrid.msg.Host; import org.simgrid.msg.Msg; import org.simgrid.msg.Comm; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; public class Node extends Process { protected int id; protected RoutingTable table; protected int deadline; protected int findNodeSuccedded = 0; protected int findNodeFailed = 0; protected Comm comm; public Node(Host host, String name, String[]args) { super(host,name,args); } @Override public void main(String[] args) throws MsgException { //Check the number of arguments. if (args.length != 2 && args.length != 3) { Msg.info("Wrong argument count."); return; } this.id = Integer.parseInt(args[0]); this.table = new RoutingTable(this.id); if (args.length == 3) { this.deadline = Integer.parseInt(args[2]); Msg.info("Hi, I'm going to join the network with the id " + id + "!"); if (joinNetwork(Integer.parseInt(args[1]))) { this.mainLoop(); } else { Msg.info("I couldn't join the network :("); } } else { this.deadline = Integer.parseInt(args[1]); Msg.info("Hi, I'm going to create the network with the id " + id + "!"); table.update(this.id); this.mainLoop(); } Msg.debug("I'm leaving the network"); Msg.debug("Here is my routing table:" + table); } public void mainLoop() { double next_lookup_time = Msg.getClock() + Common.RANDOM_LOOKUP_INTERVAL; while (Msg.getClock() < this.deadline) { try { if (comm == null) { comm = Task.irecv(Integer.toString(id)); } if (!comm.test()) { if (Msg.getClock() >= next_lookup_time) { randomLookup(); next_lookup_time += Common.RANDOM_LOOKUP_INTERVAL; } else { waitFor(1); } } else { Task task = comm.getTask(); handleTask(task); comm = null; } } catch (Exception e) { } } Msg.info(findNodeSuccedded + "/" + (findNodeSuccedded + findNodeFailed) + " FIND_NODE have succedded."); } /** * @brief Try to make the node join the network * @param idKnown Id of someone we know in the system */ public boolean joinNetwork(int idKnown) { boolean answerGot = false; double timeBegin = Msg.getClock(); Msg.debug("Joining the network knowing " + idKnown); //Add ourselves and the node we know to our routing table table.update(this.id); table.update(idKnown); //Send a "FIND_NODE" to the node we know. sendFindNode(idKnown,this.id); //Wait for the answer. int trials = 0; do { try { if (comm == null) { comm = Task.irecv(Integer.toString(id)); } if (comm != null) { if (!comm.test()) { waitFor(1); } else { Task task = comm.getTask(); if (task instanceof FindNodeAnswerTask) { //Retrieve the node list and ping them FindNodeAnswerTask answerTask = (FindNodeAnswerTask)task; Answer answer = answerTask.getAnswer(); answerGot = true; if (answer.getDestinationId() == this.id) { //Ping everyone in the list for (Contact c : answer.getNodes()) { table.update(c.getId()); } } } else { handleTask(task); } comm = null; } } } catch (Exception ex) { trials++; Msg.info("FIND_NODE failed"); } } while (!answerGot && trials < Common.MAX_JOIN_TRIALS); /* Second step: Send a FIND_NODE in a node in each bucket */ int bucketId = table.findBucket(idKnown).getId(); for (int i = 0; ((bucketId - i) > 0 || (bucketId + i) <= Common.IDENTIFIER_SIZE) && i < Common.JOIN_BUCKETS_QUERIES; i++) { if (bucketId - i > 0) { int idInBucket = table.getIdInPrefix(this.id,bucketId - i); this.findNode(idInBucket,false); } if (bucketId + i <= Common.IDENTIFIER_SIZE) { int idInBucket = table.getIdInPrefix(this.id,bucketId + i); findNode(idInBucket,false); } } Msg.debug("Time spent:" + (Msg.getClock() - timeBegin)); return answerGot; } /* Send a request to find a node in the node's routing table. */ public boolean findNode(int destination, boolean counts) { int queries; int answers; int nodesAdded; boolean destinationFound; int steps = 0; double timeBeginReceive; double timeout; double globalTimeout = Msg.getClock() + Common.FIND_NODE_GLOBAL_TIMEOUT; //Build a list of the closest nodes we already know. Answer nodeList = table.findClosest(destination); Msg.verb("Doing a FIND_NODE on " + destination); do { timeBeginReceive = Msg.getClock(); answers = 0; queries = this.sendFindNodeToBest(nodeList); nodesAdded = 0; timeout = Msg.getClock() + Common.FIND_NODE_TIMEOUT; steps++; do { try { if (comm == null) { comm = Task.irecv(Integer.toString(id)); } if (!comm.test()) { waitFor(1); } else { Task task = comm.getTask(); if (task instanceof FindNodeAnswerTask) { FindNodeAnswerTask answerTask = (FindNodeAnswerTask)task; //Check if we received what we are looking for. if (answerTask.getDestinationId() == destination) { table.update(answerTask.getSenderId()); //Add the answer to our routing table for (Contact c: answerTask.getAnswer().getNodes()) { table.update(c.getId()); } answers++; nodesAdded = nodeList.merge(answerTask.getAnswer()); } else { /* If it's not our answer, we answer to the node that has queried us anyway */ handleTask(task); //Update the timeout if it's not our answer. timeout += Msg.getClock() - timeBeginReceive; timeBeginReceive = Msg.getClock(); } } else { handleTask(task); timeout += Msg.getClock() - timeBeginReceive; timeBeginReceive = Msg.getClock(); } comm = null; } } catch (Exception e) { comm = null; } } while (answers < queries && Msg.getClock() < timeout); destinationFound = nodeList.destinationFound(); } while (!destinationFound && (nodesAdded > 0 || answers == 0) && Msg.getClock() < globalTimeout && steps < Common.MAX_STEPS); if (destinationFound) { if (counts) { findNodeSuccedded++; } Msg.debug("Find node on " + destination + " succedded"); } else { Msg.debug("Find node on " + destination + " failed"); Msg.debug("Queried " + queries + " nodes to find " + destination); Msg.debug(nodeList.toString()); if (counts) { findNodeFailed++; } } return destinationFound; } /** * @brief Sends a "PING" request to a node * @param destination Ping destination id. */ public void ping(int destination) { boolean destinationFound = false; double timeout = Msg.getClock() + Common.PING_TIMEOUT; PingTask pingTask = new PingTask(this.id); /* Sending the ping task */ pingTask.dsend(Integer.toString(destination)); do { try { Task task = Task.receive(Integer.toString(this.id),Common.PING_TIMEOUT); if (task instanceof PingAnswerTask) { PingAnswerTask answerTask = (PingAnswerTask)task; if (answerTask.getSenderId() == destination) { this.table.update(destination); destinationFound = true; } else { handleTask(task); } } else { handleTask(task); } waitFor(1); } catch (Exception ex) { } } while (Msg.getClock() < timeout && !destinationFound); } /** * @brief Sends a "FIND_NODE" request (task) to a node we know. * @param id Id of the node we are querying * @param destination id of the node we are trying to find. */ public void sendFindNode(int id, int destination) { Msg.debug("Sending a FIND_NODE to " + Integer.toString(id) + " to find " + destination ); FindNodeTask task = new FindNodeTask(this.id,destination); task.dsend(Integer.toString(id)); } /** Sends a "FIND_NODE" request to the best "alpha" nodes in a node list */ public int sendFindNodeToBest(Answer nodeList) { int destination = nodeList.getDestinationId(); int i; for (i = 0; i < Common.alpha && i < nodeList.size(); i++) { Contact node = nodeList.getNodes().get(i); if (node.getId() != this.id) { this.sendFindNode(node.getId(),destination); } } return i; } public void randomLookup() { findNode(0,true); } /** * @brief Handles an incomming task * @param task The task we need to handle */ public void handleTask(Task task) { if (task instanceof KademliaTask) { table.update(((KademliaTask) task).getSenderId()); if (task instanceof FindNodeTask) { handleFindNode((FindNodeTask)task); } else if (task instanceof PingTask) { handlePing((PingTask)task); } } } public void handleFindNode(FindNodeTask task) { Msg.debug("Received a FIND_NODE from " + task.getSenderId()); Answer answer = table.findClosest(task.getDestination()); FindNodeAnswerTask taskToSend = new FindNodeAnswerTask(this.id,task.getDestination(),answer); taskToSend.dsend(Integer.toString(task.getSenderId())); } public void handlePing(PingTask task) { Msg.debug("Received a PING from " + task.getSenderId()); PingAnswerTask taskToSend = new PingAnswerTask(this.id); taskToSend.dsend(Integer.toString(task.getSenderId())); } } SimGrid-3.14.159/examples/java/dht/kademlia/Common.java000644 001750 001750 00000002350 13031031276 024254 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; public class Common { /* Common constants used all over the simulation */ public static final int COMM_SIZE = 1; public static final int COMP_SIZE = 0; public static final int RANDOM_LOOKUP_INTERVAL = 100; public static final int alpha = 3; public static final int IDENTIFIER_SIZE = 32; /* Maximum size of the buckets */ public static final int BUCKET_SIZE = 20; /* Maximum number of trials for the "JOIN" request */ public static final int MAX_JOIN_TRIALS = 4; /* Timeout for a "FIND_NODE" request to a node */ public static final int FIND_NODE_TIMEOUT = 10; /* Global timeout for a FIND_NODE request */ public static final int FIND_NODE_GLOBAL_TIMEOUT = 50; /* Timeout for a "PING" request */ public static final int PING_TIMEOUT = 35; public static final int MAX_STEPS = 10; public static final int JOIN_BUCKETS_QUERIES = 1; private Common() { throw new IllegalAccessError("Utility class"); } } SimGrid-3.14.159/examples/java/dht/kademlia/Contact.java000644 001750 001750 00000002233 13031031276 024417 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package dht.kademlia; public class Contact implements Comparable { private int id; private int distance; public Contact(int id, int distance) { this.id = id; this.distance = distance; } public int getId() { return id; } public int getDistance() { return distance; } @Override public boolean equals(Object x) { return (x == null) ? false : x.equals(id) ; } @Override public int hashCode() { int hash = 1; hash = hash * 17 + id; hash = hash * 31 + distance; return hash; } @Override public int compareTo(Object o) { Contact c = (Contact)o; if (distance < c.distance) { return -1; } else if (distance == c.distance) { return 0; } else { return 1; } } @Override public String toString() { return "Contact [id=" + id + ", distance=" + distance + "]"; } }SimGrid-3.14.159/examples/java/trace/000755 001750 001750 00000000000 13031031277 020732 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/trace/pingpong/000755 001750 001750 00000000000 13031031303 022541 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/trace/pingpong/Main.java000644 001750 001750 00000003001 13031031277 024274 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2007, 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package trace.pingpong; import org.simgrid.msg.Msg; import org.simgrid.trace.Trace; import org.simgrid.msg.MsgException; import org.simgrid.msg.NativeException; public class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws MsgException, NativeException { Msg.init(args); if(args.length < 1) { Msg.info("Usage : Main platform_file"); Msg.info("example : Main ../platforms/platform.xml"); System.exit(1); } /* construct the platform and deploy the application */ Msg.createEnvironment(args[0]); new Sender("Jacquelin", "Sender", new String[] {"Boivin", "Tremblay"}).start(); new Receiver ("Boivin", "Receiver", null).start(); new Receiver ("Tremblay", "Receiver", null).start(); /* Initialize some state for the hosts */ Trace.hostStateDeclare ("PM_STATE"); Trace.hostStateDeclareValue ("PM_STATE", "waitingPing", "0 0 1"); Trace.hostStateDeclareValue ("PM_STATE", "sendingPong", "0 1 0"); Trace.hostStateDeclareValue ("PM_STATE", "sendingPing", "0 1 1"); Trace.hostStateDeclareValue ("PM_STATE", "waitingPong", "1 0 0"); /* execute the simulation. */ Msg.run(); } } SimGrid-3.14.159/examples/java/trace/pingpong/trace_pingpong.tesh000644 001750 001750 00000004752 13031031303 026435 0ustar00mquinsonmquinson000000 000000 #! tesh ! output sort 19 $ java -classpath ${classpath:=.} trace/pingpong/Main ${srcdir:=.}/../platforms/small_platform.xml --cfg=tracing:yes --cfg=tracing/filename:simulation.trace --cfg=tracing/platform:yes > [0.000000] [jmsg/INFO] Using regular java threads. > [0.000000] [xbt_cfg/INFO] Configuration change: Set 'tracing' to 'yes' > [0.000000] [xbt_cfg/INFO] Configuration change: Set 'tracing/filename' to 'simulation.trace' > [0.000000] [xbt_cfg/INFO] Configuration change: Set 'tracing/platform' to 'yes' > [3.817809] [jmsg/INFO] MSG_main finished; Cleaning up the simulation... > [Boivin:Receiver:(2) 0.000000] [jmsg/INFO] hello! > [Boivin:Receiver:(2) 0.000000] [jmsg/INFO] try to get a task > [Boivin:Receiver:(2) 1.048882] [jmsg/INFO] Got at time 1.0488818628325232 > [Boivin:Receiver:(2) 1.048882] [jmsg/INFO] Was sent at time 0.0 > [Boivin:Receiver:(2) 1.048882] [jmsg/INFO] Communication time : 1.0488818628325232 > [Boivin:Receiver:(2) 1.048882] [jmsg/INFO] --- bw 9.533962169004269E7 ---- > [Boivin:Receiver:(2) 2.097764] [jmsg/INFO] goodbye! > [Jacquelin:Sender:(1) 0.000000] [jmsg/INFO] hello ! > [Jacquelin:Sender:(1) 0.000000] [jmsg/INFO] host count: 2 > [Jacquelin:Sender:(1) 0.000000] [jmsg/INFO] sender time: 0.0 > [Jacquelin:Sender:(1) 2.097764] [jmsg/INFO] Got at time 2.0977637256650463 > [Jacquelin:Sender:(1) 2.097764] [jmsg/INFO] Was sent at time 0.0 > [Jacquelin:Sender:(1) 2.097764] [jmsg/INFO] Communication time : 2.0977637256650463 > [Jacquelin:Sender:(1) 2.097764] [jmsg/INFO] --- bw 4.7669810845021345E7 ---- > [Jacquelin:Sender:(1) 2.097764] [jmsg/INFO] sender time: 2.0977637256650463 > [Jacquelin:Sender:(1) 3.817809] [jmsg/INFO] Got at time 3.8178087458100927 > [Jacquelin:Sender:(1) 3.817809] [jmsg/INFO] Was sent at time 2.0977637256650463 > [Jacquelin:Sender:(1) 3.817809] [jmsg/INFO] Communication time : 1.7200450201450463 > [Jacquelin:Sender:(1) 3.817809] [jmsg/INFO] --- bw 5.813801315012516E7 ---- > [Jacquelin:Sender:(1) 3.817809] [jmsg/INFO] goodbye! > [Tremblay:Receiver:(3) 0.000000] [jmsg/INFO] hello! > [Tremblay:Receiver:(3) 0.000000] [jmsg/INFO] try to get a task > [Tremblay:Receiver:(3) 2.957786] [jmsg/INFO] Got at time 2.9577862357375695 > [Tremblay:Receiver:(3) 2.957786] [jmsg/INFO] Was sent at time 2.0977637256650463 > [Tremblay:Receiver:(3) 2.957786] [jmsg/INFO] Communication time : 0.8600225100725232 > [Tremblay:Receiver:(3) 2.957786] [jmsg/INFO] --- bw 1.1627602630025032E8 ---- > [Tremblay:Receiver:(3) 3.817809] [jmsg/INFO] goodbye! $ rm -rf simulation.trace SimGrid-3.14.159/examples/java/trace/pingpong/Sender.java000644 001750 001750 00000004543 13031031277 024644 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package trace.pingpong; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; import org.simgrid.msg.NativeException; import org.simgrid.msg.HostNotFoundException; import org.simgrid.trace.Trace; public class Sender extends Process { private static final double commSizeLat = 1; private static final double commSizeBw = 100000000; private static final String PM_STATE = "PM_STATE"; public Sender(String hostname, String name, String[] args) throws HostNotFoundException, NativeException { super(hostname,name,args); } public void main(String[] args) throws MsgException { Msg.info("hello !"); Trace.hostPushState (getHost().getName(), PM_STATE, "sendingPing"); int hostCount = args.length; Msg.info("host count: " + hostCount); String[] mailboxes = new String[hostCount]; double time; double computeDuration = 0; PingPongTask ping; for(int pos = 0; pos < args.length ; pos++) { mailboxes[pos] = Host.getByName(args[pos]).getName(); } for (int pos = 0; pos < hostCount; pos++) { time = Msg.getClock(); Msg.info("sender time: " + time); ping = new PingPongTask("no name",computeDuration,commSizeLat); ping.setTime(time); ping.send(mailboxes[pos]); Trace.hostPushState (getHost().getName(), PM_STATE, "waitingPong"); Task.receive(getHost().getName()); double timeGot = Msg.getClock(); double timeSent = ping.getTime(); double communicationTime; Msg.info("Got at time "+ timeGot); Msg.info("Was sent at time "+timeSent); time = timeSent; communicationTime=timeGot - time; Msg.info("Communication time : " + communicationTime); Msg.info(" --- bw "+ commSizeBw/communicationTime + " ----"); /* Pop the last state (going back to sending ping) */ Trace.hostPopState (getHost().getName(), PM_STATE); } /* Pop the sendingPong state */ Trace.hostPopState (getHost().getName(), PM_STATE); Msg.info("goodbye!"); } } SimGrid-3.14.159/examples/java/trace/pingpong/PingPongTask.java000644 001750 001750 00000001453 13031031277 025765 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package trace.pingpong; import org.simgrid.msg.NativeException; import org.simgrid.msg.Task; public class PingPongTask extends Task { private double timeVal; public PingPongTask() throws NativeException { this.timeVal = 0; } public PingPongTask(String name, double computeDuration, double messageSize) throws NativeException { super(name,computeDuration,messageSize); } public void setTime(double timeVal){ this.timeVal = timeVal; } public double getTime() { return this.timeVal; } } SimGrid-3.14.159/examples/java/trace/pingpong/Receiver.java000644 001750 001750 00000003663 13031031277 025172 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2007, 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package trace.pingpong; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; import org.simgrid.msg.NativeException; import org.simgrid.msg.HostNotFoundException; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.trace.Trace; public class Receiver extends Process { private static final double COMM_SIZE_LAT = 1; private static final double COMM_SIZE_BW = 100000000; public Receiver(String hostname, String name, String[]args) throws HostNotFoundException, NativeException { super(hostname,name,args); } public void main(String[] args) throws MsgException { Msg.info("hello!"); Trace.hostPushState (getHost().getName(), "PM_STATE", "waitingPing"); /* Wait for the ping */ Msg.info("try to get a task"); PingPongTask ping = (PingPongTask)Task.receive(getHost().getName()); double timeGot = Msg.getClock(); double timeSent = ping.getTime(); Msg.info("Got at time "+ timeGot); Msg.info("Was sent at time "+timeSent); double time=timeSent; double communicationTime=timeGot - time; Msg.info("Communication time : " + communicationTime); Msg.info(" --- bw "+ COMM_SIZE_BW/communicationTime + " ----"); /* Send the pong */ Trace.hostPushState (getHost().getName(), "PM_STATE", "sendingPong"); double computeDuration = 0; PingPongTask pong = new PingPongTask("no name",computeDuration,COMM_SIZE_LAT); pong.setTime(time); pong.send(ping.getSource().getName()); /* Pop the two states */ Trace.hostPopState (getHost().getName(), "PM_STATE"); Trace.hostPopState (getHost().getName(), "PM_STATE"); Msg.info("goodbye!"); } } SimGrid-3.14.159/examples/java/app/000755 001750 001750 00000000000 13031031276 020413 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/app/tokenring/000755 001750 001750 00000000000 13031031303 022402 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/app/tokenring/Main.java000644 001750 001750 00000002175 13031031276 024147 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2016. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.tokenring; import org.simgrid.msg.Host; import org.simgrid.msg.HostNotFoundException; import org.simgrid.msg.Msg; import org.simgrid.msg.NativeException; import org.simgrid.msg.Process; class Main { private Main() { /* This is just to prevent anyone from creating an instance of this singleton */ throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws HostNotFoundException, NativeException { Msg.init(args); String platform = "../platforms/small_platform.xml"; if(args.length >= 1) platform = args[0]; Msg.createEnvironment(platform); Host[] hosts = Host.all(); for (int rank = 0; rank < hosts.length; rank++) { Process proc = new RelayRunner(hosts[rank], Integer.toString(rank), null); proc.start(); } Msg.info("Number of hosts '"+hosts.length+"'"); Msg.run(); Msg.info("Simulation time "+Msg.getClock()); } } SimGrid-3.14.159/examples/java/app/tokenring/RelayRunner.java000644 001750 001750 00000004041 13031031276 025523 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2016. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.tokenring; import org.simgrid.msg.Msg; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; import org.simgrid.msg.NativeException; import org.simgrid.msg.Host; public class RelayRunner extends Process { private static final int TASK_COMM_SIZE = 1000000; /* The token is 1MB long*/ public RelayRunner(Host host, String name, String[]args) throws NativeException { super(host,name,args); } /* This is the function executed by this kind of processes */ @Override public void main(String[] args) throws MsgException { // In this example, the processes are given numerical names: "0", "1", "2", and so on int rank = Integer.parseInt(this.getName()); if (rank == 0) { /* The root (rank 0) first sends the token then waits to receive it back */ String mailbox = "1"; Task token = new Task("Token", 0/* no computation associated*/ , TASK_COMM_SIZE ); Msg.info("Host '"+rank+"' send '"+token.getName()+"' to Host '"+mailbox+"'"); token.send(mailbox); token = Task.receive(this.getName()); // Get a message from the mailbox having the same name as the current processor Msg.info("Host '"+rank+"' received '"+token.getName()+"'"); } else { /* The others processes receive on their name (coming from their left neighbor -- rank-1) * and send to their right neighbor (rank+1) */ Task token = Task.receive(this.getName()); Msg.info("Host '"+rank+"' received '"+token.getName()+"'"); String mailbox = Integer.toString(rank+1); if (rank+1 == Host.getCount()) { /* The last process has no right neighbor, so it sends the token back to rank 0 */ mailbox = "0"; } Msg.info("Host '"+rank+"' send '"+token.getName()+"' to Host '"+mailbox+"'"); token.send(mailbox); } } }SimGrid-3.14.159/examples/java/app/tokenring/app_tokenring.tesh000644 001750 001750 00000024533 13031031303 026136 0ustar00mquinsonmquinson000000 000000 #! ./tesh $ java -classpath ${classpath:=.} app/tokenring/Main ${srcdir:=.}/../platforms/routing_cluster.xml '--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n' > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (0:maestro@) Number of hosts '6' > [ 0.000000] (1:0@host1) Host '0' send 'Token' to Host '1' > [ 0.017354] (2:1@host2) Host '1' received 'Token' > [ 0.017354] (2:1@host2) Host '1' send 'Token' to Host '2' > [ 0.035121] (3:2@host3) Host '2' received 'Token' > [ 0.035121] (3:2@host3) Host '2' send 'Token' to Host '3' > [ 0.065898] (4:3@host4) Host '3' received 'Token' > [ 0.065898] (4:3@host4) Host '3' send 'Token' to Host '4' > [ 0.083252] (5:4@host5) Host '4' received 'Token' > [ 0.083252] (5:4@host5) Host '4' send 'Token' to Host '5' > [ 0.101019] (6:5@host6) Host '5' received 'Token' > [ 0.101019] (6:5@host6) Host '5' send 'Token' to Host '0' > [ 0.131796] (1:0@host1) Host '0' received 'Token' > [ 0.131796] (0:maestro@) MSG_main finished; Cleaning up the simulation... > [ 0.131796] (0:maestro@) Simulation time 0.13179602061855672 $ java -classpath ${classpath:=.} app/tokenring/Main ${srcdir:=.}/../platforms/two_peers.xml '--log=root.fmt:[%12.6r]%e(%i:%P@%h)%e%m%n' > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (0:maestro@) Number of hosts '2' > [ 0.000000] (1:0@100030591) Host '0' send 'Token' to Host '1' > [ 0.637910] (2:1@100036570) Host '1' received 'Token' > [ 0.637910] (2:1@100036570) Host '1' send 'Token' to Host '0' > [ 1.275820] (1:0@100030591) Host '0' received 'Token' > [ 1.275820] (0:maestro@) MSG_main finished; Cleaning up the simulation... > [ 1.275820] (0:maestro@) Simulation time 1.2758201322136908 $ java -classpath ${classpath:=.} app/tokenring/Main ${srcdir:=.}/../platforms/meta_cluster.xml '--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n' > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (0:maestro@) Number of hosts '60' > [ 0.000000] (1:0@host-1.cluster1) Host '0' send 'Token' to Host '1' > [ 0.030364] (2:1@host-1.cluster2) Host '1' received 'Token' > [ 0.030364] (2:1@host-1.cluster2) Host '1' send 'Token' to Host '2' > [ 0.060729] (3:2@host-10.cluster1) Host '2' received 'Token' > [ 0.060729] (3:2@host-10.cluster1) Host '2' send 'Token' to Host '3' > [ 0.091093] (4:3@host-10.cluster2) Host '3' received 'Token' > [ 0.091093] (4:3@host-10.cluster2) Host '3' send 'Token' to Host '4' > [ 0.121458] (5:4@host-11.cluster1) Host '4' received 'Token' > [ 0.121458] (5:4@host-11.cluster1) Host '4' send 'Token' to Host '5' > [ 0.151822] (6:5@host-11.cluster2) Host '5' received 'Token' > [ 0.151822] (6:5@host-11.cluster2) Host '5' send 'Token' to Host '6' > [ 0.182187] (7:6@host-12.cluster1) Host '6' received 'Token' > [ 0.182187] (7:6@host-12.cluster1) Host '6' send 'Token' to Host '7' > [ 0.212551] (8:7@host-12.cluster2) Host '7' received 'Token' > [ 0.212551] (8:7@host-12.cluster2) Host '7' send 'Token' to Host '8' > [ 0.242915] (9:8@host-13.cluster1) Host '8' received 'Token' > [ 0.242915] (9:8@host-13.cluster1) Host '8' send 'Token' to Host '9' > [ 0.273280] (10:9@host-13.cluster2) Host '9' received 'Token' > [ 0.273280] (10:9@host-13.cluster2) Host '9' send 'Token' to Host '10' > [ 0.303644] (11:10@host-14.cluster1) Host '10' received 'Token' > [ 0.303644] (11:10@host-14.cluster1) Host '10' send 'Token' to Host '11' > [ 0.334009] (12:11@host-14.cluster2) Host '11' received 'Token' > [ 0.334009] (12:11@host-14.cluster2) Host '11' send 'Token' to Host '12' > [ 0.364373] (13:12@host-15.cluster1) Host '12' received 'Token' > [ 0.364373] (13:12@host-15.cluster1) Host '12' send 'Token' to Host '13' > [ 0.394737] (14:13@host-15.cluster2) Host '13' received 'Token' > [ 0.394737] (14:13@host-15.cluster2) Host '13' send 'Token' to Host '14' > [ 0.425102] (15:14@host-16.cluster1) Host '14' received 'Token' > [ 0.425102] (15:14@host-16.cluster1) Host '14' send 'Token' to Host '15' > [ 0.455466] (16:15@host-16.cluster2) Host '15' received 'Token' > [ 0.455466] (16:15@host-16.cluster2) Host '15' send 'Token' to Host '16' > [ 0.485831] (17:16@host-17.cluster1) Host '16' received 'Token' > [ 0.485831] (17:16@host-17.cluster1) Host '16' send 'Token' to Host '17' > [ 0.516195] (18:17@host-17.cluster2) Host '17' received 'Token' > [ 0.516195] (18:17@host-17.cluster2) Host '17' send 'Token' to Host '18' > [ 0.546560] (19:18@host-18.cluster1) Host '18' received 'Token' > [ 0.546560] (19:18@host-18.cluster1) Host '18' send 'Token' to Host '19' > [ 0.576924] (20:19@host-18.cluster2) Host '19' received 'Token' > [ 0.576924] (20:19@host-18.cluster2) Host '19' send 'Token' to Host '20' > [ 0.607288] (21:20@host-19.cluster1) Host '20' received 'Token' > [ 0.607288] (21:20@host-19.cluster1) Host '20' send 'Token' to Host '21' > [ 0.637653] (22:21@host-19.cluster2) Host '21' received 'Token' > [ 0.637653] (22:21@host-19.cluster2) Host '21' send 'Token' to Host '22' > [ 0.668017] (23:22@host-2.cluster1) Host '22' received 'Token' > [ 0.668017] (23:22@host-2.cluster1) Host '22' send 'Token' to Host '23' > [ 0.698382] (24:23@host-2.cluster2) Host '23' received 'Token' > [ 0.698382] (24:23@host-2.cluster2) Host '23' send 'Token' to Host '24' > [ 0.728746] (25:24@host-20.cluster1) Host '24' received 'Token' > [ 0.728746] (25:24@host-20.cluster1) Host '24' send 'Token' to Host '25' > [ 0.759111] (26:25@host-20.cluster2) Host '25' received 'Token' > [ 0.759111] (26:25@host-20.cluster2) Host '25' send 'Token' to Host '26' > [ 0.789475] (27:26@host-21.cluster1) Host '26' received 'Token' > [ 0.789475] (27:26@host-21.cluster1) Host '26' send 'Token' to Host '27' > [ 0.819839] (28:27@host-21.cluster2) Host '27' received 'Token' > [ 0.819839] (28:27@host-21.cluster2) Host '27' send 'Token' to Host '28' > [ 0.850204] (29:28@host-22.cluster1) Host '28' received 'Token' > [ 0.850204] (29:28@host-22.cluster1) Host '28' send 'Token' to Host '29' > [ 0.880568] (30:29@host-22.cluster2) Host '29' received 'Token' > [ 0.880568] (30:29@host-22.cluster2) Host '29' send 'Token' to Host '30' > [ 0.910933] (31:30@host-23.cluster1) Host '30' received 'Token' > [ 0.910933] (31:30@host-23.cluster1) Host '30' send 'Token' to Host '31' > [ 0.941297] (32:31@host-23.cluster2) Host '31' received 'Token' > [ 0.941297] (32:31@host-23.cluster2) Host '31' send 'Token' to Host '32' > [ 0.971662] (33:32@host-24.cluster1) Host '32' received 'Token' > [ 0.971662] (33:32@host-24.cluster1) Host '32' send 'Token' to Host '33' > [ 1.002026] (34:33@host-24.cluster2) Host '33' received 'Token' > [ 1.002026] (34:33@host-24.cluster2) Host '33' send 'Token' to Host '34' > [ 1.032390] (35:34@host-25.cluster1) Host '34' received 'Token' > [ 1.032390] (35:34@host-25.cluster1) Host '34' send 'Token' to Host '35' > [ 1.062755] (36:35@host-25.cluster2) Host '35' received 'Token' > [ 1.062755] (36:35@host-25.cluster2) Host '35' send 'Token' to Host '36' > [ 1.093119] (37:36@host-26.cluster1) Host '36' received 'Token' > [ 1.093119] (37:36@host-26.cluster1) Host '36' send 'Token' to Host '37' > [ 1.123484] (38:37@host-26.cluster2) Host '37' received 'Token' > [ 1.123484] (38:37@host-26.cluster2) Host '37' send 'Token' to Host '38' > [ 1.153848] (39:38@host-27.cluster1) Host '38' received 'Token' > [ 1.153848] (39:38@host-27.cluster1) Host '38' send 'Token' to Host '39' > [ 1.184212] (40:39@host-27.cluster2) Host '39' received 'Token' > [ 1.184212] (40:39@host-27.cluster2) Host '39' send 'Token' to Host '40' > [ 1.214577] (41:40@host-28.cluster1) Host '40' received 'Token' > [ 1.214577] (41:40@host-28.cluster1) Host '40' send 'Token' to Host '41' > [ 1.244941] (42:41@host-28.cluster2) Host '41' received 'Token' > [ 1.244941] (42:41@host-28.cluster2) Host '41' send 'Token' to Host '42' > [ 1.275306] (43:42@host-29.cluster1) Host '42' received 'Token' > [ 1.275306] (43:42@host-29.cluster1) Host '42' send 'Token' to Host '43' > [ 1.305670] (44:43@host-29.cluster2) Host '43' received 'Token' > [ 1.305670] (44:43@host-29.cluster2) Host '43' send 'Token' to Host '44' > [ 1.336035] (45:44@host-3.cluster1) Host '44' received 'Token' > [ 1.336035] (45:44@host-3.cluster1) Host '44' send 'Token' to Host '45' > [ 1.366399] (46:45@host-3.cluster2) Host '45' received 'Token' > [ 1.366399] (46:45@host-3.cluster2) Host '45' send 'Token' to Host '46' > [ 1.396763] (47:46@host-30.cluster1) Host '46' received 'Token' > [ 1.396763] (47:46@host-30.cluster1) Host '46' send 'Token' to Host '47' > [ 1.427128] (48:47@host-30.cluster2) Host '47' received 'Token' > [ 1.427128] (48:47@host-30.cluster2) Host '47' send 'Token' to Host '48' > [ 1.457492] (49:48@host-4.cluster1) Host '48' received 'Token' > [ 1.457492] (49:48@host-4.cluster1) Host '48' send 'Token' to Host '49' > [ 1.487857] (50:49@host-4.cluster2) Host '49' received 'Token' > [ 1.487857] (50:49@host-4.cluster2) Host '49' send 'Token' to Host '50' > [ 1.518221] (51:50@host-5.cluster1) Host '50' received 'Token' > [ 1.518221] (51:50@host-5.cluster1) Host '50' send 'Token' to Host '51' > [ 1.548586] (52:51@host-5.cluster2) Host '51' received 'Token' > [ 1.548586] (52:51@host-5.cluster2) Host '51' send 'Token' to Host '52' > [ 1.578950] (53:52@host-6.cluster1) Host '52' received 'Token' > [ 1.578950] (53:52@host-6.cluster1) Host '52' send 'Token' to Host '53' > [ 1.609314] (54:53@host-6.cluster2) Host '53' received 'Token' > [ 1.609314] (54:53@host-6.cluster2) Host '53' send 'Token' to Host '54' > [ 1.639679] (55:54@host-7.cluster1) Host '54' received 'Token' > [ 1.639679] (55:54@host-7.cluster1) Host '54' send 'Token' to Host '55' > [ 1.670043] (56:55@host-7.cluster2) Host '55' received 'Token' > [ 1.670043] (56:55@host-7.cluster2) Host '55' send 'Token' to Host '56' > [ 1.700408] (57:56@host-8.cluster1) Host '56' received 'Token' > [ 1.700408] (57:56@host-8.cluster1) Host '56' send 'Token' to Host '57' > [ 1.730772] (58:57@host-8.cluster2) Host '57' received 'Token' > [ 1.730772] (58:57@host-8.cluster2) Host '57' send 'Token' to Host '58' > [ 1.761137] (59:58@host-9.cluster1) Host '58' received 'Token' > [ 1.761137] (59:58@host-9.cluster1) Host '58' send 'Token' to Host '59' > [ 1.791501] (60:59@host-9.cluster2) Host '59' received 'Token' > [ 1.791501] (60:59@host-9.cluster2) Host '59' send 'Token' to Host '0' > [ 1.821865] (1:0@host-1.cluster1) Host '0' received 'Token' > [ 1.821865] (0:maestro@) MSG_main finished; Cleaning up the simulation... > [ 1.821865] (0:maestro@) Simulation time 1.8218653608247406 SimGrid-3.14.159/examples/java/app/bittorrent/000755 001750 001750 00000000000 13031031310 022574 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/app/bittorrent/app_bittorrent.tesh000644 001750 001750 00000003630 13031031303 026521 0ustar00mquinsonmquinson000000 000000 #! tesh ! output sort 19 ! timeout 15 $ java -classpath ${classpath:=.} app/bittorrent/Main ${srcdir:=.}/../platforms/cluster.xml ${srcdir:=.}/app/bittorrent/bittorrent.xml > [0.000000] [jmsg/INFO] Using regular java threads. > [5000.046836] [jmsg/INFO] MSG_main finished; Cleaning up the simulation... > [node-0.acme.org:app.bittorrent.Tracker:(1) 0.000000] [jmsg/INFO] Tracker launched. > [node-0.acme.org:app.bittorrent.Tracker:(1) 3000.000000] [jmsg/INFO] Tracker is leaving > [node-1.acme.org:app.bittorrent.Peer:(2) 0.000000] [jmsg/INFO] Hi, I'm joining the network with id 2 > [node-1.acme.org:app.bittorrent.Peer:(2) 5000.007806] [jmsg/INFO] Here is my current status: 1111111111 > [node-2.acme.org:app.bittorrent.Peer:(3) 0.000000] [jmsg/INFO] Hi, I'm joining the network with id 3 > [node-2.acme.org:app.bittorrent.Peer:(3) 5000.023418] [jmsg/INFO] Here is my current status: 1111111111 > [node-3.acme.org:app.bittorrent.Peer:(4) 0.000000] [jmsg/INFO] Hi, I'm joining the network with id 4 > [node-3.acme.org:app.bittorrent.Peer:(4) 5000.023418] [jmsg/INFO] Here is my current status: 1111111111 > [node-4.acme.org:app.bittorrent.Peer:(5) 0.000000] [jmsg/INFO] Hi, I'm joining the network with id 5 > [node-4.acme.org:app.bittorrent.Peer:(5) 5000.007806] [jmsg/INFO] Here is my current status: 1111111111 > [node-5.acme.org:app.bittorrent.Peer:(6) 0.000000] [jmsg/INFO] Hi, I'm joining the network with id 6 > [node-5.acme.org:app.bittorrent.Peer:(6) 5000.023418] [jmsg/INFO] Here is my current status: 1111111111 > [node-6.acme.org:app.bittorrent.Peer:(7) 0.000000] [jmsg/INFO] Hi, I'm joining the network with id 7 > [node-6.acme.org:app.bittorrent.Peer:(7) 5000.046836] [jmsg/INFO] Here is my current status: 1111111111 > [node-7.acme.org:app.bittorrent.Peer:(8) 0.000000] [jmsg/INFO] Hi, I'm joining the network with id 8 > [node-7.acme.org:app.bittorrent.Peer:(8) 5000.031224] [jmsg/INFO] Here is my current status: 1111111111 SimGrid-3.14.159/examples/java/app/bittorrent/Main.java000644 001750 001750 00000002063 13031031276 024337 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014,2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.bittorrent; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; import org.simgrid.msg.RngStream; class Main{ private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws MsgException { int[] seed = { 12345, 12345, 12345, 12345, 12345, 12345 }; RngStream.setPackageSeed(seed); Msg.init(args); if(args.length < 2) { Msg.info("Usage : Bittorrent platform_file deployment_file"); Msg.info("example : Bittorrent ../platforms/cluster.xml bittorrent.xml"); System.exit(1); } /* construct the platform and deploy the application */ Msg.createEnvironment(args[0]); Msg.deployApplication(args[1]); /* execute the simulation. */ Msg.run(); } } SimGrid-3.14.159/examples/java/app/bittorrent/generate.py000755 001750 001750 00000003133 13031031276 024756 0ustar00mquinsonmquinson000000 000000 #!/usr/bin/env python # Copyright (c) 2013-2014. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. # This script generates a specific deployment file for the Bittorrent example. # It assumes that the platform will be a cluster. # Usage: python generate.py nb_nodes nb_bits end_date percentage # Example: python generate.py 10000 5000 import sys import random if len(sys.argv) != 4: print( "Usage: python generate.py nb_nodes end_date seed_percentage > deployment_file.xml") sys.exit(1) nb_nodes = int(sys.argv[1]) end_date = int(sys.argv[2]) seed_percentage = int(sys.argv[3]) nb_bits = 24 max_id = 2 ** nb_bits - 1 all_ids = [42] sys.stdout.write("\n" "\n" "\n" " \n" % end_date) for i in range(1, nb_nodes): ok = False while not ok: my_id = random.randint(0, max_id) ok = not my_id in all_ids start_date = i * 10 line = " " % ( i, my_id, end_date) if random.randint(0, 100) < seed_percentage: line += "" line += "\n" sys.stdout.write(line) all_ids.append(my_id) sys.stdout.write("") SimGrid-3.14.159/examples/java/app/bittorrent/Peer.java000644 001750 001750 00000053335 13031031276 024356 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.bittorrent; import java.util.ArrayList; import java.util.HashMap; import java.util.Iterator; import java.util.Map.Entry; import org.simgrid.msg.Msg; import org.simgrid.msg.Comm; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.RngStream; import org.simgrid.msg.MsgException; public class Peer extends Process { protected int round = 0; protected double beginReceiveTime; protected double deadline; protected static RngStream stream = new RngStream(); protected int id; protected String mailbox; protected String mailboxTracker; protected String hostname; protected int pieces = 0; protected char[] bitfield = new char[Common.FILE_PIECES]; protected char[][] bitfieldBlocks = new char[Common.FILE_PIECES][Common.PIECES_BLOCKS]; protected short[] piecesCount = new short[Common.FILE_PIECES]; protected int piecesRequested = 0; protected ArrayList currentPieces = new ArrayList<>(); protected int currentPiece = -1; protected HashMap activePeers = new HashMap<>(); protected HashMap peers = new HashMap<>(); protected Comm commReceived = null; public Peer(Host host, String name, String[]args) { super(host,name,args); } @Override public void main(String[] args) throws MsgException { //Check arguments if (args.length != 3 && args.length != 2) { Msg.info("Wrong number of arguments"); } if (args.length == 3) { init(Integer.parseInt(args[0]),true); } else { init(Integer.parseInt(args[0]),false); } //Retrieve the deadline deadline = Double.parseDouble(args[1]); if (deadline < 0) { Msg.info("Wrong deadline supplied"); return; } Msg.info("Hi, I'm joining the network with id " + id); //Getting peer data from the tracker if (getPeersData()) { Msg.debug("Got " + peers.size() + " peers from the tracker"); Msg.debug("Here is my current status: " + getStatus()); beginReceiveTime = Msg.getClock(); if (hasFinished()) { pieces = Common.FILE_PIECES; sendHandshakeAll(); seedLoop(); } else { leechLoop(); seedLoop(); } } else { Msg.info("Couldn't contact the tracker."); } Msg.info("Here is my current status: " + getStatus()); } private void leechLoop() { double nextChokedUpdate = Msg.getClock() + Common.UPDATE_CHOKED_INTERVAL; Msg.debug("Start downloading."); // Send a "handshake" message to all the peers it got(it couldn't have gotten more than 50 peers anyway) sendHandshakeAll(); //Wait for at least one "bitfield" message. waitForPieces(); Msg.debug("Starting main leech loop"); while (Msg.getClock() < deadline && pieces < Common.FILE_PIECES) { if (commReceived == null) { commReceived = Task.irecv(mailbox); } try { if (commReceived.test()) { handleMessage(commReceived.getTask()); commReceived = null; } else { //If the user has a pending interesting if (currentPiece != -1) { sendInterestedToPeers(); } else { if (currentPieces.size() < Common.MAX_PIECES) { updateCurrentPiece(); } } //We don't execute the choke algorithm if we don't already have a piece if (Msg.getClock() >= nextChokedUpdate && pieces > 0) { updateChokedPeers(); nextChokedUpdate += Common.UPDATE_CHOKED_INTERVAL; } else { waitFor(1); } } } catch (MsgException e) { e.printStackTrace(); commReceived = null; } } } private void seedLoop() { double nextChokedUpdate = Msg.getClock() + Common.UPDATE_CHOKED_INTERVAL; Msg.debug("Start seeding."); //start the main seed loop while (Msg.getClock() < deadline) { if (commReceived == null) { commReceived = Task.irecv(mailbox); } try { if (commReceived.test()) { handleMessage(commReceived.getTask()); commReceived = null; } else { if (Msg.getClock() >= nextChokedUpdate) { updateChokedPeers(); //TODO: Change the choked peer algorithm when seeding nextChokedUpdate += Common.UPDATE_CHOKED_INTERVAL; } else { waitFor(1); } } } catch (MsgException e) { commReceived = null; } } } /** * @brief Initialize the various peer data * @param id id of the peer to take in the network * @param seed indicates if the peer is a seed */ private void init(int id, boolean seed) { this.id = id; this.mailbox = Integer.toString(id); this.mailboxTracker = "tracker_" + Integer.toString(id); if (seed) { for (int i = 0; i < bitfield.length; i++) { bitfield[i] = '1'; for (int j = 0; j < bitfieldBlocks[i].length; j++) { bitfieldBlocks[i][j] = '1'; } } } else { for (int i = 0; i < bitfield.length; i++) { bitfield[i] = '0'; for (int j = 0; j < bitfieldBlocks[i].length; j++) { bitfieldBlocks[i][j] = '0' ; } } } this.hostname = getHost().getName(); } private boolean getPeersData() { boolean success = false; boolean sendSuccess = false; double timeout = Msg.getClock() + Common.GET_PEERS_TIMEOUT; //Build the task to send to the tracker TrackerTask taskSend = new TrackerTask(hostname, mailboxTracker, id); while (!sendSuccess && Msg.getClock() < timeout) { try { Msg.debug("Sending a peer request to the tracker."); taskSend.send(Common.TRACKER_MAILBOX,Common.GET_PEERS_TIMEOUT); sendSuccess = true; } catch (MsgException e) { e.printStackTrace(); } } while (!success && Msg.getClock() < timeout) { commReceived = Task.irecv(this.mailboxTracker); try { commReceived.waitCompletion(Common.GET_PEERS_TIMEOUT); if (commReceived.getTask() instanceof TrackerTask) { TrackerTask task = (TrackerTask)commReceived.getTask(); for (Integer peerId: task.peers) { if (peerId != this.id) { peers.put(peerId, new Connection(peerId)); } } success = true; } } catch (MsgException e) { e.printStackTrace(); } commReceived = null; } commReceived = null; return success; } void handleMessage(Task task) { MessageTask message = (MessageTask)task; Connection remotePeer = peers.get(message.peerId); switch (message.type) { case HANDSHAKE: Msg.debug("Received a HANDSHAKE message from " + message.mailbox); //Check if the peer is in our connection list if (remotePeer == null) { peers.put(message.peerId, new Connection(message.peerId)); sendHandshake(message.mailbox); } //Send our bitfield to the pair sendBitfield(message.mailbox); break; case BITFIELD: Msg.debug("Received a BITFIELD message from " + message.peerId + " (" + message.issuerHostname + ")"); //update the pieces list updatePiecesCountFromBitfield(message.bitfield); //Update the current piece if (currentPiece == -1 && pieces < Common.FILE_PIECES && currentPieces.size() < Common.MAX_PIECES) { updateCurrentPiece(); } remotePeer.bitfield = message.bitfield.clone(); break; case INTERESTED: Msg.debug("Received an INTERESTED message from " + message.peerId + " (" + message.issuerHostname + ")"); assert remotePeer != null; remotePeer.interested = true; break; case NOTINTERESTED: Msg.debug("Received a NOTINTERESTED message from " + message.peerId + " (" + message.issuerHostname + ")"); assert remotePeer != null; remotePeer.interested = false; break; case UNCHOKE: Msg.debug("Received an UNCHOKE message from " + message.peerId + "(" + message.issuerHostname + ")"); assert remotePeer != null; remotePeer.chokedDownload = false; activePeers.put(remotePeer.id,remotePeer); sendRequestsToPeer(remotePeer); break; case CHOKE: Msg.debug("Received a CHOKE message from " + message.peerId + " (" + message.issuerHostname + ")"); assert remotePeer != null; remotePeer.chokedDownload = true; activePeers.remove(remotePeer.id); break; case HAVE: if (remotePeer.bitfield == null) { return; } Msg.debug("Received a HAVE message from " + message.peerId + " (" + message.issuerHostname + ")"); assert message.index >= 0 && message.index < Common.FILE_PIECES; assert remotePeer.bitfield != null; remotePeer.bitfield[message.index] = '1'; piecesCount[message.index]++; //Send interested message to the peer if he has what we want if (!remotePeer.amInterested && currentPieces.contains(message.index) ) { remotePeer.amInterested = true; sendInterested(remotePeer.mailbox); } if (currentPieces.contains(message.index)) { int blockIndex = getFirstBlock(message.index); int blockLength = Common.PIECES_BLOCKS - blockIndex ; blockLength = blockLength > Common.BLOCKS_REQUESTED ? Common.BLOCKS_REQUESTED : blockLength; sendRequest(message.mailbox,message.index,blockIndex,blockLength); } break; case REQUEST: assert message.index >= 0 && message.index < Common.FILE_PIECES; if (!remotePeer.chokedUpload) { Msg.debug("Received a REQUEST from " + message.peerId + "(" + message.issuerHostname + ") for " + message.peerId); if (bitfield[message.index] == '1') { sendPiece(message.mailbox,message.index,false,message.blockIndex,message.blockLength); } else { Msg.debug("Received a REQUEST from " + message.peerId + " (" + message.issuerHostname + ") but he is choked" ); } } break; case PIECE: if (message.stalled) { Msg.debug("The received piece " + message.index + " from " + message.peerId + " (" + message.issuerHostname + ") is stalled"); } else { Msg.debug("Received piece " + message.index + " from " + message.peerId + " (" + message.issuerHostname + ")"); if (bitfield[message.index] == '0') { updateBitfieldBlocks(message.index,message.blockIndex,message.blockLength); if (pieceComplete(message.index)) { piecesRequested--; //Removing the piece from our piece list. currentPieces.remove((Object)Integer.valueOf(message.index)); //Setting the fact that we have the piece bitfield[message.index] = '1'; pieces++; Msg.debug("My status is now " + getStatus()); //Sending the information to all the peers we are connected to sendHave(message.index); //sending UNINTERESTED to peers that doesn't have what we want. updateInterestedAfterReceive(); } } else { Msg.debug("However, we already have it."); } } break; } if (remotePeer != null) { remotePeer.addSpeedValue(1 / (Msg.getClock() - beginReceiveTime)); } beginReceiveTime = Msg.getClock(); } void waitForPieces() { boolean finished = false; while (Msg.getClock() < deadline && !finished) { if (commReceived == null) { commReceived = Task.irecv(mailbox); } try { commReceived.waitCompletion(Common.TIMEOUT_MESSAGE); handleMessage(commReceived.getTask()); if (currentPiece != -1) { finished = true; } commReceived = null; } catch (MsgException e) { commReceived = null; } } } private boolean hasFinished() { for (int i = 0; i < bitfield.length; i++) { if (bitfield[i] == '1') { return true; } } return false; } /** * @brief Updates the list of who has a piece from a bitfield * @param bitfield bitfield */ private void updatePiecesCountFromBitfield(char[] bitfield) { for (int i = 0; i < Common.FILE_PIECES; i++) { if (bitfield[i] == '1') { piecesCount[i]++; } } } /** * Update the piece the peer is currently interested in. * There is two cases (as described in "Bittorrent Architecture Protocol", Ryan Toole : * If the peer has less than 3 pieces, he chooses a piece at random. * If the peer has more than pieces, he downloads the pieces that are the less * replicated */ void updateCurrentPiece() { if (currentPieces.size() >= (Common.FILE_PIECES - pieces)) { return; } // if (pieces < 3) { do { currentPiece = stream.randInt(0,Common.FILE_PIECES - 1); } while (!(bitfield[currentPiece] == '0' && !currentPieces.contains(currentPiece))); // } // else { //TODO trivial min algorithm. // } currentPieces.add(currentPiece); Msg.debug("New interested piece: " + currentPiece); assert currentPiece >= 0 && currentPiece < Common.FILE_PIECES; } // Update the list of current choked and unchoked peers, using the choke algorithm private void updateChokedPeers() { round = (round + 1) % 3; if (peers.size() == 0) { return; } //remove a peer from the list Iterator> it = activePeers.entrySet().iterator(); if (it.hasNext()) { Entry e = it.next(); Connection peerChoked = e.getValue(); peerChoked.chokedUpload = true; sendChoked(peerChoked.mailbox); activePeers.remove(e.getKey()); } Connection peerChoosed = null; //Separate the case from when the peer is seeding. if (pieces == Common.FILE_PIECES) { //Find the last unchoked peer. double unchokeTime = deadline + 1; for (Connection connection : peers.values()) { if (connection.lastUnchoke < unchokeTime && connection.interested) { peerChoosed = connection; unchokeTime = connection.lastUnchoke; } } } else { //Random optimistic unchoking if (round == 0) { int j = 0; do { int i = 0; int idChosen = stream.randInt(0,peers.size() - 1); for (Connection connection : peers.values()) { if (i == idChosen) { peerChoosed = connection; break; } i++; } //TODO: Not really the best way ever if (peerChoosed != null && !peerChoosed.interested) { peerChoosed = null; } j++; } while (peerChoosed == null && j < Common.MAXIMUM_PEERS); } else { Connection fastest = null; double fastestSpeed = 0; for (Connection c : peers.values()) { if (c.peerSpeed > fastestSpeed && c.interested && c.chokedUpload) { fastest = c; fastestSpeed = c.peerSpeed; } } peerChoosed = fastest; } } if (peerChoosed != null) { activePeers.put(peerChoosed.id,peerChoosed); peerChoosed.chokedUpload = false; peerChoosed.lastUnchoke = Msg.getClock(); sendUnchoked(peerChoosed.mailbox); } } // Updates our "interested" state about peers: send "not interested" to peers that don't have any more pieces we want. private void updateInterestedAfterReceive() { boolean interested; for (Connection connection : peers.values()) { interested = false; if (connection.amInterested) { for (Integer piece : currentPieces) { if (connection.bitfield[piece] == '1') { interested = true; break; } } if (!interested) { connection.amInterested = false; sendNotInterested(connection.mailbox); } } } } private void updateBitfieldBlocks(int index, int blockIndex, int blockLength) { for (int i = blockIndex; i < (blockIndex + blockLength); i++) { bitfieldBlocks[index][i] = '1'; } } // Returns if a piece is complete in the peer's bitfield. private boolean pieceComplete(int index) { for (int i = 0; i < bitfieldBlocks[index].length; i++) { if (bitfieldBlocks[index][i] == '0') { return false; } } return true; } // Returns the first block of a piece that we don't have. private int getFirstBlock(int piece) { int blockIndex = -1; for (int i = 0; i < Common.PIECES_BLOCKS; i++) { if (bitfieldBlocks[piece][i] == '0') { blockIndex = i; break; } } return blockIndex; } /** * @brief Send request messages to a peer that have unchoked us * @param remotePeer peer data to the peer we want to send the request */ private void sendRequestsToPeer(Connection remotePeer) { if (remotePeer.bitfield == null) { return; } for (Integer piece : currentPieces) { //Getting the block to send. int blockIndex = getFirstBlock(piece); int blockLength = Common.PIECES_BLOCKS - blockIndex ; blockLength = blockLength > Common.BLOCKS_REQUESTED ? Common.BLOCKS_REQUESTED : blockLength; if (remotePeer.bitfield[piece] == '1') { sendRequest(remotePeer.mailbox, piece, blockIndex, blockLength); } } } // Find the peers that have the current interested piece and send them the "interested" message private void sendInterestedToPeers() { if (currentPiece == -1) { return; } for (Connection connection : peers.values()) { if (connection.bitfield != null && connection.bitfield[currentPiece] == '1' && !connection.amInterested) { connection.amInterested = true; MessageTask task = new MessageTask(MessageTask.Type.INTERESTED, hostname, this.mailbox, this.id); task.dsend(connection.mailbox); } } currentPiece = -1; piecesRequested++; } // Send a "interested" message to a peer. private void sendInterested(String mailbox) { MessageTask task = new MessageTask(MessageTask.Type.INTERESTED, hostname, this.mailbox, this.id); task.dsend(mailbox); } /** * @brief Send a "not interested" message to a peer * @param mailbox mailbox destination mailbox */ private void sendNotInterested(String mailbox) { MessageTask task = new MessageTask(MessageTask.Type.NOTINTERESTED, hostname, this.mailbox, this.id); task.dsend(mailbox); } // Send a handshake message to all the peers the peer has. private void sendHandshakeAll() { for (Connection remotePeer : peers.values()) { MessageTask task = new MessageTask(MessageTask.Type.HANDSHAKE, hostname, mailbox, id); task.dsend(remotePeer.mailbox); } } /** * @brief Send a "handshake" message to an user * @param mailbox mailbox where to we send the message */ private void sendHandshake(String mailbox) { Msg.debug("Sending a HANDSHAKE to " + mailbox); MessageTask task = new MessageTask(MessageTask.Type.HANDSHAKE, hostname, this.mailbox, this.id); task.dsend(mailbox); } // Send a "choked" message to a peer private void sendChoked(String mailbox) { Msg.debug("Sending a CHOKE to " + mailbox); MessageTask task = new MessageTask(MessageTask.Type.CHOKE, hostname, this.mailbox, this.id); task.dsend(mailbox); } // Send a "unchoked" message to a peer private void sendUnchoked(String mailbox) { Msg.debug("Sending a UNCHOKE to " + mailbox); MessageTask task = new MessageTask(MessageTask.Type.UNCHOKE, hostname, this.mailbox, this.id); task.dsend(mailbox); } // Send a "HAVE" message to all peers we are connected to private void sendHave(int piece) { Msg.debug("Sending HAVE message to all my peers"); for (Connection remotePeer : peers.values()) { MessageTask task = new MessageTask(MessageTask.Type.HAVE, hostname, this.mailbox, this.id, piece); task.dsend(remotePeer.mailbox); } } // Send a bitfield message to all the peers the peer has. private void sendBitfield(String mailbox) { Msg.debug("Sending a BITFIELD to " + mailbox); MessageTask task = new MessageTask(MessageTask.Type.BITFIELD, hostname, this.mailbox, this.id, this.bitfield); task.dsend(mailbox); } // Send a "request" message to a peer, containing a request for a piece private void sendRequest(String mailbox, int piece, int blockIndex, int blockLength) { Msg.debug("Sending a REQUEST to " + mailbox + " for piece " + piece + " and blocks " + blockIndex + "," + (blockIndex + blockLength)); MessageTask task = new MessageTask(MessageTask.Type.REQUEST, hostname, this.mailbox, this.id, piece, blockIndex, blockLength); task.dsend(mailbox); } // Send a "piece" message to a peer, containing a piece of the file private void sendPiece(String mailbox, int piece, boolean stalled, int blockIndex, int blockLength) { Msg.debug("Sending the PIECE " + piece + " to " + mailbox); MessageTask task = new MessageTask(MessageTask.Type.PIECE, hostname, this.mailbox, this.id, piece, stalled, blockIndex, blockLength); task.dsend(mailbox); } private String getStatus() { String s = ""; for (int i = 0; i < Common.FILE_PIECES; i++) { s = s + bitfield[i]; } return s; } } SimGrid-3.14.159/examples/java/app/bittorrent/TrackerTask.java000644 001750 001750 00000002414 13031031276 025671 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.bittorrent; import java.util.ArrayList; import org.simgrid.msg.Task; /* Task exchanged between the tracker and the peers. */ public class TrackerTask extends Task { public enum Type { REQUEST, ANSWER } protected Type type; protected String hostname; protected String mailbox; protected int peerId; protected int uploaded; protected int downloaded; protected int left; protected double interval; protected ArrayList peers; public TrackerTask(String hostname, String mailbox, int peerId) { this(hostname, mailbox, peerId, 0, 0, Common.FILE_SIZE); } public TrackerTask(String hostname, String mailbox, int peerId, int uploaded, int downloaded, int left) { super("", 0, Common.TRACKER_COMM_SIZE); this.type = Type.REQUEST; this.hostname = hostname; this.mailbox = mailbox; this.peerId = peerId; this.uploaded = uploaded; this.downloaded = downloaded; this.left = left; this.peers = new ArrayList<>(); } } SimGrid-3.14.159/examples/java/app/bittorrent/bittorrent.xml000644 001750 001750 00000003357 13031031310 025522 0ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/app/bittorrent/Connection.java000644 001750 001750 00000003145 13031031276 025554 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.bittorrent; import java.util.Arrays; public class Connection { protected int id; protected char[] bitfield; protected String mailbox; // Indicates if we are interested in something this peer has protected boolean amInterested = false; // Indicates if the peer is interested in one of our pieces protected boolean interested = false; // Indicates if the peer is choked for the current peer protected boolean chokedUpload = true; // Indicates if the peer has choked the current peer protected boolean chokedDownload = true; // Number of messages we have received from the peer protected int messagesCount = 0; protected double peerSpeed = 0; protected double lastUnchoke = 0; public Connection(int id) { this.id = id; this.mailbox = Integer.toString(id); } // Add a new value to the peer speed average public void addSpeedValue(double speed) { peerSpeed = peerSpeed * 0.55 + speed * 0.45; // peerSpeed = (peerSpeed * messagesCount + speed) / (++messagesCount); } @Override public String toString() { return "Connection [id=" + id + ", bitfield=" + Arrays.toString(bitfield) + ", mailbox=" + mailbox + ", amInterested=" + amInterested + ", interested=" + interested + ", chokedUpload=" + chokedUpload + ", chokedDownload=" + chokedDownload + "]"; } } SimGrid-3.14.159/examples/java/app/bittorrent/MessageTask.java000644 001750 001750 00000003774 13031031276 025674 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.bittorrent; import org.simgrid.msg.Task; public class MessageTask extends Task { public enum Type { HANDSHAKE, CHOKE, UNCHOKE, INTERESTED, NOTINTERESTED, HAVE, BITFIELD, REQUEST, PIECE } protected Type type; protected String issuerHostname; protected String mailbox; protected int peerId; protected char[] bitfield; protected int index; protected int blockIndex; protected int blockLength; protected boolean stalled; public MessageTask(Type type, String issuerHostname, String mailbox, int peerId) { this(type,issuerHostname,mailbox,peerId,-1,false,-1,-1); } public MessageTask(Type type, String issuerHostname, String mailbox, int peerId, int index) { this(type,issuerHostname,mailbox,peerId,index,false,-1,-1); } // builds a new bitfield message public MessageTask(Type type, String issuerHostname, String mailbox, int peerId, char[] bitfield) { this(type,issuerHostname,mailbox,peerId,-1,false,-1,-1); this.bitfield = bitfield; } // build a new "request" message public MessageTask(Type type, String issuerHostname, String mailbox, int peerId, int index, int blockIndex, int blockLength) { this(type,issuerHostname,mailbox,peerId,index,false,blockIndex,blockLength); } // build a new "piece" message public MessageTask(Type type, String issuerHostname, String mailbox, int peerId, int index, boolean stalled, int blockIndex, int blockLength) { this.type = type; this.issuerHostname = issuerHostname; this.mailbox = mailbox; this.peerId = peerId; this.index = index; this.stalled = stalled; this.blockIndex = blockIndex; this.blockLength = blockLength; } } SimGrid-3.14.159/examples/java/app/bittorrent/Common.java000644 001750 001750 00000003304 13031031276 024702 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.bittorrent; /* Common constants for use in the simulation */ class Common { public static final String TRACKER_MAILBOX = "tracker_mailbox"; public static final int FILE_SIZE = 5120; public static final int FILE_PIECE_SIZE = 512; public static final int FILE_PIECES = 10; public static final int PIECES_BLOCKS = 5; public static final int BLOCKS_REQUESTED = 2; public static final int PIECE_COMM_SIZE = 1; public static final int MESSAGE_SIZE = 1; /* Information message size */ public static final int MAXIMUM_PEERS = 50; /* Max number of peers sent by the tracker to clients */ public static final int TRACKER_QUERY_INTERVAL = 1000; /* Interval of time where the peer should send a request to the tracker */ public static final double TRACKER_COMM_SIZE = 0.01; /* Communication size for a task to the tracker */ public static final int GET_PEERS_TIMEOUT = 10000; /* Timeout for the get peers data */ public static final int TIMEOUT_MESSAGE = 10; public static final int TRACKER_RECEIVE_TIMEOUT = 10; public static final int MAX_UNCHOKED_PEERS = 4; /* Number of peers that can be unchocked at a given time */ public static final int UPDATE_CHOKED_INTERVAL = 30; /* Interval between each update of the choked peers */ public static final int MAX_PIECES = 1; /* Number of pieces the peer asks for simultaneously */ private Common() { throw new IllegalAccessError("Utility class"); } } SimGrid-3.14.159/examples/java/app/bittorrent/Tracker.java000644 001750 001750 00000004775 13031031276 025062 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.bittorrent; import java.util.ArrayList; import org.simgrid.msg.Msg; import org.simgrid.msg.Comm; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.RngStream; import org.simgrid.msg.MsgException; public class Tracker extends Process { protected RngStream stream; protected ArrayList peersList; protected double deadline; protected Comm commReceived = null; public Tracker(Host host, String name, String[]args) { super(host,name,args); } @Override public void main(String[] args) throws MsgException { if (args.length != 1) { Msg.info("Wrong number of arguments for the tracker."); return; } //Build the RngStream object for randomness stream = new RngStream("tracker"); //Retrieve the end time deadline = Double.parseDouble(args[0]); //Building peers array peersList = new ArrayList<>(); Msg.info("Tracker launched."); while (Msg.getClock() < deadline) { if (commReceived == null) { commReceived = Task.irecv(Common.TRACKER_MAILBOX); } try { if (commReceived.test()) { Task task = commReceived.getTask(); if (task instanceof TrackerTask) { TrackerTask tTask = (TrackerTask)task; //Sending peers to the peer int nbPeers = 0; while (nbPeers < Common.MAXIMUM_PEERS && nbPeers < peersList.size()) { int nextPeer; do { nextPeer = stream.randInt(0, peersList.size() - 1); } while (tTask.peers.contains(peersList.get(nextPeer))); tTask.peers.add(peersList.get(nextPeer)); nbPeers++; } //Adding the peer to our list peersList.add(tTask.peerId); tTask.type = TrackerTask.Type.ANSWER; //Setting the interval tTask.interval = Common.TRACKER_QUERY_INTERVAL; //Sending the task back to the peer tTask.dsend(tTask.mailbox); } commReceived = null; } else { waitFor(1); } } catch (MsgException e) { commReceived = null; } } Msg.info("Tracker is leaving"); } } SimGrid-3.14.159/examples/java/app/masterworker/000755 001750 001750 00000000000 13031031310 023125 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/app/masterworker/Main.java000644 001750 001750 00000002347 13031031276 024675 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.masterworker; import java.io.File; import org.simgrid.msg.Msg; import org.simgrid.msg.NativeException; class Main { public static final int TASK_COMP_SIZE = 10000000; public static final int TASK_COMM_SIZE = 10000000; private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws NativeException { /* initialize the MSG simulation. Must be done before anything else (even logging). */ Msg.init(args); String platf = args.length > 1 ? args[0] : "examples/java/platform.xml"; String deploy = args.length > 1 ? args[1] : "examples/java/masterworker/masterworkerDeployment.xml"; Msg.verb("Platform: "+platf+"; Deployment:"+deploy+"; Current directory: "+new File(".").getAbsolutePath()); /* construct the platform and deploy the application */ Msg.createEnvironment(platf); Msg.deployApplication(deploy); /* execute the simulation. */ Msg.run(); } } SimGrid-3.14.159/examples/java/app/masterworker/Worker.java000644 001750 001750 00000002636 13031031276 025263 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.masterworker; import org.simgrid.msg.Host; import org.simgrid.msg.HostFailureException; import org.simgrid.msg.Msg; import org.simgrid.msg.Task; import org.simgrid.msg.TaskCancelledException; import org.simgrid.msg.TimeoutException; import org.simgrid.msg.TransferFailureException; import org.simgrid.msg.Process; public class Worker extends Process { public Worker(Host host, String name, String[]args) { super(host,name,args); } public void main(String[] args) throws TransferFailureException, HostFailureException, TimeoutException { if (args.length < 1) { Msg.info("Worker needs 1 argument (its number)"); System.exit(1); } int num = Integer.parseInt(args[0]); Msg.debug("Receiving on 'worker_"+num+"'"); while(true) { Task task = Task.receive("worker_"+num); if ("finalize".equals(task.getName())) { break; } Msg.info("Received \"" + task.getName() + "\". Processing it."); try { task.execute(); } catch (TaskCancelledException e) { e.printStackTrace(); } } Msg.info("Received Finalize. I'm done. See you!"); } } SimGrid-3.14.159/examples/java/app/masterworker/masterworker.xml000644 001750 001750 00000002536 13031031310 026402 0ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/app/masterworker/README000644 001750 001750 00000000543 13031031306 024014 0ustar00mquinsonmquinson000000 000000 This is a somehow basic master/workers example. There is 2 kind of processes: * Master: creates some tasks, and dispatches them to its workers * Worker: get tasks from the master and run them At the end of the execution: - the master sends a Task whose name is "finalize" to every known worker to stop them - On reception of such tasks workers stop.SimGrid-3.14.159/examples/java/app/masterworker/app_masterworker.tesh000644 001750 001750 00000003216 13031031303 027403 0ustar00mquinsonmquinson000000 000000 #! tesh $ java -classpath ${classpath:=.} app/masterworker/Main ${srcdir:=.}/../platforms/small_platform.xml ${srcdir:=.}/app/masterworker/masterworker.xml "--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n" > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (1:app.masterworker.Master@Jacquelin) Hello! Got 7 workers and 5 tasks to process > [ 0.860026] (2:app.masterworker.Worker@Tremblay) Received "Task_0". Processing it. > [ 1.752187] (3:app.masterworker.Worker@Fafard) Received "Task_1". Processing it. > [ 1.757531] (4:app.masterworker.Worker@Bourassa) Received "Task_2". Processing it. > [ 2.806417] (5:app.masterworker.Worker@Boivin) Received "Task_3". Processing it. > [ 2.811761] (6:app.masterworker.Worker@Ginette) Received "Task_4". Processing it. > [ 2.811761] (1:app.masterworker.Master@Jacquelin) All tasks have been dispatched. Let's tell everybody the computation is over. > [ 3.671783] (2:app.masterworker.Worker@Tremblay) Received Finalize. I'm done. See you! > [ 4.563940] (3:app.masterworker.Worker@Fafard) Received Finalize. I'm done. See you! > [ 4.569280] (4:app.masterworker.Worker@Bourassa) Received Finalize. I'm done. See you! > [ 5.618161] (5:app.masterworker.Worker@Boivin) Received Finalize. I'm done. See you! > [ 5.623501] (6:app.masterworker.Worker@Ginette) Received Finalize. I'm done. See you! > [ 5.628842] (7:app.masterworker.Worker@Jupiter) Received Finalize. I'm done. See you! > [ 5.629037] (8:app.masterworker.Worker@Jacquelin) Received Finalize. I'm done. See you! > [ 5.629037] (1:app.masterworker.Master@Jacquelin) Goodbye now! > [ 5.629037] (0:maestro@) MSG_main finished; Cleaning up the simulation... SimGrid-3.14.159/examples/java/app/masterworker/Master.java000644 001750 001750 00000003171 13031031276 025240 0ustar00mquinsonmquinson000000 000000 /* Master of a basic master/worker example in Java */ /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.masterworker; import org.simgrid.msg.Host; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; import org.simgrid.msg.Task; import org.simgrid.msg.Process; public class Master extends Process { public Master(Host host, String name, String[]args) { super(host,name,args); } public void main(String[] args) throws MsgException { if (args.length < 4) { Msg.info("Master needs 4 arguments"); System.exit(1); } int tasksCount = Integer.parseInt(args[0]); double taskComputeSize = Double.parseDouble(args[1]); double taskCommunicateSize = Double.parseDouble(args[2]); int workersCount = Integer.parseInt(args[3]); Msg.info("Hello! Got "+ workersCount + " workers and "+tasksCount+" tasks to process"); for (int i = 0; i < tasksCount; i++) { Task task = new Task("Task_" + i, taskComputeSize, taskCommunicateSize); Msg.debug("Sending \"" + task.getName()+ "\" to \"worker_" + i % workersCount + "\""); task.send("worker_"+(i%workersCount)); } Msg.info("All tasks have been dispatched. Let's tell everybody the computation is over."); for (int i = 0; i < workersCount; i++) { Task task = new Task("finalize", 0, 0); task.send("worker_"+(i%workersCount)); } Msg.info("Goodbye now!"); } } SimGrid-3.14.159/examples/java/app/pingpong/000755 001750 001750 00000000000 13031031303 022223 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/app/pingpong/Main.java000644 001750 001750 00000001711 13031031276 023763 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.pingpong; import org.simgrid.msg.Msg; import org.simgrid.msg.HostNotFoundException; import org.simgrid.msg.NativeException; class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws HostNotFoundException, NativeException{ Msg.init(args); if(args.length < 1) { Msg.info("Usage : Main platform_file"); Msg.info("example : Main ../platforms/platform.xml"); System.exit(1); } Msg.createEnvironment(args[0]); new Sender("Jacquelin", "Sender", new String[] {"Boivin"}).start(); new Receiver ("Boivin", "Receiver", null).start(); Msg.run(); } } SimGrid-3.14.159/examples/java/app/pingpong/Sender.java000644 001750 001750 00000003076 13031031276 024325 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.pingpong; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; import org.simgrid.msg.NativeException; import org.simgrid.msg.HostNotFoundException; public class Sender extends Process { private static final double COMM_SIZE_LAT = 1; public Sender(String hostname, String name, String[] args) throws HostNotFoundException, NativeException { super(hostname,name,args); } public void main(String[] args) throws MsgException { Msg.info("hello!"); int hostCount = args.length; Msg.info("host count: " + hostCount); String[] mailboxes = new String[hostCount]; double time; double computeDuration = 0; PingPongTask task; for(int pos = 0; pos < args.length ; pos++) { try { mailboxes[pos] = Host.getByName(args[pos]).getName(); } catch (HostNotFoundException e) { e.printStackTrace(); Msg.info("Invalid deployment file: " + e.toString()); } } for (int pos = 0; pos < hostCount; pos++) { time = Msg.getClock(); Msg.info("sender time: " + time); task = new PingPongTask("no name",computeDuration,COMM_SIZE_LAT); task.setTime(time); task.send(mailboxes[pos]); } Msg.info("goodbye!"); } }SimGrid-3.14.159/examples/java/app/pingpong/app_pingpong.tesh000644 001750 001750 00000001674 13031031303 025601 0ustar00mquinsonmquinson000000 000000 #! tesh ! output sort 19 $ java -classpath ${classpath:=.} app/pingpong/Main ${srcdir:=.}/../platforms/small_platform.xml > [0.000000] [jmsg/INFO] Using regular java threads. > [1.048882] [jmsg/INFO] MSG_main finished; Cleaning up the simulation... > [Boivin:Receiver:(2) 0.000000] [jmsg/INFO] hello! > [Boivin:Receiver:(2) 0.000000] [jmsg/INFO] try to get a task > [Boivin:Receiver:(2) 1.048882] [jmsg/INFO] Got at time 1.0488818628325232 > [Boivin:Receiver:(2) 1.048882] [jmsg/INFO] Was sent at time 0.0 > [Boivin:Receiver:(2) 1.048882] [jmsg/INFO] Communication time : 1.0488818628325232 > [Boivin:Receiver:(2) 1.048882] [jmsg/INFO] --- bw 9.533962169004269E7 ---- > [Boivin:Receiver:(2) 1.048882] [jmsg/INFO] goodbye! > [Jacquelin:Sender:(1) 0.000000] [jmsg/INFO] hello! > [Jacquelin:Sender:(1) 0.000000] [jmsg/INFO] host count: 1 > [Jacquelin:Sender:(1) 0.000000] [jmsg/INFO] sender time: 0.0 > [Jacquelin:Sender:(1) 1.048882] [jmsg/INFO] goodbye! SimGrid-3.14.159/examples/java/app/pingpong/PingPongTask.java000644 001750 001750 00000001451 13031031276 025444 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.pingpong; import org.simgrid.msg.Task; import org.simgrid.msg.NativeException; public class PingPongTask extends Task { private double timeVal; public PingPongTask() throws NativeException { this.timeVal = 0; } public PingPongTask(String name, double computeDuration, double messageSize) throws NativeException { super(name,computeDuration,messageSize); } public void setTime(double timeVal) { this.timeVal = timeVal; } public double getTime() { return this.timeVal; } } SimGrid-3.14.159/examples/java/app/pingpong/Receiver.java000644 001750 001750 00000002474 13031031276 024652 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.pingpong; import org.simgrid.msg.Msg; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; import org.simgrid.msg.NativeException; import org.simgrid.msg.HostNotFoundException; public class Receiver extends Process { private static final double COMM_SIZE_BW = 100000000; public Receiver(String hostname, String name, String[]args) throws HostNotFoundException, NativeException{ super(hostname,name,args); } public void main(String[] args) throws MsgException { Msg.info("hello!"); Msg.info("try to get a task"); PingPongTask task = (PingPongTask)Task.receive(getHost().getName()); double timeGot = Msg.getClock(); double timeSent = task.getTime(); Msg.info("Got at time "+ timeGot); Msg.info("Was sent at time "+timeSent); double time = timeSent; double communicationTime = timeGot - time; Msg.info("Communication time : " + communicationTime); Msg.info(" --- bw "+ COMM_SIZE_BW/communicationTime + " ----"); Msg.info("goodbye!"); } }SimGrid-3.14.159/examples/java/app/centralizedmutex/000755 001750 001750 00000000000 13031031310 023767 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/app/centralizedmutex/centralizedmutex.xml000644 001750 001750 00000001162 13031031310 030100 0ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/app/centralizedmutex/Main.java000644 001750 001750 00000001613 13031031276 025532 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.centralizedmutex; import org.simgrid.msg.Msg; import org.simgrid.msg.NativeException; class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws NativeException { Msg.init(args); String platf = args.length > 1 ? args[0] : "../platforms/small_platform.xml"; String deploy = args.length > 1 ? args[1] : "./centralizedmutex.xml"; /* construct the platform and deploy the application */ Msg.createEnvironment(platf); Msg.deployApplication(deploy); /* execute the simulation. */ Msg.run(); } } SimGrid-3.14.159/examples/java/app/centralizedmutex/Coordinator.java000644 001750 001750 00000003144 13031031276 027132 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.centralizedmutex; import java.util.LinkedList; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; public class Coordinator extends Process { LinkedList waitingQueue=new LinkedList<>(); int csToServe; public Coordinator(Host host, String name, String[]args) { super(host,name,args); } public void main(String[] args) throws MsgException { csToServe = Integer.parseInt(args[0]); Task task; while (csToServe >0) { task = Task.receive("coordinator"); if (task instanceof RequestTask) { RequestTask t = (RequestTask) task; if (waitingQueue.isEmpty()) { Msg.info("Got a request from "+t.from+". Queue empty: grant it"); GrantTask tosend = new GrantTask(); tosend.send(t.from); } else { waitingQueue.addFirst(t); } } else if (task instanceof ReleaseTask) { if (!waitingQueue.isEmpty()) { RequestTask req = waitingQueue.removeLast(); GrantTask tosend = new GrantTask(); tosend.send(req.from); } csToServe--; if (waitingQueue.isEmpty() && csToServe==0) { Msg.info("we should shutdown the simulation now"); } } } } } SimGrid-3.14.159/examples/java/app/centralizedmutex/RequestTask.java000644 001750 001750 00000000730 13031031276 027120 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.centralizedmutex; import org.simgrid.msg.Task; public class RequestTask extends Task { String from; public RequestTask(String name) { super(); from=name; } } SimGrid-3.14.159/examples/java/app/centralizedmutex/ReleaseTask.java000644 001750 001750 00000000566 13031031276 027057 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.centralizedmutex; public class ReleaseTask extends org.simgrid.msg.Task {} SimGrid-3.14.159/examples/java/app/centralizedmutex/Node.java000644 001750 001750 00000002147 13031031276 025536 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.centralizedmutex; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; public class Node extends Process { public Node(Host host, String name, String[]args) { super(host,name,args); } public void request(double CStime) throws MsgException { RequestTask req = new RequestTask(getName()); Msg.info("Send a request to the coordinator"); req.send("coordinator"); Msg.info("Wait for a grant from the coordinator"); GrantTask.receive(getName()); Task compute = new Task("CS", CStime, 0); compute.execute(); ReleaseTask release = new ReleaseTask(); release.send("coordinator"); } public void main(String[] args) throws MsgException { request(Double.parseDouble(args[1])); } }SimGrid-3.14.159/examples/java/app/centralizedmutex/GrantTask.java000644 001750 001750 00000000564 13031031276 026550 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package app.centralizedmutex; public class GrantTask extends org.simgrid.msg.Task {} SimGrid-3.14.159/examples/java/app/centralizedmutex/app_centralizedmutex.tesh000644 001750 001750 00000002107 13031031303 031105 0ustar00mquinsonmquinson000000 000000 #! tesh ! output sort 19 $ java -classpath ${classpath:=.} app/centralizedmutex/Main ${srcdir:=.}/../platforms/small_platform.xml ${srcdir:=.}/app/centralizedmutex/centralizedmutex.xml > [0.000000] [jmsg/INFO] Using regular java threads. > [Jupiter:app.centralizedmutex.Node:(2) 0.000000] [jmsg/INFO] Send a request to the coordinator > [Fafard:app.centralizedmutex.Node:(3) 0.000000] [jmsg/INFO] Send a request to the coordinator > [Tremblay:app.centralizedmutex.Coordinator:(1) 0.019014] [jmsg/INFO] Got a request from app.centralizedmutex.Node. Queue empty: grant it > [Jupiter:app.centralizedmutex.Node:(2) 0.019014] [jmsg/INFO] Wait for a grant from the coordinator > [Fafard:app.centralizedmutex.Node:(3) 0.063737] [jmsg/INFO] Wait for a grant from the coordinator > [Tremblay:app.centralizedmutex.Coordinator:(1) 0.063737] [jmsg/INFO] Got a request from app.centralizedmutex.Node. Queue empty: grant it > [Tremblay:app.centralizedmutex.Coordinator:(1) 0.134167] [jmsg/INFO] we should shutdown the simulation now > [0.134167] [jmsg/INFO] MSG_main finished; Cleaning up the simulation... SimGrid-3.14.159/examples/java/async/000755 001750 001750 00000000000 13031031276 020750 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/async/dsend/000755 001750 001750 00000000000 13031031303 022034 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/async/dsend/Main.java000644 001750 001750 00000002736 13031031276 023604 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package async.dsend; /** This example demonstrates the use of the Task.dsend() method. * * This way, the sender can be detached from the communication: it is not blocked as with Task.send() * and has nothing to do at the end as with Task.isend() where it must do a Comm.wait(). */ import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.NativeException; import org.simgrid.msg.HostNotFoundException; class Main { private Main() { /* This is just to ensure that nobody creates an instance of this singleton */ throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws NativeException, HostNotFoundException { Msg.init(args); String platform = "../platforms/small_platform.xml"; if (args.length >= 1) { platform = args[0]; // Override the default value if passed on the command line } /* construct the platform and deploy the application */ Msg.createEnvironment(platform); Host[] hosts = Host.all(); new Sender(hosts[0],"Sender").start(); for (int i=1; i < hosts.length; i++){ new Receiver(hosts[i], "Receiver").start(); } /* execute the simulation. */ Msg.run(); } } SimGrid-3.14.159/examples/java/async/dsend/async_dsend.tesh000644 001750 001750 00000003336 13031031303 025220 0ustar00mquinsonmquinson000000 000000 #! tesh ! timeout 30 $ java -classpath ${classpath:=.} async/dsend/Main ${srcdir:=.}/../platforms/small_platform.xml "--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n" > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (1:Sender@Boivin) Hello! Got 6 receivers to contact > [ 0.000000] (1:Sender@Boivin) Sending "Task_1" to "Bourassa" > [ 0.000000] (2:Receiver@Bourassa) Receiving on 'Bourassa' > [ 0.000000] (3:Receiver@Fafard) Receiving on 'Fafard' > [ 0.000000] (4:Receiver@Ginette) Receiving on 'Ginette' > [ 0.000000] (5:Receiver@Jacquelin) Receiving on 'Jacquelin' > [ 0.000000] (6:Receiver@Jupiter) Receiving on 'Jupiter' > [ 0.000000] (7:Receiver@Tremblay) Receiving on 'Tremblay' > [ 0.000000] (1:Sender@Boivin) Sending "Task_2" to "Fafard" > [ 0.000000] (1:Sender@Boivin) Sending "Task_3" to "Ginette" > [ 0.000000] (1:Sender@Boivin) Sending "Task_4" to "Jacquelin" > [ 0.000000] (1:Sender@Boivin) Sending "Task_5" to "Jupiter" > [ 0.000000] (1:Sender@Boivin) Sending "Task_6" to "Tremblay" > [ 0.000000] (1:Sender@Boivin) All tasks have been (asynchronously) dispatched. Let's sleep for 20s so that nobody gets a message from a terminated process. > [ 1.933362] (2:Receiver@Bourassa) Received a task. I'm done. See you! > [ 1.933362] (4:Receiver@Ginette) Received a task. I'm done. See you! > [ 1.933362] (6:Receiver@Jupiter) Received a task. I'm done. See you! > [ 2.449247] (7:Receiver@Tremblay) Received a task. I'm done. See you! > [ 2.964768] (3:Receiver@Fafard) Received a task. I'm done. See you! > [ 4.162002] (5:Receiver@Jacquelin) Received a task. I'm done. See you! > [ 20.000000] (1:Sender@Boivin) Done sleeping. Goodbye now! > [ 20.000000] (0:maestro@) MSG_main finished; Cleaning up the simulation... SimGrid-3.14.159/examples/java/async/dsend/Sender.java000644 001750 001750 00000002503 13031031276 024130 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package async.dsend; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; public class Sender extends Process { public Sender (Host host, String name){ super(host,name); } @Override public void main(String[] args) throws MsgException { double taskComputeSize =0; double taskCommunicateSize = 5000000; Host[] hosts = Host.all(); int receiverCount = hosts.length - 1; Msg.info("Hello! Got "+ receiverCount + " receivers to contact"); for (int i = 1; i <= receiverCount; i++) { Task task = new Task("Task_" + i, taskComputeSize, taskCommunicateSize); Msg.info("Sending \"" + task.getName()+ "\" to \"" + hosts[i].getName() + "\""); task.dsend(hosts[i].getName()); } Msg.info("All tasks have been (asynchronously) dispatched."+ " Let's sleep for 20s so that nobody gets a message from a terminated process."); waitFor(20); Msg.info("Done sleeping. Goodbye now!"); } } SimGrid-3.14.159/examples/java/async/dsend/Receiver.java000644 001750 001750 00000001707 13031031276 024461 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2007, 2010, 2013-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package async.dsend; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.HostFailureException; import org.simgrid.msg.TimeoutException; import org.simgrid.msg.TransferFailureException; public class Receiver extends Process { public Receiver (Host host, String name) { super(host,name); } @Override public void main(String[] args) throws TransferFailureException, HostFailureException, TimeoutException { Msg.info("Receiving on '"+ getHost().getName() + "'"); Task.receive(getHost().getName()); Msg.info("Received a task. I'm done. See you!"); } }SimGrid-3.14.159/examples/java/async/waitall/000755 001750 001750 00000000000 13031031303 022374 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/async/waitall/Main.java000644 001750 001750 00000002744 13031031276 024143 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package async.waitall; /** This example demonstrates the use of the asynchrounous communications * * Task.isend() and Task.irecv() are used to start the communications in non-blocking mode. * * The sends are then blocked onto with Comm.waitCompletion(), that locks until the given * communication terminates. * * The receives are packed into an array, and the sender blocks until all of them terminate * with Comm.waitAll(). */ import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.MsgException; class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws MsgException { Msg.init(args); String platform = "../platforms/small_platform.xml"; if (args.length >= 1) { platform = args[0]; // Override the default value if passed on the command line } /* construct the platform and deploy the application */ Msg.createEnvironment(platform); Host[] hosts = Host.all(); new Sender(hosts[0],"Sender").start(); for (int i=1; i < hosts.length; i++){ new Receiver(hosts[i], "Receiver").start(); } /* execute the simulation. */ Msg.run(); } } SimGrid-3.14.159/examples/java/async/waitall/async_waitall.tesh000644 001750 001750 00000003760 13031031303 026121 0ustar00mquinsonmquinson000000 000000 #! tesh $ java -classpath ${classpath:=.} async/waitall/Main ${srcdir:=.}/../platforms/small_platform.xml "--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n" > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (1:Sender@Boivin) I have 6 receivers to contact > [ 0.000000] (1:Sender@Boivin) Start the Sending 'Task_1' to 'Bourassa' > [ 0.000000] (1:Sender@Boivin) Start the Sending 'Task_2' to 'Fafard' > [ 0.000000] (2:Receiver@Bourassa) I started receiving on 'Bourassa. Wait 0.1 second, and block on the communication. > [ 0.000000] (3:Receiver@Fafard) I started receiving on 'Fafard. Wait 0.1 second, and block on the communication. > [ 0.000000] (4:Receiver@Ginette) I started receiving on 'Ginette. Wait 0.1 second, and block on the communication. > [ 0.000000] (5:Receiver@Jacquelin) I started receiving on 'Jacquelin. Wait 0.1 second, and block on the communication. > [ 0.000000] (6:Receiver@Jupiter) I started receiving on 'Jupiter. Wait 0.1 second, and block on the communication. > [ 0.000000] (7:Receiver@Tremblay) I started receiving on 'Tremblay. Wait 0.1 second, and block on the communication. > [ 0.000000] (1:Sender@Boivin) Start the Sending 'Task_3' to 'Ginette' > [ 0.000000] (1:Sender@Boivin) Start the Sending 'Task_4' to 'Jacquelin' > [ 0.000000] (1:Sender@Boivin) Start the Sending 'Task_5' to 'Jupiter' > [ 0.000000] (1:Sender@Boivin) Start the Sending 'Task_6' to 'Tremblay' > [ 0.000000] (1:Sender@Boivin) All tasks have been (asynchronously) dispatched. Let's wait for their completion. > [ 1.933362] (2:Receiver@Bourassa) I got my task, good bye. > [ 1.933362] (4:Receiver@Ginette) I got my task, good bye. > [ 1.933362] (6:Receiver@Jupiter) I got my task, good bye. > [ 2.449247] (7:Receiver@Tremblay) I got my task, good bye. > [ 2.964768] (3:Receiver@Fafard) I got my task, good bye. > [ 4.162002] (5:Receiver@Jacquelin) I got my task, good bye. > [ 4.162002] (1:Sender@Boivin) Goodbye now! > [ 4.162002] (0:maestro@) MSG_main finished; Cleaning up the simulation... SimGrid-3.14.159/examples/java/async/waitall/Sender.java000644 001750 001750 00000002560 13031031276 024473 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package async.waitall; import org.simgrid.msg.Comm; import org.simgrid.msg.Host; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; import org.simgrid.msg.Process; import org.simgrid.msg.Task; public class Sender extends Process { public Sender (Host host, String name){ super(host,name); } @Override public void main(String[] args) throws MsgException { double taskComputeSize =0; double taskCommunicateSize = 5000000; Host[] hosts = Host.all(); int receiverCount = hosts.length - 1; Msg.info("I have "+ receiverCount + " receivers to contact"); Comm[] communicators = new Comm[receiverCount]; for (int i = 1; i <= receiverCount; i++) { Task task = new Task("Task_" + i, taskComputeSize, taskCommunicateSize); Msg.info("Start the Sending '" + task.getName()+ "' to '" + hosts[i].getName() + "'"); communicators[i-1] = task.isend(hosts[i].getName()); } Msg.info("All tasks have been (asynchronously) dispatched. Let's wait for their completion."); Comm.waitAll(communicators); Msg.info("Goodbye now!"); } } SimGrid-3.14.159/examples/java/async/waitall/Receiver.java000644 001750 001750 00000002327 13031031276 025020 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2007, 2010, 2013-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package async.waitall; import org.simgrid.msg.Msg; import org.simgrid.msg.Comm; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.HostFailureException; import org.simgrid.msg.TimeoutException; import org.simgrid.msg.TransferFailureException; public class Receiver extends Process { public Receiver (Host host, String name) { super(host,name); } @Override public void main(String[] args) throws TransferFailureException, HostFailureException, TimeoutException { Comm comm = Task.irecv(getHost().getName()); Msg.info("I started receiving on '"+ getHost().getName() +". Wait 0.1 second, and block on the communication."); waitFor(0.1); try { comm.waitCompletion(); } catch (TimeoutException e) { Msg.info("Timeout while waiting for my task"); throw e; // Stop this process } Msg.info("I got my task, good bye."); } }SimGrid-3.14.159/examples/java/process/000755 001750 001750 00000000000 13031031277 021312 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/process/kill/000755 001750 001750 00000000000 13031031303 022233 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/process/kill/Main.java000644 001750 001750 00000001726 13031031276 024001 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package process.kill; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; import org.simgrid.msg.NativeException; public class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws NativeException { /* initialize the MSG simulation. Must be done before anything else (even logging). */ Msg.init(args); Msg.createEnvironment(args[0]); /* bypass deploymemt */ try { Killer killer = new Killer("Jacquelin","killer"); killer.start(); } catch (MsgException e){ System.out.println("Create processes failed!"); } /* execute the simulation. */ Msg.run(); } } SimGrid-3.14.159/examples/java/process/kill/Victim.java000644 001750 001750 00000001553 13031031276 024346 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package process.kill; import org.simgrid.msg.Msg; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; import org.simgrid.msg.HostNotFoundException; public class Victim extends Process { public Victim(String hostname, String name) throws HostNotFoundException { super(hostname, name); } public void main(String[] args) throws MsgException{ Msg.info("Hello!"); Msg.info("Suspending myself"); suspend(); Msg.info("OK, OK. Let's work"); Task task = new Task("work", 1e9, 0); task.execute(); Msg.info("Bye"); } }SimGrid-3.14.159/examples/java/process/kill/process_kill.tesh000644 001750 001750 00000001246 13031031303 025614 0ustar00mquinsonmquinson000000 000000 #! tesh ! output sort 19 $ java -classpath ${classpath:=.} process/kill/Main ${srcdir:=.}/../platforms/small_platform.xml --lof=no_loc > [0.000000] [jmsg/INFO] Using regular java threads. > [11.000000] [jmsg/INFO] MSG_main finished; Cleaning up the simulation... > [Boivin:victim:(2) 0.000000] [jmsg/INFO] Hello! > [Boivin:victim:(2) 0.000000] [jmsg/INFO] Suspending myself > [Boivin:victim:(2) 10.000000] [jmsg/INFO] OK, OK. Let's work > [Jacquelin:killer:(1) 0.000000] [jmsg/INFO] Hello! > [Jacquelin:killer:(1) 10.000000] [jmsg/INFO] Resume Process > [Jacquelin:killer:(1) 11.000000] [jmsg/INFO] Kill Process > [Jacquelin:killer:(1) 11.000000] [jmsg/INFO] Ok, goodbye now. SimGrid-3.14.159/examples/java/process/kill/Killer.java000644 001750 001750 00000002157 13031031276 024336 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package process.kill; import org.simgrid.msg.Msg; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; import org.simgrid.msg.HostNotFoundException; import process.kill.Victim; public class Killer extends Process { public Killer(String hostname, String name) throws HostNotFoundException { super(hostname, name); } public void main(String[] args) throws MsgException { Victim poorVictim = null; Msg.info("Hello!"); try { poorVictim = new Victim("Boivin","victim"); poorVictim.start(); } catch (MsgException e){ e.printStackTrace(); Msg.error("Cannot create the victim process!"); return; } sleep(10000); Msg.info("Resume Process"); poorVictim.resume(); sleep(1000); Msg.info("Kill Process"); poorVictim.kill(); Msg.info("Ok, goodbye now."); } } SimGrid-3.14.159/examples/java/process/startkilltime/000755 001750 001750 00000000000 13031031310 024166 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/process/startkilltime/Main.java000644 001750 001750 00000001715 13031031277 025735 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package process.startkilltime; import org.simgrid.msg.Msg; import org.simgrid.msg.NativeException; public class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws NativeException { Msg.init(args); if(args.length < 2) { Msg.info("Usage : StartKilltime platform_file deployment_file"); Msg.info("example : StartKilltime ../platforms/platform.xml deployment_start_kill.xml"); System.exit(1); } /* construct the platform and deploy the application */ Msg.createEnvironment(args[0]); Msg.deployApplication(args[1]); /* execute the simulation. */ Msg.run(); } } SimGrid-3.14.159/examples/java/process/startkilltime/Sleeper.java000644 001750 001750 00000001500 13031031277 026440 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package process.startkilltime; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; public class Sleeper extends Process { public Sleeper(Host host, String name, String[]args) { super(host,name,args); } public void main(String[] args) throws MsgException { Msg.info("Hello! I go to sleep."); try { waitFor(Integer.parseInt(args[0])); Msg.info("Done sleeping"); } catch (MsgException e) { Msg.debug("Wait cancelled."); } } } SimGrid-3.14.159/examples/java/process/startkilltime/startkilltime.xml000644 001750 001750 00000001757 13031031310 027612 0ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/process/startkilltime/process_startkilltime.tesh000644 001750 001750 00000002030 13031031303 031476 0ustar00mquinsonmquinson000000 000000 #! tesh $ java -classpath ${classpath:=.} process/startkilltime/Main ${srcdir:=.}/../platforms/cluster.xml ${srcdir:=.}/process/startkilltime/startkilltime.xml > [0.000000] [jmsg/INFO] Using regular java threads. > [node-0.acme.org:process.startkilltime.Sleeper:(1) 0.000000] [jmsg/INFO] Hello! I go to sleep. > [node-1.acme.org:process.startkilltime.Sleeper:(2) 1.000000] [jmsg/INFO] Hello! I go to sleep. > [node-2.acme.org:process.startkilltime.Sleeper:(3) 2.000000] [jmsg/INFO] Hello! I go to sleep. > [node-3.acme.org:process.startkilltime.Sleeper:(4) 3.000000] [jmsg/INFO] Hello! I go to sleep. > [node-4.acme.org:process.startkilltime.Sleeper:(5) 4.000000] [jmsg/INFO] Hello! I go to sleep. > [node-5.acme.org:process.startkilltime.Sleeper:(6) 5.000000] [jmsg/INFO] Hello! I go to sleep. > [node-2.acme.org:process.startkilltime.Sleeper:(3) 6.000000] [jmsg/INFO] Done sleeping > [node-3.acme.org:process.startkilltime.Sleeper:(4) 7.000000] [jmsg/INFO] Done sleeping > [10.000000] [jmsg/INFO] MSG_main finished; Cleaning up the simulation... SimGrid-3.14.159/examples/java/process/suspend/000755 001750 001750 00000000000 13031031303 022761 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/process/suspend/Main.java000644 001750 001750 00000002050 13031031277 024517 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package process.suspend; import org.simgrid.msg.Msg; import org.simgrid.msg.MsgException; public class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) { Msg.init(args); if(args.length < 1) { Msg.info("Usage : Suspend platform_file"); Msg.info("example : Suspend ../platforms/platform.xml"); System.exit(1); } /* construct the platform and deploy the application */ Msg.createEnvironment(args[0]); try { DreamMaster process1 = new DreamMaster("Jacquelin","DreamMaster"); process1.start(); } catch (MsgException e){ System.out.println("Create processes failed!"); } /* execute the simulation. */ Msg.run(); } } SimGrid-3.14.159/examples/java/process/suspend/process_suspend.tesh000644 001750 001750 00000001412 13031031303 027063 0ustar00mquinsonmquinson000000 000000 #! tesh ! output sort 19 $ java -classpath ${classpath:=.} process/suspend/Main ${srcdir:=.}/../platforms/small_platform.xml "--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n" > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (1:DreamMaster@Jacquelin) Let's create a lazy guy. > [ 0.000000] (1:DreamMaster@Jacquelin) Let's wait a little bit... > [ 0.000000] (2:Lazy@Jacquelin) Nobody's watching me ? Let's go to sleep. > [ 10.000000] (0:maestro@) MSG_main finished; Cleaning up the simulation... > [ 10.000000] (1:DreamMaster@Jacquelin) Let's wake the lazy guy up! >:) BOOOOOUUUHHH!!!! > [ 10.000000] (1:DreamMaster@Jacquelin) OK, goodbye now. > [ 10.000000] (2:Lazy@Jacquelin) Uuuh ? Did somebody call me ? > [ 10.000000] (2:Lazy@Jacquelin) Mmmh, goodbye now. SimGrid-3.14.159/examples/java/process/suspend/LazyGuy.java000644 001750 001750 00000001421 13031031277 025240 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package process.suspend; import org.simgrid.msg.Host; import org.simgrid.msg.Msg; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; public class LazyGuy extends Process { public LazyGuy(Host host, String name, String[]args) { super(host,name,args); } public void main(String[] args) throws MsgException { Msg.info("Nobody's watching me ? Let's go to sleep."); suspend(); Msg.info("Uuuh ? Did somebody call me ?"); Msg.info("Mmmh, goodbye now."); } }SimGrid-3.14.159/examples/java/process/suspend/DreamMaster.java000644 001750 001750 00000001713 13031031277 026044 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package process.suspend; import org.simgrid.msg.Msg; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; import org.simgrid.msg.HostNotFoundException; public class DreamMaster extends Process { public DreamMaster(String hostname, String name) throws HostNotFoundException{ super(hostname,name); } public void main(String[] args) throws MsgException { Msg.info("Let's create a lazy guy."); Process lazyGuy = new LazyGuy(getHost(),"Lazy",null); lazyGuy.start(); Msg.info("Let's wait a little bit..."); waitFor(10); Msg.info("Let's wake the lazy guy up! >:) BOOOOOUUUHHH!!!!"); lazyGuy.resume(); Msg.info("OK, goodbye now."); } }SimGrid-3.14.159/examples/java/process/migration/000755 001750 001750 00000000000 13031031303 023271 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/process/migration/Main.java000644 001750 001750 00000002643 13031031277 025037 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package process.migration; import org.simgrid.msg.Msg; import org.simgrid.msg.Mutex; import org.simgrid.msg.Process; import org.simgrid.msg.NativeException; import org.simgrid.msg.HostNotFoundException; class Main { protected static Mutex mutex; protected static Process processToMigrate = null; private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws NativeException { Msg.init(args); if(args.length < 1) { Msg.info("Usage : Migration platform_file"); Msg.info("example : Migration ../platforms/platform.xml"); System.exit(1); } /* Create the mutex */ mutex = new Mutex(); /* construct the platform*/ Msg.createEnvironment(args[0]); /* bypass deploymemt */ try { Policeman policeman = new Policeman("Boivin","policeman"); policeman.start(); Emigrant emigrant = new Emigrant("Jacquelin","emigrant"); emigrant.start(); } catch (HostNotFoundException e){ Msg.error("Create processes failed!"); e.printStackTrace(); } /* execute the simulation. */ Msg.run(); } } SimGrid-3.14.159/examples/java/process/migration/Policeman.java000644 001750 001750 00000001660 13031031277 026060 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package process.migration; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; import org.simgrid.msg.HostNotFoundException; public class Policeman extends Process { public Policeman(String hostname, String name) throws HostNotFoundException { super(hostname, name); } public void main(String[] args) throws MsgException { waitFor(1); Msg.info("Wait a bit before migrating the emigrant."); Main.mutex.acquire(); Main.processToMigrate.migrate(Host.getByName("Jacquelin")); Msg.info("I moved the emigrant"); Main.processToMigrate.resume(); } }SimGrid-3.14.159/examples/java/process/migration/process_migration.tesh000644 001750 001750 00000001503 13031031303 027704 0ustar00mquinsonmquinson000000 000000 #! tesh ! output sort 19 $ java -classpath ${classpath:=.} process/migration/Main ${srcdir:=.}/../platforms/small_platform.xml "--log=root.fmt:[%10.6r]%e(%i:%P@%h)%e%m%n" > [ 0.000000] (0:maestro@) Using regular java threads. > [ 0.000000] (2:emigrant@Jacquelin) I'll look for a new job on another machine where the grass is greener. > [ 0.000000] (2:emigrant@Boivin) Yeah, found something to do > [ 1.000000] (1:policeman@Boivin) Wait a bit before migrating the emigrant. > [ 3.000000] (2:emigrant@Boivin) Moving back to home after work > [ 7.000000] (0:maestro@) MSG_main finished; Cleaning up the simulation... > [ 7.000000] (2:emigrant@Jacquelin) I've been moved on this new host:Jacquelin > [ 7.000000] (2:emigrant@Jacquelin) Uh, nothing to do here. Stopping now > [ 7.000000] (1:policeman@Boivin) I moved the emigrant SimGrid-3.14.159/examples/java/process/migration/Emigrant.java000644 001750 001750 00000002520 13031031277 025713 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014, 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package process.migration; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Task; import org.simgrid.msg.Process; import org.simgrid.msg.MsgException; import org.simgrid.msg.HostNotFoundException; public class Emigrant extends Process { public Emigrant(String hostname, String name) throws HostNotFoundException { super(hostname, name); } public void main(String[] args) throws MsgException { Main.mutex.acquire(); Msg.info("I'll look for a new job on another machine where the grass is greener."); migrate(Host.getByName("Boivin")); Msg.info("Yeah, found something to do"); Task task = new Task("job", 98095000, 0); task.execute(); waitFor(2); Msg.info("Moving back to home after work"); migrate(Host.getByName("Jacquelin")); migrate(Host.getByName("Boivin")); waitFor(4); Main.processToMigrate = this; Main.mutex.release(); suspend(); Msg.info("I've been moved on this new host:" + getHost().getName()); Msg.info("Uh, nothing to do here. Stopping now"); } }SimGrid-3.14.159/examples/java/io/000755 001750 001750 00000000000 13031031276 020242 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/io/storage/000755 001750 001750 00000000000 13031031303 021675 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/io/storage/Main.java000644 001750 001750 00000001573 13031031276 023443 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package io.storage; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.MsgException; public class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws MsgException { Msg.init(args); if(args.length < 1) { Msg.info("Usage : Storage platform_file "); Msg.info("example : Storage ../platforms/storage/storage.xml "); System.exit(1); } Msg.createEnvironment(args[0]); Host[] hosts = Host.all(); new io.storage.Client(hosts[3],0).start(); Msg.run(); } } SimGrid-3.14.159/examples/java/io/storage/Client.java000644 001750 001750 00000004431 13031031276 023771 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /********************* Files and Storage handling **************************** * This example implements some storage related functions of the MSG API * * Scenario : * - display information on the disks mounted by the current host * - attach some properties to a disk * - list all the storage elements in the platform * ******************************************************************************/ package io.storage; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.Process; import org.simgrid.msg.Storage; import org.simgrid.msg.HostNotFoundException; import org.simgrid.msg.MsgException; public class Client extends Process { public Client(Host host, int number) throws HostNotFoundException { super(host, Integer.toString(number), null); } public void main(String[] args) throws MsgException { // Retrieve all mount points of current host Storage[] storages = getHost().getMountedStorage(); for (int i = 0; i < storages.length; i++) { // For each disk mounted on host Msg.info("------------------------------------"); Msg.info("Disk name: "+storages[i].getName()); Msg.info("Size: "+storages[i].getSize()+" bytes."); Msg.info("Free Size: "+storages[i].getFreeSize()+" bytes."); Msg.info("Used Size: "+storages[i].getUsedSize()+" bytes."); } Storage st = Storage.getByName("Disk2"); Msg.info("Disk name: "+st.getName()); Msg.info("Attached to host:"+st.getHost()); st.setProperty("key","Pierre"); Msg.info("Property key: "+st.getProperty("key")); Host h = Host.currentHost(); h.setProperty("key2","Pierre"); Msg.info("Property key2: "+h.getProperty("key2")); String[] attach = h.getAttachedStorage(); for (int j = 0; j < attach.length; j++) { Msg.info("Disk attached: "+attach[j]); } Msg.info("**************** ALL *************************"); Storage[] stos = Storage.all(); for (int i = 0; i < stos.length; i++) { Msg.info("Disk: "+ stos[i].getName()); } } }SimGrid-3.14.159/examples/java/io/storage/io_storage.tesh000644 001750 001750 00000002675 13031031303 024727 0ustar00mquinsonmquinson000000 000000 #! tesh $ java -classpath ${classpath:=.} io/storage/Main ${srcdir:=.}/../platforms/storage/storage.xml > [0.000000] [jmsg/INFO] Using regular java threads. > [denise:0:(1) 0.000000] [jmsg/INFO] ------------------------------------ > [denise:0:(1) 0.000000] [jmsg/INFO] Disk name: Disk2 > [denise:0:(1) 0.000000] [jmsg/INFO] Size: 536870912000 bytes. > [denise:0:(1) 0.000000] [jmsg/INFO] Free Size: 534479374867 bytes. > [denise:0:(1) 0.000000] [jmsg/INFO] Used Size: 2391537133 bytes. > [denise:0:(1) 0.000000] [jmsg/INFO] ------------------------------------ > [denise:0:(1) 0.000000] [jmsg/INFO] Disk name: Disk4 > [denise:0:(1) 0.000000] [jmsg/INFO] Size: 536870912000 bytes. > [denise:0:(1) 0.000000] [jmsg/INFO] Free Size: 536857690006 bytes. > [denise:0:(1) 0.000000] [jmsg/INFO] Used Size: 13221994 bytes. > [denise:0:(1) 0.000000] [jmsg/INFO] Disk name: Disk2 > [denise:0:(1) 0.000000] [jmsg/INFO] Attached to host:alice > [denise:0:(1) 0.000000] [jmsg/INFO] Property key: Pierre > [denise:0:(1) 0.000000] [jmsg/INFO] Property key2: Pierre > [denise:0:(1) 0.000000] [jmsg/INFO] Disk attached: Disk4 > [denise:0:(1) 0.000000] [jmsg/INFO] **************** ALL ************************* > [denise:0:(1) 0.000000] [jmsg/INFO] Disk: Disk1 > [denise:0:(1) 0.000000] [jmsg/INFO] Disk: Disk2 > [denise:0:(1) 0.000000] [jmsg/INFO] Disk: Disk3 > [denise:0:(1) 0.000000] [jmsg/INFO] Disk: Disk4 > [0.000000] [jmsg/INFO] MSG_main finished; Cleaning up the simulation... SimGrid-3.14.159/examples/java/io/file/000755 001750 001750 00000000000 13031031303 021150 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/java/io/file/Main.java000644 001750 001750 00000001727 13031031276 022717 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ package io.file; import org.simgrid.msg.Msg; import org.simgrid.msg.Host; import org.simgrid.msg.MsgException; public class Main { private Main() { throw new IllegalAccessError("Utility class"); } public static void main(String[] args) throws MsgException { Msg.init(args); if(args.length < 1) { Msg.info("Usage : IO platform_file "); Msg.info("example : IO ../platforms/storage/storage.xml "); System.exit(1); } Msg.createEnvironment(args[0]); Host[] hosts = Host.all(); Msg.info("Number of hosts:" + hosts.length); for (int i = 0; i < hosts.length && i < 4; i++) { new io.file.Node(hosts[i],i).start(); } Msg.run(); } }SimGrid-3.14.159/examples/java/io/file/Node.java000644 001750 001750 00000004660 13031031276 022717 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /********************* Files and Storage handling **************************** * This example implements all main file functions of the MSG API * * Scenario: Each host * - opens a file * - tries to read 10,000 bytes in this file * - writes 100,000 bytes in the file * - seeks back to the beginning of the file * - tries to read 150,000 bytes in this file * ******************************************************************************/ package io.file; import org.simgrid.msg.Msg; import org.simgrid.msg.File; import org.simgrid.msg.Host; import org.simgrid.msg.Process; import org.simgrid.msg.HostNotFoundException; import org.simgrid.msg.MsgException; public class Node extends Process { private static String FILENAME1 = "/doc/simgrid/examples/platforms/g5k.xml"; private static String FILENAME2 = "\\Windows\\setupact.log"; private static String FILENAME3 = "/doc/simgrid/examples/platforms/g5k_cabinets.xml"; private static String FILENAME4 = "/doc/simgrid/examples/platforms/nancy.xml"; protected int rank; public Node(Host host, int number) throws HostNotFoundException { super(host, Integer.toString(number), null); this.rank = number; } public void main(String[] args) throws MsgException { String mount; String filename; switch (rank) { case 4: mount = "/home"; filename = mount + FILENAME1; break; case 0: mount = "c:"; filename = mount + FILENAME2; break; case 2: mount = "/home"; filename = mount + FILENAME3; break; case 1: mount = "/home"; filename = mount + FILENAME4; break; default: mount = "/home"; filename = mount + FILENAME1; } Msg.info("Open file " + filename); File file = new File(filename); long read = file.read(10000,1); Msg.info("Having read " + read + " on " + filename); long write = file.write(100000,1); Msg.info("Having write " + write + " on " + filename); Msg.info("Seek back to the beginning of " + filename); file.seek(0,File.SEEK_SET); read = file.read(150000,1); Msg.info("Having read " + read + " on " + filename); } } SimGrid-3.14.159/examples/java/io/file/io_file.tesh000644 001750 001750 00000004343 13031031303 023447 0ustar00mquinsonmquinson000000 000000 #! tesh $ java -classpath ${classpath:=.} io/file/Main ${srcdir:=.}/../platforms/storage/storage.xml > [0.000000] [jmsg/INFO] Using regular java threads. > [0.000000] [jmsg/INFO] Number of hosts:4 > [alice:0:(1) 0.000000] [jmsg/INFO] Open file c:\Windows\setupact.log > [bob:1:(2) 0.000000] [jmsg/INFO] Open file /home/doc/simgrid/examples/platforms/nancy.xml > [carl:2:(3) 0.000000] [jmsg/INFO] Open file /home/doc/simgrid/examples/platforms/g5k_cabinets.xml > [denise:3:(4) 0.000000] [jmsg/INFO] Open file /home/doc/simgrid/examples/platforms/g5k.xml > [bob:1:(2) 0.000040] [jmsg/INFO] Having read 4028 on /home/doc/simgrid/examples/platforms/nancy.xml > [alice:0:(1) 0.000050] [jmsg/INFO] Having read 10000 on c:\Windows\setupact.log > [denise:3:(4) 0.000050] [jmsg/INFO] Having read 10000 on /home/doc/simgrid/examples/platforms/g5k.xml > [carl:2:(3) 0.000100] [jmsg/INFO] Having read 10000 on /home/doc/simgrid/examples/platforms/g5k_cabinets.xml > [alice:0:(1) 0.001717] [jmsg/INFO] Having write 100000 on c:\Windows\setupact.log > [alice:0:(1) 0.001717] [jmsg/INFO] Seek back to the beginning of c:\Windows\setupact.log > [denise:3:(4) 0.001717] [jmsg/INFO] Having write 100000 on /home/doc/simgrid/examples/platforms/g5k.xml > [denise:3:(4) 0.001717] [jmsg/INFO] Seek back to the beginning of /home/doc/simgrid/examples/platforms/g5k.xml > [alice:0:(1) 0.002267] [jmsg/INFO] Having read 110000 on c:\Windows\setupact.log > [denise:3:(4) 0.002267] [jmsg/INFO] Having read 110000 on /home/doc/simgrid/examples/platforms/g5k.xml > [bob:1:(2) 0.003374] [jmsg/INFO] Having write 100000 on /home/doc/simgrid/examples/platforms/nancy.xml > [bob:1:(2) 0.003374] [jmsg/INFO] Seek back to the beginning of /home/doc/simgrid/examples/platforms/nancy.xml > [carl:2:(3) 0.003433] [jmsg/INFO] Having write 100000 on /home/doc/simgrid/examples/platforms/g5k_cabinets.xml > [carl:2:(3) 0.003433] [jmsg/INFO] Seek back to the beginning of /home/doc/simgrid/examples/platforms/g5k_cabinets.xml > [bob:1:(2) 0.004414] [jmsg/INFO] Having read 104028 on /home/doc/simgrid/examples/platforms/nancy.xml > [carl:2:(3) 0.004533] [jmsg/INFO] Having read 110000 on /home/doc/simgrid/examples/platforms/g5k_cabinets.xml > [0.004533] [jmsg/INFO] MSG_main finished; Cleaning up the simulation... SimGrid-3.14.159/examples/simdag/000755 001750 001750 00000000000 13031031300 020142 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/simdag/schedule-dotload/000755 001750 001750 00000000000 13031031306 023370 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/simdag/schedule-dotload/sd_schedule-dotload.tesh000644 001750 001750 00000031503 13031031304 030163 0ustar00mquinsonmquinson000000 000000 #! ./tesh p Test the loader of DAG written in the DOT format ! expect return 2 $ $SG_TEST_EXENV ${bindir:=.}/schedule-dotload/sd_schedule-dotload --log=no_loc "--log=sd_dotparse.thres:verbose" ${srcdir:=.}/../platforms/two_clusters.xml ${srcdir:=.}/schedule-dotload/dag_with_bad_schedule.dot > [0.000000] [xbt_cfg/INFO] Switching to the L07 model to handle parallel tasks. > [0.000000] [sd_dotparse/VERBOSE] The schedule is ignored, task 'end' can not be scheduled on -1 hosts > [0.000000] [sd_dotparse/VERBOSE] The schedule is ignored, task '1' can not be scheduled on 0 hosts > [0.000000] [sd_dotparse/VERBOSE] Task '0' wants to start on performer '1' at the same position '1' as task '2' > [0.000000] [sd_dotparse/VERBOSE] The schedule is ignored, task '3' can not be scheduled on -1 hosts > [0.000000] [sd_dotparse/VERBOSE] The schedule is ignored, task 'root' can not be scheduled on -1 hosts > [0.000000] [sd_dotparse/WARNING] The scheduling is ignored > [0.000000] [test/CRITICAL] The dot file with the provided scheduling is wrong, more information with the option : --log=sd_dotparse.thres:verbose $ $SG_TEST_EXENV ${bindir:=.}/schedule-dotload/sd_schedule-dotload --log=no_loc ${srcdir:=.}/../platforms/two_clusters.xml ${srcdir:=.}/schedule-dotload/dag_with_good_schedule.dot > [0.000000] [xbt_cfg/INFO] Switching to the L07 model to handle parallel tasks. > [0.000000] [test/INFO] ------------------- Display all tasks of the loaded DAG --------------------------- > [0.000000] [sd_task/INFO] Displaying task root > [0.000000] [sd_task/INFO] - state: runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 0 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 0 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 0 > [0.000000] [sd_task/INFO] 1 > [0.000000] [sd_task/INFO] root->5 > [0.000000] [sd_task/INFO] Displaying task 0 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000129 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] root > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 2 > [0.000000] [sd_task/INFO] 0->2 > [0.000000] [sd_task/INFO] Displaying task 1 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000131 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] root > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 4 > [0.000000] [sd_task/INFO] 1->2 > [0.000000] [sd_task/INFO] Displaying task 2 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000121 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 3 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 0 > [0.000000] [sd_task/INFO] 0->2 > [0.000000] [sd_task/INFO] 1->2 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 3 > [0.000000] [sd_task/INFO] 2->3 > [0.000000] [sd_task/INFO] Displaying task 3 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000231 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 2 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 2 > [0.000000] [sd_task/INFO] 2->3 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 4 > [0.000000] [sd_task/INFO] 8 > [0.000000] [sd_task/INFO] Displaying task 4 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000005 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 2 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 1 > [0.000000] [sd_task/INFO] 3 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 5 > [0.000000] [sd_task/INFO] 4->5 > [0.000000] [sd_task/INFO] Displaying task 5 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000046 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 3 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 4 > [0.000000] [sd_task/INFO] root->5 > [0.000000] [sd_task/INFO] 4->5 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 6 > [0.000000] [sd_task/INFO] Displaying task 6 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000092 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 5 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 7 > [0.000000] [sd_task/INFO] 6->7 > [0.000000] [sd_task/INFO] Displaying task 7 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000041 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 2 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 6 > [0.000000] [sd_task/INFO] 6->7 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] end > [0.000000] [sd_task/INFO] 7->8 > [0.000000] [sd_task/INFO] 7->end > [0.000000] [sd_task/INFO] Displaying task 8 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000250 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 2 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 3 > [0.000000] [sd_task/INFO] 7->8 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 9 > [0.000000] [sd_task/INFO] Displaying task 9 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000079 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 8 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] end > [0.000000] [sd_task/INFO] Displaying task root->5 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10014000 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] root > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 5 > [0.000000] [sd_task/INFO] Displaying task 0->2 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10001 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 0 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 2 > [0.000000] [sd_task/INFO] Displaying task 1->2 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10004 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 1 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 2 > [0.000000] [sd_task/INFO] Displaying task 2->3 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10002 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 2 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 3 > [0.000000] [sd_task/INFO] Displaying task 4->5 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10029 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 4 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 5 > [0.000000] [sd_task/INFO] Displaying task 6->7 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10005 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 6 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 7 > [0.000000] [sd_task/INFO] Displaying task 7->8 > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10000 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 7 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 8 > [0.000000] [sd_task/INFO] Displaying task 7->end > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10014000 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 7 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] end > [0.000000] [sd_task/INFO] Displaying task end > [0.000000] [sd_task/INFO] - state: scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000129 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 3 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 7 > [0.000000] [sd_task/INFO] 9 > [0.000000] [sd_task/INFO] 7->end > [0.000000] [test/INFO] ------------------- Run the schedule --------------------------- > [100.003561] [test/INFO] ------------------- Produce the trace file--------------------------- > [100.003561] [test/INFO] Producing the trace of the run into dag_with_good_schedule.trace $ cat ${srcdir:=.}/schedule-dotload/dag_with_good_schedule.trace > [0.000000->0.000000] node-0.acme.org compute 0.000000 flops # root > [0.000000->10.000000] node-1.acme.org compute 10000000129.452715 flops # 0 > [0.000000->10.000000] node-0.acme.org compute 10000000131.133657 flops # 1 > [10.000760->20.000760] node-1.acme.org compute 10000000121.124870 flops # 2 > [20.001440->30.001441] node-1.acme.org compute 10000000230.608025 flops # 3 > [30.001441->40.001441] node-0.acme.org compute 10000000004.994019 flops # 4 > [40.002121->50.002121] node-0.acme.org compute 10000000046.016401 flops # 5 > [50.002121->60.002121] node-0.acme.org compute 10000000091.598791 flops # 6 > [60.002801->70.002801] node-0.acme.org compute 10000000040.679438 flops # 7 > [70.003561->80.003561] node-1.acme.org compute 10000000250.490017 flops # 8 > [80.003561->90.003561] node-1.acme.org compute 10000000079.267649 flops # 9 > [0.000000 -> 0.080712] node-0.acme.org -> node-0.acme.org transfer of 10014000 bytes # root->5 > [10.000000 -> 10.000760] node-1.acme.org -> node-1.acme.org transfer of 10001 bytes # 0->2 > [10.000000 -> 10.000760] node-0.acme.org -> node-1.acme.org transfer of 10004 bytes # 1->2 > [20.000760 -> 20.001440] node-1.acme.org -> node-1.acme.org transfer of 10002 bytes # 2->3 > [40.001441 -> 40.002121] node-0.acme.org -> node-0.acme.org transfer of 10029 bytes # 4->5 > [60.002121 -> 60.002801] node-0.acme.org -> node-0.acme.org transfer of 10005 bytes # 6->7 > [70.002801 -> 70.003561] node-0.acme.org -> node-1.acme.org transfer of 10000 bytes # 7->8 > [70.002801 -> 70.083593] node-0.acme.org -> node-0.acme.org transfer of 10014000 bytes # 7->end > [90.003561->100.003561] node-0.acme.org compute 10000000129.452715 flops # end $ rm -f ${srcdir:=.}/schedule-dotload/dag_with_good_schedule.trace SimGrid-3.14.159/examples/simdag/schedule-dotload/dag_with_bad_schedule.dot000644 001750 001750 00000002067 13031031306 030355 0ustar00mquinsonmquinson000000 000000 digraph G { end [size="10000000129.452715"]; 0 [size="10000000129.452715" category="taskA" performer="1" order="1"]; 1 [size="10000000131.133657" category="taskA" performer="0"]; 2 [size="10000000121.12487" category="taskA" performer="1" order="1"]; 3 [size="10000000230.608025" category="taskA" order="1"]; 4 [size="10000000004.994019" category="taskB" performer="1" order="2"]; 5 [size="10000000046.016401" category="taskB" performer="1" order="3"]; 6 [size="10000000091.598791" category="taskB" performer="1" order="4"]; 7 [size="10000000040.679438" category="taskB" performer="1" order="5"]; 8 [size="10000000250.490017" category="taskC" performer="1" order="6"]; 9 [size="10000000079.267649" category="taskC" performer="1" order="9"]; 0->1 [size="10001.389601075407" category="oi"]; 1->2 [size="10004.164631264117"]; 2->3 [size="10001.781644976922"]; 3->4 [size="-1"]; 4->5 [size="10029.262823275711"]; 5->6 [size="0.0"]; 6->7 [size="10004.920415194067"]; 7->8 [size="10000.234048984707"]; 8->9 ; 7->end [size="10014000.0"]; root->5 [size="10014000.0"]; } SimGrid-3.14.159/examples/simdag/schedule-dotload/dag_with_good_schedule.dot000644 001750 001750 00000002207 13031031306 030553 0ustar00mquinsonmquinson000000 000000 digraph G { end [size="10000000129.452715" performer="0" order="7"]; root [performer="0" order="1"]; 0 [size="10000000129.452715" category="taskA" performer="1" order="1"]; 1 [size="10000000131.133657" category="taskA" performer="0" order="2"]; 2 [size="10000000121.12487" category="taskA" performer="1" order="2"]; 3 [size="10000000230.608025" category="taskA" performer="1" order="3"]; 4 [size="10000000004.994019" category="taskB" performer="0" order="3"]; 5 [size="10000000046.016401" category="taskB" performer="0" order="4"]; 6 [size="10000000091.598791" category="taskB" performer="0" order="5"]; 7 [size="10000000040.679438" category="taskB" performer="0" order="6"]; 8 [size="10000000250.490017" category="taskC" performer="1" order="4"]; 9 [size="10000000079.267649" category="taskC" performer="1" order="5"]; 0->2 [size="10001.389601075407" category="oi"]; 1->2 [size="10004.164631264117"]; 2->3 [size="10001.781644976922"]; 3->4 [size="-1"]; 4->5 [size="10029.262823275711"]; 5->6 [size="0.0"]; 6->7 [size="10004.920415194067"]; 7->8 [size="10000.234048984707"]; 8->9 ; 7->end [size="10014000.0"]; root->5 [size="10014000.0"]; } SimGrid-3.14.159/examples/simdag/schedule-dotload/sd_schedule-dotload.c000644 001750 001750 00000005643 13031031300 027444 0ustar00mquinsonmquinson000000 000000 /* simple test trying to load a DOT file. */ /* Copyright (c) 2010-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include "simgrid/simdag.h" #include #include XBT_LOG_NEW_DEFAULT_CATEGORY(test, "Logging specific to this SimDag example"); int main(int argc, char **argv) { xbt_dynar_t dot; unsigned int cursor; SD_task_t task; /* initialization of SD */ SD_init(&argc, argv); /* Check our arguments */ xbt_assert(argc > 2, "Usage: %s platform_file dot_file [trace_file]" "example: %s ../2clusters.xml dag.dot dag.mytrace", argv[0], argv[0]); /* creation of the environment */ SD_create_environment(argv[1]); /* load the DOT file and schedule tasks */ dot = SD_dotload_with_sched(argv[2]); if(!dot){ XBT_CRITICAL("The dot file with the provided scheduling is wrong," " more information with the option : --log=sd_dotparse.thres:verbose"); SD_exit(); exit(2); } char *tracefilename; char *last = strrchr(argv[2], '.'); tracefilename = bprintf("%.*s.trace", (int) (last == NULL ? strlen(argv[2]) : last - argv[2]),argv[2]); if (argc == 4) tracefilename = xbt_strdup(argv[3]); /* Display all the tasks */ XBT_INFO("------------------- Display all tasks of the loaded DAG ---------------------------"); xbt_dynar_foreach(dot, cursor, task) { SD_task_dump(task); } XBT_INFO("------------------- Run the schedule ---------------------------"); SD_simulate(-1); XBT_INFO("------------------- Produce the trace file---------------------------"); XBT_INFO("Producing the trace of the run into %s", basename(tracefilename)); FILE *out = fopen(tracefilename, "w"); xbt_assert(out, "Cannot write to %s", tracefilename); free(tracefilename); xbt_dynar_foreach(dot, cursor, task) { int kind = SD_task_get_kind(task); sg_host_t *wsl = SD_task_get_workstation_list(task); switch (kind) { case SD_TASK_COMP_SEQ: fprintf(out, "[%f->%f] %s compute %f flops # %s\n", SD_task_get_start_time(task), SD_task_get_finish_time(task), sg_host_get_name(wsl[0]), SD_task_get_amount(task), SD_task_get_name(task)); break; case SD_TASK_COMM_E2E: fprintf(out, "[%f -> %f] %s -> %s transfer of %.0f bytes # %s\n", SD_task_get_start_time(task), SD_task_get_finish_time(task), sg_host_get_name(wsl[0]), sg_host_get_name(wsl[1]), SD_task_get_amount(task), SD_task_get_name(task)); break; default: xbt_die("Task %s is of unknown kind %d", SD_task_get_name(task), SD_task_get_kind(task)); } SD_task_destroy(task); } fclose(out); xbt_dynar_free_container(&dot); /* exit */ SD_exit(); return 0; } SimGrid-3.14.159/examples/simdag/CMakeLists.txt000644 001750 001750 00000006202 13031031274 022714 0ustar00mquinsonmquinson000000 000000 foreach(x availability daxload fail typed_tasks properties throttling scheduling) add_executable (sd_${x} ${x}/sd_${x}.c) target_link_libraries(sd_${x} simgrid) set_target_properties(sd_${x} PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/${x}) set(examples_src ${examples_src} ${CMAKE_CURRENT_SOURCE_DIR}/${x}/sd_${x}.c) set(tesh_files ${tesh_files} ${CMAKE_CURRENT_SOURCE_DIR}/${x}/sd_${x}.tesh) endforeach() foreach(x test) add_executable (sd_${x} ${x}/sd_${x}.cpp) target_link_libraries(sd_${x} simgrid) set_target_properties(sd_${x} PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/${x}) set(examples_src ${examples_src} ${CMAKE_CURRENT_SOURCE_DIR}/${x}/sd_${x}.cpp) set(tesh_files ${tesh_files} ${CMAKE_CURRENT_SOURCE_DIR}/${x}/sd_${x}.tesh) endforeach() foreach(x dag-dotload ptg-dotload schedule-dotload) if(HAVE_GRAPHVIZ) add_executable (sd_${x} ${x}/sd_${x}.c) target_link_libraries(sd_${x} simgrid) set_target_properties(sd_${x} PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/${x}) endif() set(examples_src ${examples_src} ${CMAKE_CURRENT_SOURCE_DIR}/${x}/sd_${x}.c) set(tesh_files ${tesh_files} ${CMAKE_CURRENT_SOURCE_DIR}/${x}/sd_${x}.tesh) endforeach() add_executable (goal_test goal/goal_test.c) target_link_libraries(goal_test simgrid) set_target_properties(goal_test PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/goal) set(tesh_files ${tesh_files} PARENT_SCOPE) set(examples_src ${examples_src} ${CMAKE_CURRENT_SOURCE_DIR}/goal/goal_test.c PARENT_SCOPE) set(xml_files ${xml_files} ${CMAKE_CURRENT_SOURCE_DIR}/scheduling/Montage_25.xml ${CMAKE_CURRENT_SOURCE_DIR}/daxload/simple_dax_with_cycle.xml ${CMAKE_CURRENT_SOURCE_DIR}/daxload/smalldax.xml PARENT_SCOPE) set(txt_files ${txt_files} ${CMAKE_CURRENT_SOURCE_DIR}/dag-dotload/dag_with_cycle.dot ${CMAKE_CURRENT_SOURCE_DIR}/dag-dotload/dag.dot ${CMAKE_CURRENT_SOURCE_DIR}/ptg-dotload/ptg.dot ${CMAKE_CURRENT_SOURCE_DIR}/schedule-dotload/dag_with_bad_schedule.dot ${CMAKE_CURRENT_SOURCE_DIR}/schedule-dotload/dag_with_good_schedule.dot ${CMAKE_CURRENT_SOURCE_DIR}/scheduling/expected_output.jed PARENT_SCOPE) foreach(x availability daxload fail typed_tasks properties throttling scheduling test) ADD_TESH(simdag-${x} --setenv srcdir=${CMAKE_HOME_DIRECTORY}/examples/simdag --cd ${CMAKE_BINARY_DIR}/examples/simdag ${CMAKE_HOME_DIRECTORY}/examples/simdag/${x}/sd_${x}.tesh) endforeach() if(HAVE_GRAPHVIZ) foreach(x dag-dotload ptg-dotload schedule-dotload) ADD_TESH(simdag-${x} --setenv srcdir=${CMAKE_HOME_DIRECTORY}/examples/simdag --cd ${CMAKE_BINARY_DIR}/examples/simdag ${CMAKE_HOME_DIRECTORY}/examples/simdag/${x}/sd_${x}.tesh) endforeach() endif() SimGrid-3.14.159/examples/simdag/dag-dotload/000755 001750 001750 00000000000 13031031306 022327 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/simdag/dag-dotload/sd_dag-dotload.tesh000644 001750 001750 00000030345 13031031304 026064 0ustar00mquinsonmquinson000000 000000 #! ./tesh p Test the loader of DAG written in the DOT format $ $SG_TEST_EXENV ${bindir:=.}/dag-dotload/sd_dag-dotload --log=no_loc ${srcdir:=.}/../platforms/two_clusters.xml ${srcdir:=.}/dag-dotload/dag.dot > [0.000000] [xbt_cfg/INFO] Switching to the L07 model to handle parallel tasks. > [0.000000] [test/INFO] ------------------- Display all tasks of the loaded DAG --------------------------- > [0.000000] [sd_task/INFO] Displaying task root > [0.000000] [sd_task/INFO] - state: schedulable not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 0 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 0 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 0 > [0.000000] [sd_task/INFO] root->5 > [0.000000] [sd_task/INFO] Displaying task 0 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000129 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] root > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 0->1 > [0.000000] [sd_task/INFO] Displaying task 1 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000131 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 0->1 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 1->2 > [0.000000] [sd_task/INFO] Displaying task 2 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000121 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 1->2 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 2->3 > [0.000000] [sd_task/INFO] Displaying task 3 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000231 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 2->3 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 4 > [0.000000] [sd_task/INFO] Displaying task 4 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000005 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 3 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 4->5 > [0.000000] [sd_task/INFO] Displaying task 5 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000046 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 2 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 4->5 > [0.000000] [sd_task/INFO] root->5 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 6 > [0.000000] [sd_task/INFO] Displaying task 6 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000092 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 5 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 6->7 > [0.000000] [sd_task/INFO] Displaying task 7 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000041 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 6->7 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 7->8 > [0.000000] [sd_task/INFO] 7->end > [0.000000] [sd_task/INFO] Displaying task 8 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000250 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 7->8 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 9 > [0.000000] [sd_task/INFO] Displaying task 9 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000079 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 8 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] end > [0.000000] [sd_task/INFO] Displaying task 0->1 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10001 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 0 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 1 > [0.000000] [sd_task/INFO] Displaying task 1->2 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10004 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 1 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 2 > [0.000000] [sd_task/INFO] Displaying task 2->3 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10002 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 2 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 3 > [0.000000] [sd_task/INFO] Displaying task 4->5 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10029 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 4 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 5 > [0.000000] [sd_task/INFO] Displaying task 6->7 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10005 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 6 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 7 > [0.000000] [sd_task/INFO] Displaying task 7->8 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10000 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 7 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 8 > [0.000000] [sd_task/INFO] Displaying task 7->end > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10014000 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 7 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] end > [0.000000] [sd_task/INFO] Displaying task root->5 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: end-to-end communication > [0.000000] [sd_task/INFO] - amount: 10014000 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] root > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] 5 > [0.000000] [sd_task/INFO] Displaying task end > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: sequential computation > [0.000000] [sd_task/INFO] - amount: 10000000129 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 2 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] 9 > [0.000000] [sd_task/INFO] 7->end > [0.000000] [test/INFO] ------------------- Schedule tasks --------------------------- > [0.000000] [test/INFO] ------------------- Run the schedule --------------------------- > [110.005082] [test/INFO] ------------------- Produce the trace file--------------------------- > [110.005082] [test/INFO] Producing the trace of the run into dag.trace $ cat ${srcdir:=.}/dag-dotload/dag.trace > [0.000000->0.000000] node-0.acme.org compute 0.000000 flops # root > [0.000000->10.000000] node-1.acme.org compute 10000000129.452715 flops # 0 > [10.001680->20.001680] node-10.acme.org compute 10000000131.133657 flops # 1 > [20.002360->30.002360] node-11.acme.org compute 10000000121.124870 flops # 2 > [30.003040->40.003041] node-12.acme.org compute 10000000230.608025 flops # 3 > [40.003041->50.003041] node-13.acme.org compute 10000000004.994019 flops # 4 > [50.003721->60.003721] node-14.acme.org compute 10000000046.016401 flops # 5 > [60.003721->70.003721] node-15.acme.org compute 10000000091.598791 flops # 6 > [70.004401->80.004401] node-16.acme.org compute 10000000040.679438 flops # 7 > [80.005081->90.005081] node-17.acme.org compute 10000000250.490017 flops # 8 > [90.005081->100.005081] node-18.acme.org compute 10000000079.267649 flops # 9 > [10.000000 -> 10.001680] node-1.acme.org -> node-10.acme.org transfer of 10001 bytes # 0->1 > [20.001680 -> 20.002360] node-10.acme.org -> node-11.acme.org transfer of 10004 bytes # 1->2 > [30.002360 -> 30.003040] node-11.acme.org -> node-12.acme.org transfer of 10002 bytes # 2->3 > [50.003041 -> 50.003721] node-13.acme.org -> node-14.acme.org transfer of 10029 bytes # 4->5 > [70.003721 -> 70.004401] node-15.acme.org -> node-16.acme.org transfer of 10005 bytes # 6->7 > [80.004401 -> 80.005081] node-16.acme.org -> node-17.acme.org transfer of 10000 bytes # 7->8 > [80.004401 -> 80.086113] node-16.acme.org -> node-0.acme.org transfer of 10014000 bytes # 7->end > [0.000000 -> 0.081712] node-0.acme.org -> node-14.acme.org transfer of 10014000 bytes # root->5 > [100.005081->110.005082] node-0.acme.org compute 10000000129.452715 flops # end $ rm -f ${srcdir:=.}/dag-dotload/dag.trace ${srcdir:=.}/dot.dot ! expect return 2 $ $SG_TEST_EXENV ${bindir:=.}/dag-dotload/sd_dag-dotload --log=no_loc ${srcdir:=.}/../platforms/two_clusters.xml ${srcdir:=.}/dag-dotload/dag_with_cycle.dot > [0.000000] [xbt_cfg/INFO] Switching to the L07 model to handle parallel tasks. > [0.000000] [sd_daxparse/WARNING] the task root is not marked > [0.000000] [sd_daxparse/WARNING] the task 1 is in a cycle > [0.000000] [sd_daxparse/WARNING] the task 2 is in a cycle > [0.000000] [sd_daxparse/WARNING] the task 3 is in a cycle > [0.000000] [sd_daxparse/WARNING] the task 4 is in a cycle > [0.000000] [sd_daxparse/WARNING] the task 5 is in a cycle > [0.000000] [sd_daxparse/WARNING] the task 6 is in a cycle > [0.000000] [sd_dotparse/ERROR] The DOT described in dag_with_cycle.dot is not a DAG. It contains a cycle. > [0.000000] [test/CRITICAL] No dot loaded. Do you have a cycle in your graph? SimGrid-3.14.159/examples/simdag/dag-dotload/sd_dag-dotload.c000644 001750 001750 00000006744 13031031300 025345 0ustar00mquinsonmquinson000000 000000 /* simple test trying to load a DOT file. */ /* Copyright (c) 2010-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/simdag.h" #include "xbt/log.h" #include #include XBT_LOG_NEW_DEFAULT_CATEGORY(test, "Logging specific to this SimDag example"); int main(int argc, char **argv) { xbt_dynar_t dot; unsigned int cursor; SD_task_t task; /* initialization of SD */ SD_init(&argc, argv); /* Check our arguments */ xbt_assert(argc > 2, "Usage: %s platform_file dot_file [trace_file]" "example: %s ../2clusters.xml dag.dot dag.mytrace", argv[0], argv[0]); /* creation of the environment */ SD_create_environment(argv[1]); /* load the DOT file */ dot = SD_dotload(argv[2]); if(dot == NULL){ XBT_CRITICAL("No dot loaded. Do you have a cycle in your graph?"); SD_exit(); exit(2); } char *tracefilename; char *last = strrchr(argv[2], '.'); tracefilename = bprintf("%.*s.trace", (int) (last == NULL ? strlen(argv[2]) : last - argv[2]),argv[2]); if (argc == 4) tracefilename = xbt_strdup(argv[3]); /* Display all the tasks */ XBT_INFO("------------------- Display all tasks of the loaded DAG ---------------------------"); xbt_dynar_foreach(dot, cursor, task) { SD_task_dump(task); } FILE *dotout = fopen("dot.dot", "w"); fprintf(dotout, "digraph A {\n"); xbt_dynar_foreach(dot, cursor, task) { SD_task_dotty(task, dotout); } fprintf(dotout, "}\n"); fclose(dotout); /* Schedule them all on the first workstation */ XBT_INFO("------------------- Schedule tasks ---------------------------"); sg_host_t *hosts = sg_host_list(); int count = sg_host_count(); xbt_dynar_foreach(dot, cursor, task) { if (SD_task_get_kind(task) == SD_TASK_COMP_SEQ) { if (!strcmp(SD_task_get_name(task), "end")) SD_task_schedulel(task, 1, hosts[0]); else SD_task_schedulel(task, 1, hosts[cursor % count]); } } xbt_free(hosts); XBT_INFO("------------------- Run the schedule ---------------------------"); SD_simulate(-1); XBT_INFO("------------------- Produce the trace file---------------------------"); XBT_INFO("Producing the trace of the run into %s", basename(tracefilename)); FILE *out = fopen(tracefilename, "w"); xbt_assert(out, "Cannot write to %s", tracefilename); free(tracefilename); xbt_dynar_foreach(dot, cursor, task) { int kind = SD_task_get_kind(task); sg_host_t *wsl = SD_task_get_workstation_list(task); switch (kind) { case SD_TASK_COMP_SEQ: fprintf(out, "[%f->%f] %s compute %f flops # %s\n", SD_task_get_start_time(task), SD_task_get_finish_time(task), sg_host_get_name(wsl[0]), SD_task_get_amount(task), SD_task_get_name(task)); break; case SD_TASK_COMM_E2E: fprintf(out, "[%f -> %f] %s -> %s transfer of %.0f bytes # %s\n", SD_task_get_start_time(task), SD_task_get_finish_time(task), sg_host_get_name(wsl[0]), sg_host_get_name(wsl[1]), SD_task_get_amount(task), SD_task_get_name(task)); break; default: xbt_die("Task %s is of unknown kind %d", SD_task_get_name(task), SD_task_get_kind(task)); } SD_task_destroy(task); } xbt_dynar_free_container(&dot); fclose(out); /* exit */ SD_exit(); return 0; } SimGrid-3.14.159/examples/simdag/dag-dotload/dag.dot000644 001750 001750 00000001530 13031031306 023571 0ustar00mquinsonmquinson000000 000000 digraph G { end [size="10000000129.452715"]; 0 [size="10000000129.452715" category="taskA"]; 1 [size="10000000131.133657" category="taskA"]; 2 [size="10000000121.12487" category="taskA"]; 3 [size="10000000230.608025" category="taskA"]; 4 [size="10000000004.994019" category="taskB"]; 5 [size="10000000046.016401" category="taskB"]; 6 [size="10000000091.598791" category="taskB"]; 7 [size="10000000040.679438" category="taskB"]; 8 [size="10000000250.490017" category="taskC"]; 9 [size="10000000079.267649" category="taskC"]; 0->1 [size="10001.389601075407" category="oi"]; 1->2 [size="10004.164631264117"]; 2->3 [size="10001.781644976922"]; 3->4 [size="-1"]; 4->5 [size="10029.262823275711"]; 5->6 [size="0.0"]; 6->7 [size="10004.920415194067"]; 7->8 [size="10000.234048984707"]; 8->9 ; 7->end [size="10014000.0"]; root->5 [size="10014000.0"]; } SimGrid-3.14.159/examples/simdag/dag-dotload/dag_with_cycle.dot000644 001750 001750 00000001536 13031031306 026011 0ustar00mquinsonmquinson000000 000000 digraph G { end [size="10000000129.452715"]; 0 [size="10000000129.452715" category="taskA"]; 1 [size="10000000131.133657" category="taskA"]; 2 [size="10000000121.12487" category="taskA"]; 3 [size="10000000230.608025" category="taskA"]; 4 [size="10000000004.994019" category="taskB"]; 5 [size="10000000046.016401" category="taskB"]; 6 [size="10000000091.598791" category="taskB"]; 7 [size="10000000040.679438" category="taskB"]; 8 [size="10000000250.490017" category="taskC"]; 9 [size="10000000079.267649" category="taskC"]; 0->2 [size="10001.389601075407" category="oi"]; 1->2 [size="10004.164631264117"]; 2->3 [size="10001.781644976922"]; 3->4 [size="-1"]; 4->5 [size="10029.262823275711"]; 5->6 [size="0.0"]; 6->7 [size="10004.920415194067"]; 7->8 [size="10000.234048984707"]; 6->1; 8->9 ; 7->end [size="10014000.0"]; root->5 [size="10014000.0"]; } SimGrid-3.14.159/examples/simdag/properties/000755 001750 001750 00000000000 13031031304 022342 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/simdag/properties/sd_properties.c000644 001750 001750 00000004561 13031031300 025372 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/simdag.h" XBT_LOG_NEW_DEFAULT_CATEGORY(test, "Property test"); int main(int argc, char **argv) { xbt_dict_cursor_t cursor = NULL; char *key, *data; char noexist[] = "NoProp"; const char *value; char exist[] = "Hdd"; /* SD initialization */ SD_init(&argc, argv); xbt_assert(argc > 1, "Usage: %s platform_file\n\tExample: %s ../../platforms/prop.xml", argv[0], argv[0]); SD_create_environment(argv[1]); /* init of platform elements */ sg_host_t h1 = sg_host_by_name("host1"); sg_host_t h2 = sg_host_by_name("host2"); const char *name1 = sg_host_get_name(h1); const char *name2 = sg_host_get_name(h2); /* Get the property list of 'host1' */ XBT_INFO("Property list for host %s", name1); xbt_dict_t props = sg_host_get_properties(h1); /* Trying to set a new property */ xbt_dict_set(props, "NewProp", strdup("newValue"), NULL); /* Print the properties of 'host1' */ xbt_dict_foreach(props, cursor, key, data) { XBT_INFO("\tProperty: %s has value: %s", key, data); } /* Try to get a property that does not exist */ value = sg_host_get_property_value(h1, noexist); XBT_INFO("\tProperty: %s has value: %s", noexist, value?value:"Undefined (NULL)"); /* Get the property list of 'host2' */ XBT_INFO("Property list for host %s", name2); props = sg_host_get_properties(h2); /* Print the properties of 'host2' */ xbt_dict_foreach(props, cursor, key, data) { XBT_INFO("\tProperty: %s on host: %s", key, data); } /* Modify an existing property test. First check it exists */ XBT_INFO("Modify an existing property"); value = sg_host_get_property_value(h2, exist); if (value == NULL) XBT_INFO("\tProperty: %s is undefined", exist); else { XBT_INFO("\tProperty: %s old value: %s", exist, value); xbt_dict_set(props, exist, strdup("250"), NULL); } /* Test if we have changed the value */ value = sg_host_get_property_value(h2, exist); XBT_INFO("\tProperty: %s new value: %s", exist, value?value:"Undefined (NULL)"); /* Test if properties are displayed by sg_host_dump */ sg_host_dump(h2); SD_exit(); return 0; } SimGrid-3.14.159/examples/simdag/properties/sd_properties.tesh000755 001750 001750 00000001770 13031031304 026121 0ustar00mquinsonmquinson000000 000000 #! ./tesh p Simple test of simdag with properties $ $SG_TEST_EXENV properties/sd_properties ${srcdir:=.}/../platforms/prop.xml > [0.000000] [xbt_cfg/INFO] Switching to the L07 model to handle parallel tasks. > [0.000000] [test/INFO] Property list for host host1 > [0.000000] [test/INFO] Property: mem has value: 4 > [0.000000] [test/INFO] Property: NewProp has value: newValue > [0.000000] [test/INFO] Property: Hdd has value: 180 > [0.000000] [test/INFO] Property: NoProp has value: Undefined (NULL) > [0.000000] [test/INFO] Property list for host host2 > [0.000000] [test/INFO] Property: Hdd on host: 120 > [0.000000] [test/INFO] Modify an existing property > [0.000000] [test/INFO] Property: Hdd old value: 120 > [0.000000] [test/INFO] Property: Hdd new value: 250 > [0.000000] [sg_host/INFO] Displaying host host2 > [0.000000] [sg_host/INFO] - speed: 1000000000 > [0.000000] [sg_host/INFO] - available speed: 1.00 > [0.000000] [sg_host/INFO] - properties: > [0.000000] [sg_host/INFO] Hdd->250 SimGrid-3.14.159/examples/simdag/ptg-dotload/000755 001750 001750 00000000000 13031031306 022366 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/simdag/ptg-dotload/sd_ptg-dotload.c000644 001750 001750 00000003526 13031031300 025436 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/simdag.h" #include "xbt/log.h" XBT_LOG_NEW_DEFAULT_CATEGORY(test, "Logging specific to this SimDag example"); /* simple test trying to load a Parallel Task Graph (PTG) as a DOT file. */ int main(int argc, char **argv){ xbt_dynar_t dot; unsigned int cursor; SD_task_t task; /* initialization of SD */ SD_init(&argc, argv); /* Check our arguments */ xbt_assert (argc > 1,"Usage: %s platform_file dot_file example: %s ../2clusters.xml ptg.dot", argv[0], argv[0]); /* creation of the environment */ SD_create_environment(argv[1]); /* load the DOT file */ dot = SD_PTG_dotload(argv[2]); if(dot == NULL){ SD_exit(); xbt_die("No dot load may be you have a cycle in your graph"); } /* Display all the tasks */ XBT_INFO("------------------- Display all tasks of the loaded DAG ---------------------------"); xbt_dynar_foreach(dot, cursor, task) { SD_task_dump(task); } /* Schedule them all on all the first host*/ XBT_INFO("------------------- Schedule tasks ---------------------------"); sg_host_t *hosts = sg_host_list(); int count = sg_host_count(); xbt_dynar_foreach(dot, cursor, task) { if (SD_task_get_kind(task) == SD_TASK_COMP_PAR_AMDAHL) { SD_task_schedulev(task, count, hosts); } } xbt_free(hosts); XBT_INFO("------------------- Run the schedule ---------------------------"); SD_simulate(-1); XBT_INFO("Makespan: %f", SD_get_clock()); xbt_dynar_foreach(dot, cursor, task) { SD_task_destroy(task); } xbt_dynar_free_container(&dot); /* exit */ SD_exit(); return 0; } SimGrid-3.14.159/examples/simdag/ptg-dotload/sd_ptg-dotload.tesh000644 001750 001750 00000012005 13031031304 026153 0ustar00mquinsonmquinson000000 000000 #! ./tesh p Test the loader of PTG (Parallel Task Graph) written in the DOT format $ $SG_TEST_EXENV ${bindir:=.}/ptg-dotload/sd_ptg-dotload ${srcdir:=.}/../platforms/two_clusters.xml ${srcdir:=.}/ptg-dotload/ptg.dot > [0.000000] [xbt_cfg/INFO] Switching to the L07 model to handle parallel tasks. > [0.000000] [test/INFO] ------------------- Display all tasks of the loaded DAG --------------------------- > [0.000000] [sd_task/INFO] Displaying task root > [0.000000] [sd_task/INFO] - state: schedulable not runnable > [0.000000] [sd_task/INFO] - kind: parallel computation following Amdahl's law > [0.000000] [sd_task/INFO] - amount: 0 > [0.000000] [sd_task/INFO] - alpha: 0.00 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 0 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] root->c1 > [0.000000] [sd_task/INFO] root->c2 > [0.000000] [sd_task/INFO] Displaying task c1 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: parallel computation following Amdahl's law > [0.000000] [sd_task/INFO] - amount: 1000000000 > [0.000000] [sd_task/INFO] - alpha: 0.20 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] root->c1 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] c1->c3 > [0.000000] [sd_task/INFO] Displaying task c2 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: parallel computation following Amdahl's law > [0.000000] [sd_task/INFO] - amount: 5000000000 > [0.000000] [sd_task/INFO] - alpha: 0.50 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] root->c2 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] c3 > [0.000000] [sd_task/INFO] Displaying task c3 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: parallel computation following Amdahl's law > [0.000000] [sd_task/INFO] - amount: 2000000000 > [0.000000] [sd_task/INFO] - alpha: 0.00 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 2 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] c2 > [0.000000] [sd_task/INFO] c1->c3 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] c3->end > [0.000000] [sd_task/INFO] Displaying task c1->c3 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: MxN data redistribution assuming 1D block distribution > [0.000000] [sd_task/INFO] - amount: 500000000 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] c1 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] c3 > [0.000000] [sd_task/INFO] Displaying task c3->end > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: MxN data redistribution assuming 1D block distribution > [0.000000] [sd_task/INFO] - amount: 200000000 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] c3 > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] end > [0.000000] [sd_task/INFO] Displaying task root->c1 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: MxN data redistribution assuming 1D block distribution > [0.000000] [sd_task/INFO] - amount: 200000000 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] root > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] c1 > [0.000000] [sd_task/INFO] Displaying task root->c2 > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: MxN data redistribution assuming 1D block distribution > [0.000000] [sd_task/INFO] - amount: 100000000 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] root > [0.000000] [sd_task/INFO] - post-dependencies: > [0.000000] [sd_task/INFO] c2 > [0.000000] [sd_task/INFO] Displaying task end > [0.000000] [sd_task/INFO] - state: not scheduled not runnable > [0.000000] [sd_task/INFO] - kind: parallel computation following Amdahl's law > [0.000000] [sd_task/INFO] - amount: 0 > [0.000000] [sd_task/INFO] - alpha: 0.00 > [0.000000] [sd_task/INFO] - Dependencies to satisfy: 1 > [0.000000] [sd_task/INFO] - pre-dependencies: > [0.000000] [sd_task/INFO] c3->end > [0.000000] [test/INFO] ------------------- Schedule tasks --------------------------- > [0.000000] [test/INFO] ------------------- Run the schedule --------------------------- > [3.126200] [test/INFO] Makespan: 3.126200 SimGrid-3.14.159/examples/simdag/ptg-dotload/ptg.dot000644 001750 001750 00000000336 13031031306 023672 0ustar00mquinsonmquinson000000 000000 digraph G { c1 [size="1e9", alpha="0.2"]; c2 [size="5e9", alpha="0.5"]; c3 [size="2e9"]; root->c1 [size="2e8"]; root->c2 [size="1e8"]; c1->c3 [size="5e8"]; c2->c3 [size="-1."]; c3->end [size="2e8"]; } SimGrid-3.14.159/examples/simdag/fail/000755 001750 001750 00000000000 13031031304 021061 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/simdag/fail/sd_fail.c000644 001750 001750 00000005652 13031031300 022632 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2010, 2012-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/simdag.h" XBT_LOG_NEW_DEFAULT_CATEGORY(sd_fail, "Logging specific to this SimDag example"); int main(int argc, char **argv) { double computation_amount[1]; double communication_amount[2] = { 0 }; sg_host_t hosts[1]; /* initialization of SD */ SD_init(&argc, argv); /* creation of the environment */ SD_create_environment(argv[1]); /* creation of a single task that will poorly fail when the workstation will stop */ XBT_INFO("First test: COMP_SEQ task"); SD_task_t task = SD_task_create_comp_seq("Poor task", NULL, 2e10); SD_task_watch(task, SD_FAILED); SD_task_watch(task, SD_DONE); XBT_INFO("Schedule task '%s' on 'Faulty Host'", SD_task_get_name(task)); SD_task_schedulel(task, 1, sg_host_by_name("Faulty Host")); xbt_dynar_t tasks = SD_simulate(-1.0); xbt_dynar_free(&tasks); SD_task_dump(task); XBT_INFO("Task '%s' has failed. %.f flops remain to be done", SD_task_get_name(task), SD_task_get_remaining_amount(task)); XBT_INFO("let's unschedule task '%s' and reschedule it on the 'Safe Host'", SD_task_get_name(task)); SD_task_unschedule(task); SD_task_schedulel(task, 1, sg_host_by_name("Safe Host")); XBT_INFO("Run the simulation again"); tasks = SD_simulate(-1.0); xbt_dynar_free(&tasks); SD_task_dump(task); XBT_INFO("Task '%s' start time: %f, finish time: %f", SD_task_get_name(task), SD_task_get_start_time(task), SD_task_get_finish_time(task)); SD_task_destroy(task); XBT_INFO("Second test: NON TYPED task"); task = SD_task_create("Poor parallel task", NULL, 2e10); SD_task_watch(task, SD_FAILED); SD_task_watch(task, SD_DONE); computation_amount[0] = 2e10; XBT_INFO("Schedule task '%s' on 'Faulty Host'", SD_task_get_name(task)); hosts[0] = sg_host_by_name("Faulty Host"); SD_task_schedule(task, 1, hosts, computation_amount, communication_amount,-1); tasks = SD_simulate(-1.0); xbt_dynar_free(&tasks); SD_task_dump(task); XBT_INFO("Task '%s' has failed. %.f flops remain to be done", SD_task_get_name(task), SD_task_get_remaining_amount(task)); XBT_INFO("let's unschedule task '%s' and reschedule it on the 'Safe Host'", SD_task_get_name(task)); SD_task_unschedule(task); hosts[0] = sg_host_by_name("Safe Host"); SD_task_schedule(task, 1, hosts, computation_amount, communication_amount,-1); XBT_INFO("Run the simulation again"); tasks = SD_simulate(-1.0); xbt_dynar_free(&tasks); SD_task_dump(task); XBT_INFO("Task '%s' start time: %f, finish time: %f", SD_task_get_name(task), SD_task_get_start_time(task), SD_task_get_finish_time(task)); SD_task_destroy(task); SD_exit(); return 0; } SimGrid-3.14.159/examples/simdag/fail/sd_fail.tesh000644 001750 001750 00000004314 13031031304 023351 0ustar00mquinsonmquinson000000 000000 #! ./tesh p Test of the management of failed tasks simdag $ $SG_TEST_EXENV ${bindir:=.}/fail/sd_fail ${srcdir:=.}/../platforms/faulty_host.xml > [0.000000] [xbt_cfg/INFO] Switching to the L07 model to handle parallel tasks. > [0.000000] [sd_fail/INFO] First test: COMP_SEQ task > [0.000000] [sd_fail/INFO] Schedule task 'Poor task' on 'Faulty Host' > [10.000000] [sd_task/INFO] Displaying task Poor task > [10.000000] [sd_task/INFO] - state: not runnable failed > [10.000000] [sd_task/INFO] - kind: sequential computation > [10.000000] [sd_task/INFO] - amount: 20000000000 > [10.000000] [sd_task/INFO] - Dependencies to satisfy: 0 > [10.000000] [sd_fail/INFO] Task 'Poor task' has failed. 20000000000 flops remain to be done > [10.000000] [sd_fail/INFO] let's unschedule task 'Poor task' and reschedule it on the 'Safe Host' > [10.000000] [sd_fail/INFO] Run the simulation again > [50.000000] [sd_task/INFO] Displaying task Poor task > [50.000000] [sd_task/INFO] - state: not runnable done > [50.000000] [sd_task/INFO] - kind: sequential computation > [50.000000] [sd_task/INFO] - amount: 20000000000 > [50.000000] [sd_task/INFO] - Dependencies to satisfy: 0 > [50.000000] [sd_fail/INFO] Task 'Poor task' start time: 10.000000, finish time: 50.000000 > [50.000000] [sd_fail/INFO] Second test: NON TYPED task > [50.000000] [sd_fail/INFO] Schedule task 'Poor parallel task' on 'Faulty Host' > [60.000000] [sd_task/INFO] Displaying task Poor parallel task > [60.000000] [sd_task/INFO] - state: not runnable failed > [60.000000] [sd_task/INFO] - amount: 20000000000 > [60.000000] [sd_task/INFO] - Dependencies to satisfy: 0 > [60.000000] [sd_fail/INFO] Task 'Poor parallel task' has failed. 20000000000 flops remain to be done > [60.000000] [sd_fail/INFO] let's unschedule task 'Poor parallel task' and reschedule it on the 'Safe Host' > [60.000000] [sd_fail/INFO] Run the simulation again > [100.000000] [sd_task/INFO] Displaying task Poor parallel task > [100.000000] [sd_task/INFO] - state: not runnable done > [100.000000] [sd_task/INFO] - amount: 20000000000 > [100.000000] [sd_task/INFO] - Dependencies to satisfy: 0 > [100.000000] [sd_fail/INFO] Task 'Poor parallel task' start time: 60.000000, finish time: 100.000000 SimGrid-3.14.159/examples/simdag/goal/000755 001750 001750 00000000000 13031031300 021064 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/simdag/goal/goal_test.c000644 001750 001750 00000005337 13031031300 023221 0ustar00mquinsonmquinson000000 000000 /* Example of scatter communication, accepting a large amount of processes. * This based the experiment of Fig. 4 in http://hal.inria.fr/hal-00650233/ * That experiment is a comparison to the LogOPSim simulator, that takes * GOAL files as an input, thus the file name. But there is no actual link * to the GOAL formalism beside of this. */ /* Copyright (c) 2011-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include "simgrid/simdag.h" #include "xbt/xbt_os_time.h" XBT_LOG_NEW_DEFAULT_CATEGORY(goal, "The GOAL loader into SimDag"); typedef struct { int i, j, k; } s_bcast_task_t,*bcast_task_t; const sg_host_t* ws_list; int count = 0; xbt_dynar_t reclaimed; static void send_one(int from, int to) { //XBT_DEBUG("send_one(%d, %d)",from,to); if (count %100000 == 0) XBT_INFO("Sending task #%d",count); count++; bcast_task_t bt; if (!xbt_dynar_is_empty(reclaimed)) { bt = xbt_dynar_pop_as(reclaimed,bcast_task_t); } else { bt = xbt_new(s_bcast_task_t,1); } bt->i=from; bt->j=(from+to)/2; bt->k=to; SD_task_t task = SD_task_create_comm_e2e(NULL,bt,424242); XBT_DEBUG("Schedule task between %d and %d",bt->i,bt->j); SD_task_schedulel(task,2,ws_list[bt->i],ws_list[bt->j]); SD_task_watch(task,SD_DONE); } int main(int argc, char **argv) { xbt_os_timer_t timer = xbt_os_timer_new(); /* initialization of SD */ SD_init(&argc, argv); if (argc > 1) { SD_create_environment(argv[1]); } else { SD_create_environment("../../platforms/One_cluster_no_backbone.xml"); } ws_list = sg_host_list(); reclaimed = xbt_dynar_new(sizeof(bcast_task_t),xbt_free_ref); xbt_dynar_t done = NULL; xbt_os_cputimer_start(timer); send_one(0,sg_host_count()); do { if (done != NULL && !xbt_dynar_is_empty(done)) { unsigned int cursor; SD_task_t task; xbt_dynar_foreach(done, cursor, task) { bcast_task_t bt = SD_task_get_data(task); if (bt->i != bt->j -1) send_one(bt->i,bt->j); if (bt->j != bt->k -1) send_one(bt->j,bt->k); if (xbt_dynar_length(reclaimed)<100) { xbt_dynar_push_as(reclaimed,bcast_task_t,bt); } else { free(bt); } SD_task_destroy(task); } xbt_dynar_free(&done); } done=SD_simulate(-1); } while(!xbt_dynar_is_empty(done)); xbt_os_cputimer_stop(timer); printf("exec_time:%f\n", xbt_os_timer_elapsed(timer) ); xbt_dynar_free(&done); xbt_dynar_free(&reclaimed); SD_exit(); XBT_INFO("Done. Bailing out"); return 0; } SimGrid-3.14.159/examples/simdag/throttling/000755 001750 001750 00000000000 13031031304 022344 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/simdag/throttling/sd_throttling.tesh000644 001750 001750 00000001735 13031031304 026123 0ustar00mquinsonmquinson000000 000000 #! ./tesh p Modify the rate of communication tasks even when they are auto-scheduled $ $SG_TEST_EXENV ./throttling/sd_throttling ${srcdir:=.}/../platforms/two_clusters.xml > [0.000000] [xbt_cfg/INFO] Switching to the L07 model to handle parallel tasks. > [5.000000] [sd_comm_throttling/INFO] Simulation stopped after 5.0000 seconds > [5.000000] [sd_comm_throttling/INFO] Task 'Task A' start time: 0.000000, finish time: 5.000000 > [10.080600] [sd_comm_throttling/INFO] Simulation stopped after 10.0806 seconds > [10.080600] [sd_comm_throttling/INFO] Task 'Task B' start time: 5.000000, finish time: 5.080600 > [10.080600] [sd_comm_throttling/INFO] Task 'Task C' start time: 5.080600, finish time: 10.080600 > [15.241200] [sd_comm_throttling/INFO] Simulation stopped after 15.2412 seconds > [15.241200] [sd_comm_throttling/INFO] Task 'Task D' start time: 10.080600, finish time: 10.241200 > [15.241200] [sd_comm_throttling/INFO] Task 'Task E' start time: 10.241200, finish time: 15.241200 SimGrid-3.14.159/examples/simdag/throttling/sd_throttling.c000644 001750 001750 00000005570 13031031300 025377 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2010, 2012-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/simdag.h" XBT_LOG_NEW_DEFAULT_CATEGORY(sd_comm_throttling, "Logging specific to this SimDag example"); int main(int argc, char **argv) { unsigned int ctr; SD_task_t task; xbt_dynar_t changed_tasks; SD_init(&argc, argv); xbt_assert(argc > 1, "Usage: %s platform_file\n\nExample: %s two_clusters.xml", argv[0], argv[0]); SD_create_environment(argv[1]); sg_host_t *hosts = sg_host_list(); /* creation of some typed tasks and their dependencies */ /* chain of five tasks, three compute tasks with two data transfers in between */ SD_task_t taskA = SD_task_create_comp_seq("Task A", NULL, 5e9); SD_task_t taskB = SD_task_create_comm_e2e("Task B", NULL, 1e7); SD_task_t taskC = SD_task_create_comp_seq("Task C", NULL, 5e9); SD_task_t taskD = SD_task_create_comm_e2e("Task D", NULL, 1e7); SD_task_t taskE = SD_task_create_comp_seq("Task E", NULL, 5e9); SD_task_dependency_add(NULL, NULL, taskA, taskB); SD_task_dependency_add(NULL, NULL, taskB, taskC); SD_task_dependency_add(NULL, NULL, taskC, taskD); SD_task_dependency_add(NULL, NULL, taskD, taskE); /* Add watchpoints on completion of compute tasks */ SD_task_watch(taskA, SD_DONE); SD_task_watch(taskC, SD_DONE); SD_task_watch(taskE, SD_DONE); /* Auto-schedule the compute tasks on three different workstations */ /* Data transfer tasks taskB and taskD are automagically scheduled */ SD_task_schedulel(taskA, 1, hosts[0]); SD_task_schedulel(taskC, 1, hosts[1]); SD_task_schedulel(taskE, 1, hosts[0]); while (!xbt_dynar_is_empty((changed_tasks = SD_simulate(-1.0)))) { XBT_INFO("Simulation stopped after %.4f seconds", SD_get_clock()); xbt_dynar_foreach(changed_tasks, ctr, task) { XBT_INFO("Task '%s' start time: %f, finish time: %f", SD_task_get_name(task), SD_task_get_start_time(task), SD_task_get_finish_time(task)); } xbt_dynar_free(&changed_tasks); /* let throttle the communication for taskD if its parent is SD_DONE */ /* the bandwidth is 1.25e8, the data size is 1e7, and we want to throttle the bandwidth by a factor 2. * The rate is then 1.25e8/(2*1e7)=6.25 * Changing the rate is possible before the task execution starts (in SD_RUNNING state). */ if (SD_task_get_state(taskC) == SD_DONE && SD_task_get_state(taskD) < SD_RUNNING) SD_task_set_rate(taskD, 6.25); } XBT_DEBUG("Destroying tasks..."); SD_task_destroy(taskA); SD_task_destroy(taskB); SD_task_destroy(taskC); SD_task_destroy(taskD); SD_task_destroy(taskE); XBT_DEBUG("Tasks destroyed. Exiting SimDag..."); SD_exit(); xbt_free(hosts); return 0; } SimGrid-3.14.159/examples/simdag/test/000755 001750 001750 00000000000 13031031304 021125 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/simdag/test/sd_test.tesh000644 001750 001750 00000003641 13031031304 023463 0ustar00mquinsonmquinson000000 000000 #! ./tesh p Simple test of simdag $ $SG_TEST_EXENV ./test/sd_test ${srcdir:=.}/../platforms/small_platform.xml > [0.000000] [xbt_cfg/INFO] Switching to the L07 model to handle parallel tasks. > [0.000000] [sd_test/INFO] Computation time for 2000000.000000 flops on Jacquelin: 0.014563 > [0.000000] [sd_test/INFO] Computation time for 1000000.000000 flops on Fafard: 0.013107 > [0.000000] [sd_test/INFO] Route between Jacquelin and Fafard: > [0.000000] [sd_test/INFO] Link 145: latency = 0.000410, bandwidth = 2583375.000000 > [0.000000] [sd_test/INFO] Link 59: latency = 0.000371, bandwidth = 11845375.000000 > [0.000000] [sd_test/INFO] Link 56: latency = 0.029589, bandwidth = 21414750.000000 > [0.000000] [sd_test/INFO] Link 54: latency = 0.035083, bandwidth = 15376875.000000 > [0.000000] [sd_test/INFO] Link 17: latency = 0.000137, bandwidth = 118682500.000000 > [0.000000] [sd_test/INFO] Link 16: latency = 0.000514, bandwidth = 34285625.000000 > [0.000000] [sd_test/INFO] Link 10: latency = 0.000514, bandwidth = 34285625.000000 > [0.000000] [sd_test/INFO] Link 6: latency = 0.000060, bandwidth = 41279125.000000 > [0.000000] [sd_test/INFO] Link 9: latency = 0.001462, bandwidth = 7209750.000000 > [0.000000] [sd_test/INFO] Link 79: latency = 0.000156, bandwidth = 8427250.000000 > [0.000000] [sd_test/INFO] Link 78: latency = 0.000278, bandwidth = 27946250.000000 > [0.000000] [sd_test/INFO] Route latency = 0.068575, route bandwidth = 2583375.000000 > [0.000000] [sd_test/INFO] Communication time for 2000000.000000 bytes between Jacquelin and Fafard: 0.842756 > [0.000000] [sd_test/INFO] Communication time for 3000000.000000 bytes between Fafard and Jacquelin: 1.229846 > [0.000000] [sd_test/INFO] Estimated time for 'Task D': 1.242953 > [4.008055] [sd_test/INFO] Task 'Task B' start time: 2.004027, finish time: 4.008055 > [4.008055] [sd_test/INFO] Task 'Task D' start time: 0.000000, finish time: 2.004027 SimGrid-3.14.159/examples/simdag/test/sd_test.cpp000644 001750 001750 00000012762 13031031300 023302 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/simdag.h" #include "xbt/ex.h" #include #include "xbt/log.h" #include #include XBT_LOG_NEW_DEFAULT_CATEGORY(sd_test, "Logging specific to this SimDag example"); int main(int argc, char **argv) { sg_host_t host_list[2]; double computation_amount[2]; double communication_amount[4] = { 0 }; /* initialization of SD */ SD_init(&argc, argv); xbt_assert(argc > 1, "Usage: %s platform_file\n\nExample: %s two_clusters.xml", argv[0], argv[0]); SD_create_environment(argv[1]); /* test the estimation functions */ const sg_host_t *hosts = sg_host_list(); sg_host_t h1 = hosts[4]; sg_host_t h2 = hosts[2]; const char *name1 = sg_host_get_name(h1); const char *name2 = sg_host_get_name(h2); double comp_amount1 = 2000000; double comp_amount2 = 1000000; double comm_amount12 = 2000000; double comm_amount21 = 3000000; XBT_INFO("Computation time for %f flops on %s: %f", comp_amount1, name1, comp_amount1/sg_host_speed(h1)); XBT_INFO("Computation time for %f flops on %s: %f", comp_amount2, name2, comp_amount2/sg_host_speed(h2)); XBT_INFO("Route between %s and %s:", name1, name2); SD_link_t *route = SD_route_get_list(h1, h2); int route_size = SD_route_get_size(h1, h2); for (int i = 0; i < route_size; i++) XBT_INFO(" Link %s: latency = %f, bandwidth = %f", sg_link_name(route[i]), sg_link_latency(route[i]), sg_link_bandwidth(route[i])); xbt_free(route); XBT_INFO("Route latency = %f, route bandwidth = %f", SD_route_get_latency(h1, h2), SD_route_get_bandwidth(h1, h2)); XBT_INFO("Communication time for %f bytes between %s and %s: %f", comm_amount12, name1, name2, SD_route_get_latency(h1, h2) + comm_amount12 / SD_route_get_bandwidth(h1, h2)); XBT_INFO("Communication time for %f bytes between %s and %s: %f", comm_amount21, name2, name1, SD_route_get_latency(h2, h1) + comm_amount21 / SD_route_get_bandwidth(h2, h1)); /* creation of the tasks and their dependencies */ SD_task_t taskA = SD_task_create("Task A", NULL, 10.0); SD_task_t taskB = SD_task_create("Task B", NULL, 40.0); SD_task_t taskC = SD_task_create("Task C", NULL, 30.0); SD_task_t taskD = SD_task_create("Task D", NULL, 60.0); /* try to attach and retrieve user data to a task */ SD_task_set_data(taskA, static_cast(&comp_amount1)); if (fabs(comp_amount1 - (*(static_cast(SD_task_get_data(taskA))))) > 1e-12) XBT_ERROR("User data was corrupted by a simple set/get"); SD_task_dependency_add(NULL, NULL, taskB, taskA); SD_task_dependency_add(NULL, NULL, taskC, taskA); SD_task_dependency_add(NULL, NULL, taskD, taskB); SD_task_dependency_add(NULL, NULL, taskD, taskC); SD_task_dependency_add(NULL, NULL, taskB, taskC); try { SD_task_dependency_add(NULL, NULL, taskA, taskA); /* shouldn't work and must raise an exception */ xbt_die("Hey, I can add a dependency between Task A and Task A!"); } catch (xbt_ex& ex) { if (ex.category != arg_error) throw; /* this is a serious error */ } try { SD_task_dependency_add(NULL, NULL, taskB, taskA); /* shouldn't work and must raise an exception */ xbt_die("Oh oh, I can add an already existing dependency!"); } catch (xbt_ex& ex) { if (ex.category != arg_error) throw; } try { SD_task_dependency_remove(taskA, taskC); /* shouldn't work and must raise an exception */ xbt_die("Dude, I can remove an unknown dependency!"); } catch (xbt_ex& ex) { if (ex.category != arg_error) throw; } try { SD_task_dependency_remove(taskC, taskC); /* shouldn't work and must raise an exception */ xbt_die("Wow, I can remove a dependency between Task C and itself!"); } catch (xbt_ex& ex) { if (ex.category != arg_error) throw; } /* if everything is ok, no exception is forwarded or rethrown by main() */ /* watch points */ SD_task_watch(taskD, SD_DONE); SD_task_watch(taskB, SD_DONE); SD_task_unwatch(taskD, SD_DONE); /* scheduling parameters */ host_list[0] = h1; host_list[1] = h2; computation_amount[0] = comp_amount1; computation_amount[1] = comp_amount2; communication_amount[1] = comm_amount12; communication_amount[2] = comm_amount21; /* estimated time */ SD_task_t task = taskD; XBT_INFO("Estimated time for '%s': %f", SD_task_get_name(task), SD_task_get_execution_time(task, 2, host_list, computation_amount, communication_amount)); SD_task_schedule(taskA, 2, host_list, computation_amount, communication_amount, -1); SD_task_schedule(taskB, 2, host_list, computation_amount, communication_amount, -1); SD_task_schedule(taskC, 2, host_list, computation_amount, communication_amount, -1); SD_task_schedule(taskD, 2, host_list, computation_amount, communication_amount, -1); std::set *changed_tasks = simgrid::sd::simulate(-1.0); for (auto task: *changed_tasks){ XBT_INFO("Task '%s' start time: %f, finish time: %f", SD_task_get_name(task), SD_task_get_start_time(task), SD_task_get_finish_time(task)); } XBT_DEBUG("Destroying tasks..."); SD_task_destroy(taskA); SD_task_destroy(taskB); SD_task_destroy(taskC); SD_task_destroy(taskD); XBT_DEBUG("Tasks destroyed. Exiting SimDag..."); SD_exit(); return 0; } SimGrid-3.14.159/examples/simdag/scheduling/000755 001750 001750 00000000000 13031031310 022270 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/simdag/scheduling/Montage_25.xml000644 001750 001750 00000055057 13031031310 024726 0ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/simdag/scheduling/sd_scheduling.c000644 001750 001750 00000017640 13031031300 025256 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2009-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* simple test to schedule a DAX file with the Min-Min algorithm. */ #include #include "simgrid/simdag.h" #if HAVE_JEDULE #include "simgrid/jedule/jedule_sd_binding.h" #endif XBT_LOG_NEW_DEFAULT_CATEGORY(test, "Logging specific to this SimDag example"); typedef struct _HostAttribute *HostAttribute; struct _HostAttribute { /* Earliest time at which a host is ready to execute a task */ double available_at; SD_task_t last_scheduled_task; }; static double sg_host_get_available_at(sg_host_t host) { HostAttribute attr = (HostAttribute) sg_host_user(host); return attr->available_at; } static void sg_host_set_available_at(sg_host_t host, double time) { HostAttribute attr = (HostAttribute) sg_host_user(host); attr->available_at = time; sg_host_user_set(host, attr); } static SD_task_t sg_host_get_last_scheduled_task( sg_host_t host){ HostAttribute attr = (HostAttribute) sg_host_user(host); return attr->last_scheduled_task; } static void sg_host_set_last_scheduled_task(sg_host_t host, SD_task_t task){ HostAttribute attr = (HostAttribute) sg_host_user(host); attr->last_scheduled_task=task; sg_host_user_set(host, attr); } static xbt_dynar_t get_ready_tasks(xbt_dynar_t dax) { unsigned int i; xbt_dynar_t ready_tasks; SD_task_t task; ready_tasks = xbt_dynar_new(sizeof(SD_task_t), NULL); xbt_dynar_foreach(dax, i, task) { if (SD_task_get_kind(task) == SD_TASK_COMP_SEQ && SD_task_get_state(task) == SD_SCHEDULABLE) { xbt_dynar_push(ready_tasks, &task); } } XBT_DEBUG("There are %lu ready tasks", xbt_dynar_length(ready_tasks)); return ready_tasks; } static double finish_on_at(SD_task_t task, sg_host_t host) { double result; unsigned int i; double data_available = 0.; double redist_time = 0; double last_data_available; xbt_dynar_t parents = SD_task_get_parents(task); if (!xbt_dynar_is_empty(parents)) { /* compute last_data_available */ SD_task_t parent; last_data_available = -1.0; xbt_dynar_foreach(parents, i, parent) { /* normal case */ if (SD_task_get_kind(parent) == SD_TASK_COMM_E2E) { sg_host_t * parent_host= SD_task_get_workstation_list(parent); /* Estimate the redistribution time from this parent */ if (SD_task_get_amount(parent) <= 1e-6){ redist_time= 0; } else { redist_time = SD_route_get_latency(parent_host[0], host) + SD_task_get_amount(parent) / SD_route_get_bandwidth(parent_host[0], host); } data_available = SD_task_get_start_time(parent) + redist_time; } /* no transfer, control dependency */ if (SD_task_get_kind(parent) == SD_TASK_COMP_SEQ) { data_available = SD_task_get_finish_time(parent); } if (last_data_available < data_available) last_data_available = data_available; } xbt_dynar_free_container(&parents); result = MAX(sg_host_get_available_at(host), last_data_available) + SD_task_get_amount(task)/sg_host_speed(host); } else { xbt_dynar_free_container(&parents); result = sg_host_get_available_at(host) + SD_task_get_amount(task)/sg_host_speed(host); } return result; } static sg_host_t SD_task_get_best_host(SD_task_t task) { sg_host_t *hosts = sg_host_list(); int nhosts = sg_host_count(); sg_host_t best_host = hosts[0]; double min_EFT = finish_on_at(task, hosts[0]); for (int i = 1; i < nhosts; i++) { double EFT = finish_on_at(task, hosts[i]); XBT_DEBUG("%s finishes on %s at %f", SD_task_get_name(task), sg_host_get_name(hosts[i]), EFT); if (EFT < min_EFT) { min_EFT = EFT; best_host = hosts[i]; } } xbt_free(hosts); return best_host; } int main(int argc, char **argv) { unsigned int cursor; double min_finish_time = -1.0; SD_task_t task, selected_task = NULL; xbt_dynar_t ready_tasks; sg_host_t selected_host = NULL; char * tracefilename = NULL; /* initialization of SD */ SD_init(&argc, argv); /* Check our arguments */ xbt_assert(argc > 2, "Usage: %s platform_file dax_file [jedule_file]\n" "\tExample: %s simulacrum_7_hosts.xml Montage_25.xml Montage_25.jed", argv[0], argv[0]); if (argc == 4) tracefilename = xbt_strdup(argv[3]); /* creation of the environment */ SD_create_environment(argv[1]); /* Allocating the host attribute */ int total_nhosts = sg_host_count(); sg_host_t *hosts = sg_host_list(); for (cursor = 0; cursor < total_nhosts; cursor++) sg_host_user_set(hosts[cursor], xbt_new0(struct _HostAttribute, 1)); /* load the DAX file */ xbt_dynar_t dax = SD_daxload(argv[2]); xbt_dynar_foreach(dax, cursor, task) { SD_task_watch(task, SD_DONE); } /* Schedule the root first */ xbt_dynar_get_cpy(dax, 0, &task); sg_host_t host = SD_task_get_best_host(task); SD_task_schedulel(task, 1, host); while (!xbt_dynar_is_empty(SD_simulate(-1.0))) { /* Get the set of ready tasks */ ready_tasks = get_ready_tasks(dax); if (xbt_dynar_is_empty(ready_tasks)) { xbt_dynar_free_container(&ready_tasks); /* there is no ready task, let advance the simulation */ continue; } /* For each ready task: * get the host that minimizes the completion time. * select the task that has the minimum completion time on its best host. */ xbt_dynar_foreach(ready_tasks, cursor, task) { XBT_DEBUG("%s is ready", SD_task_get_name(task)); host = SD_task_get_best_host(task); double finish_time = finish_on_at(task, host); if (min_finish_time < 0 || finish_time < min_finish_time) { min_finish_time = finish_time; selected_task = task; selected_host = host; } } XBT_INFO("Schedule %s on %s", SD_task_get_name(selected_task), sg_host_get_name(selected_host)); SD_task_schedulel(selected_task, 1, selected_host); /* * SimDag allows tasks to be executed concurrently when they can by default. * Yet schedulers take decisions assuming that tasks wait for resource availability to start. * The solution (well crude hack is to keep track of the last task scheduled on a host and add a special type of * dependency if needed to force the sequential execution meant by the scheduler. * If the last scheduled task is already done, has failed or is a predecessor of the current task, no need for a * new dependency */ SD_task_t last_scheduled_task = sg_host_get_last_scheduled_task(selected_host); if (last_scheduled_task && (SD_task_get_state(last_scheduled_task) != SD_DONE) && (SD_task_get_state(last_scheduled_task) != SD_FAILED) && !SD_task_dependency_exists(sg_host_get_last_scheduled_task(selected_host), selected_task)) SD_task_dependency_add("resource", NULL, last_scheduled_task, selected_task); sg_host_set_last_scheduled_task(selected_host, selected_task); sg_host_set_available_at(selected_host, min_finish_time); xbt_dynar_free_container(&ready_tasks); /* reset the min_finish_time for the next set of ready tasks */ min_finish_time = -1.; } XBT_INFO("Simulation Time: %f", SD_get_clock()); XBT_INFO("------------------- Produce the trace file---------------------------"); XBT_INFO("Producing a jedule output (if active) of the run into %s", tracefilename?tracefilename:"minmin_test.jed"); #if HAVE_JEDULE jedule_sd_dump(tracefilename); #endif free(tracefilename); xbt_dynar_free_container(&ready_tasks); xbt_dynar_foreach(dax, cursor, task) { SD_task_destroy(task); } xbt_dynar_free_container(&dax); for (cursor = 0; cursor < total_nhosts; cursor++) { free(sg_host_user(hosts[cursor])); sg_host_user_set(hosts[cursor], NULL); } xbt_free(hosts); /* exit */ SD_exit(); return 0; } SimGrid-3.14.159/examples/simdag/scheduling/expected_output.jed000644 001750 001750 00000123122 13031031306 026203 0ustar00mquinsonmquinson000000 000000 )) { next if ($line =~ /^#/); # 278 7.2 -9.4 h 2.3 if($line =~ /^(.*) (.*) (.*) h (.*)$/) { print "\t\t\n"; next; } die "Parse error: $line\n"; } print "\t\n"; print "\n"; SimGrid-3.14.159/examples/platforms/cluster_dragonfly.xml000644 001750 001750 00000000665 13031031275 025177 0ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/platforms/griffon.xml000644 001750 001750 00000003562 13031031275 023102 0ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/platforms/two_hosts.xml000644 001750 001750 00000001172 13031031276 023475 0ustar00mquinsonmquinson000000 000000 1.0 0.5 4.0 1.0 SimGrid-3.14.159/examples/platforms/routing_cluster.lua000644 001750 001750 00000004232 13031031276 024655 0ustar00mquinsonmquinson000000 000000 -- See routing_cluster.xml for description. -- require "simgrid" simgrid.engine.open() simgrid.engine.AS_open{id="AS0",mode="Full"} simgrid.engine.AS_open{id="my_cluster1",mode="Cluster"} simgrid.engine.router_new{id="router1"} simgrid.engine.host_new{id="host1",speed="1Gf"}; simgrid.engine.link_new{id="l1_UP",bandwidth="125MBps",lat="100us"}; simgrid.engine.link_new{id="l1_DOWN",bandwidth="125MBps",lat="100us"}; simgrid.engine.host_link_new{id="host1",up="l1_UP",down="l1_DOWN"}; simgrid.engine.host_new{id="host2",speed="1Gf"}; simgrid.engine.link_new{id="l2",bandwidth="125MBps",lat="100us",sharing_policy="FULLDUPLEX"}; simgrid.engine.host_link_new{id="host2",up="l2_UP",down="l2_DOWN"}; simgrid.engine.host_new{id="host3",speed="1Gf"}; simgrid.engine.link_new{id="l3",bandwidth="125MBps",lat="100us"}; simgrid.engine.host_link_new{id="host3",up="l3",down="l3"}; simgrid.engine.backbone_new{id="backbone1",bandwidth="2.25GBps",lat="500us"}; simgrid.engine.AS_seal() simgrid.engine.AS_open{id="my_cluster2",mode="Cluster"} simgrid.engine.router_new{id="router2"} simgrid.engine.host_new{id="host4",speed="1Gf"}; simgrid.engine.link_new{id="l4_UP",bandwidth="125MBps",lat="100us"}; simgrid.engine.link_new{id="l4_DOWN",bandwidth="125MBps",lat="100us"}; simgrid.engine.host_link_new{id="host4",up="l4_UP",down="l4_DOWN"}; simgrid.engine.host_new{id="host5",speed="1Gf"}; simgrid.engine.link_new{id="l5",bandwidth="125MBps",lat="100us",sharing_policy="FULLDUPLEX"}; simgrid.engine.host_link_new{id="host5",up="l5_UP",down="l5_DOWN"}; simgrid.engine.host_new{id="host6",speed="1Gf"}; simgrid.engine.link_new{id="l6",bandwidth="125MBps",lat="100us"}; simgrid.engine.host_link_new{id="host6",up="l6",down="l6"}; simgrid.engine.backbone_new{id="backbone2",bandwidth="2.25GBps",lat="500us"} simgrid.engine.AS_seal() simgrid.engine.link_new{id="link1-2",bandwidth="2.25GBps",lat="500us"}; simgrid.engine.ASroute_new{src="my_cluster1", dst="my_cluster2", gw_src="router1", gw_dst="router2", links="link1-2"} simgrid.engine.AS_seal() simgrid.engine.close() SimGrid-3.14.159/examples/platforms/cluster_no_backbone.xml000644 001750 001750 00000001104 13031031275 025437 0ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/examples/platforms/two_hosts_platform_with_availability.xml000644 001750 001750 00000000763 13031031276 031173 0ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/INSTALL000644 001750 001750 00000023727 13031031306 016132 0ustar00mquinsonmquinson000000 000000 This page summarizes how to compile SimGrid. The full Install documentation is available in doc/html/install.html or online at http://simgrid.gforge.inria.fr/ Getting the Dependencies ------------------------ SimGrid only uses very standard tools: - C compiler, C++ compiler, make and friends. - perl (but you may try to go without it) - cmake (version 2.8.8 or higher). You may want to use ccmake for a graphical interface over cmake. - boost: - Max OS X: with fink: fink install boost1.53.nopython, or with homebrew: brew install boost - Debian / Ubuntu: apt-get install libboost-dev libboost-context-dev - Java (if you want to build the Java bindings): - Mac OS X or Windows: Grab a full JDK - Debian / Ubuntu: apt-get install default-jdk Build Configuration ------------------- Note that compile-time options are very different from run-time options. The default configuration should be fine for most usages, but if you need to change something, there are several ways to do so. First, you can use environment variables. For example, you can change the compilers used by issuing these commands before launching cmake: export CC=gcc-4.7 export CXX=g++-4.7 Note that other variables are available, such as CFLAGS and CXXFLAGS to add options respectively for the C and C++ compilers. Another way to do so is to use the -D argument of cmake as follows. Note that the ending dot is mandatory (see Out of Tree Compilation). cmake -DCC=clang -DCXX=clang++ . Finally, you can use the ccmake graphical interface to change these settings. ccmake . Existing compilation options ---------------------------- CMAKE_INSTALL_PREFIX (path) Where to install SimGrid (/opt/simgrid, /usr/local, or elsewhere). enable_compile_optimizations (ON/OFF) Request the compiler to produce efficient code. You want to activate it, unless you plan to debug SimGrid itself. Indeed, efficient code may be appear mangled to debuggers. enable_compile_warnings (ON/OFF) Request the compiler to issue error messages whenever the source code is not perfectly clean. If you are a SimGrid developer, you have to activate this option to enforce the code quality. As a regular user, this option will bring you nothing. enable_debug (ON/OFF) Disable this option toto discard all log messages of gravity debug or below at compile time. The resulting code is faster than if you discarding these messages at runtime. However, it obviously becomes impossible to get any debug info from SimGrid if something goes wrong. enable_documentation (ON/OFF) Generate the documentation pages. enable_java (ON/OFF) To enjoy the java bindings of SimGrid. enable_jedule (ON/OFF) To get SimDag producing execution traces that can then be visualized with the Jedule external tool. enable_lua (ON/OFF) To enjoy the lua bindings to the SimGrid internals. enable_lib_in_jar (ON/OFF) Bundle the native java bindings in the jar file. enable_lto (ON/OFF) Enable the Link Time Optimization of the C compiler. This feature really speeds up the produced code, but it is fragile with some versions of GCC. enable_maintainer_mode (ON/OFF) Only needed if you plan to modify very specific parts of SimGrid (e.g., the XML parsers and other related elements). Moreover, this adds an extra dependency on flex and flexml. enable_mallocators (ON/OFF) Disabled this when tracking memory issues within SimGrid, or our internal memory caching mechanism will fool the debuggers. enable_model-checking (ON/OFF) This execution gear is very usable now, but enabling this option at compile time will hinder simulation speed even when the model-checker is not activated at run time. enable_ns3 (ON/OFF) Allow to use ns-3 as a SimGrid network model. enable_smpi (ON/OFF) Allow to run MPI code on top of SimGrid. enable_smpi_ISP_testsuite (ON/OFF) Add many extra tests for the model-checker module. enable_smpi_MPICH3_testsuite (ON/OFF) Add many extra tests for the MPI module. Reset the build configuration ----------------------------- To empty the cmake cache (either when you add a new library or when things go seriously wrong), simply delete your CMakeCache.txt. You may also want to directly edit this file in some circumstances. Out of Tree Compilation ----------------------- By default, the files produced during the compilation are placed in the source directory. It is however often better to put them all in a separate directory: cleaning the tree becomes as easy as removing this directory, and you can have several such directories to test several parameter sets or architectures. For that, go to the directory where the files should be produced, and invoke cmake (or ccmake) with the full path to the SimGrid source as last argument. mkdir build cd build cmake [options] .. make Mac OS X Builds --------------- SimGrid compiles like a charm with clang (version 3.0 or higher) on Mac OS X: cmake -DCMAKE_C_COMPILER=/path/to/clang -DCMAKE_CXX_COMPILER=/path/to/clang++ . make With the XCode version of clang 4.1, you may get the following error message: CMake Error: Parse error in cache file build_dir/CMakeCache.txt. Offending entry: /SDKs/MacOSX10.8.sdk In that case, edit the CMakeCache.txt file directly, so that the CMAKE_OSX_SYSROOT is similar to the following. Don't worry about the warning that the "-pthread" argument is not used, if it appears. CMAKE_OSX_SYSROOT:PATH=/Applications/XCode.app/Contents/Developer/Platforms/MacOSX.platform/Developer In the El Capitan version of Max OS X, Apple decided that users don't need no /usr/include directory anymore. If you are hit by this pure madness, just run the following command to restore that classical UNIX directory: xcode-select -install Windows Builds -------------- Building SimGrid on Windows may be something of an adventure: We only manage to do so ourselves with MinGW-64, ActiveState Perl and msys git). Have a look at out configuration scripts in appveyor.yml, but don't expect too much from us: we are really not fluent with Windows. Actually your help is welcome. The drawback of MinGW-64 is that the produced DLL are not compatible with MS Visual C. clang-cl sounds promising to fix this. If you get something working, please tell us. Build the Java bindings ----------------------- Once you have the full JDK installed (on Debian/Ubuntu, grab the package default-jdk for that), things should be as simple as: cmake -Denable_java=ON . make After the compilation, the file simgrid.jar is produced in the root directory. If you only want to build the jarfile and its dependencies, type make simgrid-java_jar. It will save you the time of building every C examples and other things that you don't need for Java. Sometimes, the build system fails to find the JNI headers: Error: jni could not be found. In this case, you need to first locate them as follows: $ locate jni.h /usr/lib/jvm/java-7-openjdk-amd64/include/jni.h /usr/lib/jvm/java-8-openjdk-amd64/include/jni.h Then, set the JAVA_INCLUDE_PATH environment variable to the right path, and relaunch cmake. If you have several version of jni installed (as above), use the right one (check the java version you use with javac -version). export JAVA_INCLUDE_PATH=/usr/lib/jvm/java-8-openjdk-amd64/include/ cmake -Denable_java=ON . make Note that the filename jni.h was removed from the path. 32 bits Builds on Multi-arch Linux ---------------------------------- On a multiarch x86_64 Linux, it should be possible to compile a 32 bit version of SimGrid with something like: CFLAGS=-m32 \ CXXFLAGS=-m32 \ PKG_CONFIG_LIBDIR=/usr/lib/i386-linux-gnu/pkgconfig/ \ cmake . \ -DCMAKE_SYSTEM_PROCESSOR=i386 \ -DCMAKE_Fortran_COMPILER=/some/path/to/i686-linux-gnu-gfortran \ -DGFORTRAN_EXE=/some/path/to/i686-linux-gnu-gfortran \ -DCMAKE_Fortran_FLAGS=-m32 If needed, implement i686-linux-gnu-gfortran as a script: #!/bin/sh exec gfortran -m32 "$@" Existing Compilation Targets ---------------------------- In most cases, compiling and installing SimGrid is enough: make make install # try "sudo make install" if you don't have the permission to write In addition, several compilation targets are provided in SimGrid. If your system is well configured, the full list of targets is available for completion when using the Tab key. Note that some of the existing targets are not really for public consumption so don't worry if some stuff doesn't work for you. make simgrid Build only the SimGrid library and not any example make app-masterworker Build only this example (works for any example) make clean Clean the results of a previous compilation make install Install the project (doc/ bin/ lib/ include/) make uninstall Uninstall the project (doc/ bin/ lib/ include/) make dist Build a distribution archive (tgz) make distcheck Check the dist (make + make dist + tests on the distribution) make documentation Create SimGrid documentation If you want to see what is really happening, try adding VERBOSE=1 to your compilation requests: make VERBOSE=1 Testing your build ------------------ Once everything is built, you may want to test the result. SimGrid comes with an extensive set of regression tests (as described in the insider manual). The tests are run with ctest, that comes with CMake. We run them every commit and the results are on our Jenkins. ctest # Launch all tests ctest -R msg # Launch only the tests which name match the string "msg" ctest -j4 # Launch all tests in parallel, at most 4 at the same time ctest --verbose # Display all details on what's going on ctest --output-on-failure # Only get verbose for the tests that fail ctest -R msg- -j5 --output-on-failure # You changed MSG and want to check that you didn't break anything, huh? # That's fine, I do so all the time myself. SimGrid-3.14.159/ChangeLog000644 001750 001750 00000444761 13031031306 016660 0ustar00mquinsonmquinson000000 000000 SimGrid (3.14.159) stable; urgency=low The previous release was only a crude approximation of a Pi release; this one should be a bit better. - Revert a last minute change that broke on Mac OSX. - Fix the changelog and NEWS files. - Improve the documentation, in particular of the routing module. - Integrate some patches flying around in Debian. -- Wed Dec 28 17:14:49 CET 2016 Da SimGrid team SimGrid (3.14) stable; urgency=low Documentation * The whole documentation was reworked and reorganized. There is still much room for improvement here, but we're on it. Infrastructure * We now need python3 for our testing infrastructure. * Model checking needs libevent SMPI * Privatization is now activated by default in smpirun You can turn it off with -no-privatize if something goes wrong. * Call-location tracing for SMPI. You can add the exact location (filename / linenumber) of an MPI call to your trace files and slow down or speed up the simulation between two consecutive calls by using an adjustment file (see the documentation). * Fixed computation of timings for MPI_Send, MPI_Recv & possibly also others We've found a bug that prevented SMPI to account for MPI_Send, MPI_Recv and others (in some cases) in a correct way. That is, the smpi/os, smpi/or values were ignored in some cases. The timings of these functions can now be significantly different. * smpi/cpu-threshold:-1 should become smpi/simulate-computation:no smpi/running-power is renamed to smpi/host-speed * smpi/grow-injected-times option to enable or disable multiplication of the timings injected in MPI_Iprobe, or MPI_Test. Enabled by default, which can make simulation less precise (but also much faster). * smpirun script should be (much) faster for large deployments. * SMPI tracing : fixed issue with poor matching of send/receives. * Replay : Fix broken waitall New functions and features * MSG_parallel_task_execute_with_timeout, to timeout computations. Dropped / renamed functions and features * msg_mailbox_t and associated functions. Use s4u::Mailbox instead. - MSG_mailbox_is_empty() -> Mailbox::empty() - MSG_mailbox_front() -> Mailbox::front() - MSG_mailbox_get_by_alias() -> simgrid::s4u::Mailbox::byName(name) - MSG_mailbox_get_task_ext() -> MSG_task_receive_ext() - MSG_mailbox_get_task_ext_bounded -> MSG_task_receive_ext_bounded - MSG_host_(get/set)_params -> MSG_vm_(get/set)_params * Don't pass the free_f parameter to property related functions: - MSG_host_set_property_value() - MSG_as_router_set_property_value() - MSG_storage_set_property_value() * VM properties. Since msg_vm_t are msg_host_t, just use MSG_host_get_property_value() and friends * VM I/O related things: - Ignored parameter of vm_create: core_nb, disk_path and disk_size. - Unimplemented save/restore methods * MSG_as_router_get_property_value() was redundent with MSG_environment_as_get_property_value(). - Removed MSG_as_router_*propert*() functions - Added MSG_environment_as_set_property_value() for consistency * xbt heterogeneous dictionnaries (created with xbt_dict_new()). Well, they are still there for now, but deprecated with a warning. Please switch to xbt_dict_new_homogeneous() before this is removed for real. * Task affinity. Its intended behavior (that was very badly tested and probably not really working) was deceiving what most users would have hoped here. * xbt_os_sem_get_value: unused internally, deprecated on OS X El Capitan * Option network/coordinates is now useless and should be dropped. Storage: * Added option storage/max_file_descriptors to allow more than 1024 files opened SURF: * Added option maxmin/concurrency_limit to allow more than 100 processes per host * Added Dragonfly topology support XML: * Add Exa- and Peta- units such as EiB, EB, Eib, Eb for size, and EiBps, EBps, Eibps, Ebps for bandwidth. They may become useful to some lucky ones. Java: * New functions: msg.Comm.waitAll() and msg.Comm.waitAny() * ex/app_tokenring: new example, very similar to the MSG Token Ring * ex/async_waitAll: new example, on asynchronous communications MSG: * Memory usage should be decreased for simulations with a large number of processes. This also helps for SMPI. -- Sat Dec 24 22:34:26 CET 2016 Da SimGrid team SimGrid (3.13) stable; urgency=low The Half Release, a.k.a. the Zealous Easter Trim. - We removed half of the lines, that were mostly experimental cruft. v3.12 lasted 286000 lines of code, v3.13 is only 142000 lines (not counting blanks and comments -- according to openhub.net) - The internals are now compiled in C++ (and will soon be clean C++) - We removed 75 klines of XML, 12 klines of Java, 5 klines of cmake, 59 klines of C, etc. We added only 29 klines of C++ in replacement. * Backwards Compatibility breaks - Removed Lua simulation bindings (switch to C or Java for that). Lua can still be used to describe platforms - Removed Java kernel plug-ins. Will be reintroduced after the ongoing major internals reorg. - In MSG - the following functions were removed. They were too specific and should be reimplemented in a generic way, with filter function. - MSG_task_listen_from_host - MSG_mailbox_get_count_host_waiting_tasks - MSG_mailbox_put_with_timeout was removed. Please use MSG_task_send_with_timeout instead. - In SimDag - the SD_application_reinit function was removed. It has been a noop for a while. - The ACCESS_MODE of SD_workstation has been removed. This feature was not really usable and should soon be replaced by a more flexible mechanism. - The following functions thus do not exist anymore - SD_workstation_get_access_mode - SD_workstation_set_access_mode - SD_workstation_get_current_task - Basic estimation functions have been removed but can easily be replaced - SD_route_get_communication_time => SG_route_get_latency() + amount / SD_route_get_bandwidth() - SD_workstation_get_computation_time => amount / sg_host_speed() - In Java - VM.setBound(int load) is now VM.setBound(double bound) to meet the MSG semantics. Use VM.getSpeed()*load/100 for the legacy behavior. - In CMake - option enable_tracing was removed. It was not doing anything for a while. - In the ModelChecker: - the model-checker now ptraces the model-checked process which means you cannot use a debugger on the latter anymore (we might make this optional in the feature); - removed soft-dirty page tracking; - remove model-checked side snapshot management, MC_snapshot() and MC_compare_snapshot(); - keep the MC_cut() function as a stub (it was not really working in the previous release). Options: * All options are consistently in kebab-case. Old names are kept as alias. XML platforms: * Switch to platform v4 format. - Rename from 'power' to 'speed' the attributes describing the amount of flops that a , , or can deliver per second. - In , attribute kind="POWER" is now kind="SPEED". - In and , attributes availability and state are gone. It was redundent with state and availability traces, and with peak values. - In , attributes availability_file and state_file are gone. It was too complex and unused. - Kill . Was not doing anything. - The DOCTYPE points to the right URL: http://simgrid.gforge.inria.fr/simgrid/simgrid.dtd (the file at this address now documents the changelog since its v1) - A warning is emitted for unit-less values (they are still accepted). - speed. Default: 'f' or 'flops'. Also defined: 'Yf', 'Zf', 'Ef', 'Pf', 'Tf', 'Gf', 'Mf', 'kf' 'yottaflops', 'zettaflops', 'exaflops', 'petaflops', 'teraflops', 'gigaflops', 'megaflops', 'kiloflops' - bandwidth. Default: 'Bps' bytes per second (or 'bps' for bits but 1 Bps = 8 bps) Also defined in bytes: 'TiBps', 'GiBps', 'MiBps', 'KiBps', 'TBps', 'GBps', 'MBps', 'kBps', 'Bps' And the same in bits: 'Tibps', 'Gibps', 'Mibps', 'Kibps', 'Tbps', 'Gbps', 'Mbps', 'kbps', 'bps' - latency. Default: 's' second. Also defined: 'w' week, 'd' day, 'h' hour, 'm' minute, 'ms' millisecond, 'us' microsecond, 'ns' nanosecond, 'ps' picosecond * bin/simgrid_update_xml can upgrade your files automatically (won't convert unit-less values) tools/sg_xml_unit_converter.py may help (but it's just a warning and will probably ever be). S4U * s4u::Host is now the preferred public interface to the Host features. sg_host_* functions are C bindings to the exact same behavior MSG_host_* and SD_workstation_* are #define to the sg_host_* ones MSG * The examples were completely reorganized (in C and Java), for your browsing pleasure. * Kill all deprecated functions (the ones you had when declaring MSG_DEPRECATED). They were deprecated since a few years, and probably did not even compile anymore. SimDag * The API has been profoundly modified to directly use the core objects instead of redefining its own. SD_Workstation_t and SD_link_t are now sg_host_t and sg_link_t respectively. Some functions have also been renamed for consistency. Backward compatibility is maintained, but users are encouraged to update their codes. A list of the modified functions can be found at the end of include/simgrid/simdag.h Simix * simgrid::simix::kernelImmediate() is the closure callback. It ensures that the lambda or closure passed as a parameter will run in kernel mode. All the callback functions should be rewritten to that interface at some point. Surf * Reorganizing and cleaning the internals all around the place. SMPI * Remove old default barrier/bcast buggy algorithms (see #18407) * Various bug fixes to handle more codes * Remove the need for the --foreground option of smpirun (it is still accepted for backward compatibility). XBT * Kill the setset data container: MC don't use it anymore. * Kill the queue data container: it made more sense with GRAS. * Kill the xbt_peer_t data type: it's useless without GRAS. * Kill rm_cb feature of config sets: it was never useful. * Kill graphxml parsing feature. It was not used. * Kill the deprecated code protected by XBT_USE_DEPRECATED * New functions: - xbt_dynar_sort_strings(), when the content is char* - xbt_str_parse_int / xbt_str_parse_double, wrapping strtol/strtod They throw exceptions on invalid input; * C++ support for declaring CLI flags (simgrid::config::Flag); * class for abstracting different signal backends (simgrid::xbt::signal). with no external dependencies (we need very simple signals). MC * refactoring and cleanup of the code; * ongoing process to cleanly separate the model-checking algorithms from the code model-checking support. -- Wed Apr 27 21:00:53 CEST 2016 Da SimGrid team SimGrid (3.12) stable; urgency=low The Facelift Release. Build System * Require g++ v4.7 at least to not speak prehistorical C++. * Require Boost 1.48 (for signal2 component). * Java must be version 7 at least when activated. * Builds on Windows again (including Java bindings). * Tracing is now always enabled (no way to turn it out) * Remove GTNetS. It was not working anyway. * Various cleanups in the cmake scripts. * Move headers around to sort them out on installed systems: - instr/instr.h -> simgrid/instr.h - instr/jedule/* -> simgrid/jedule - simdag/datatypes.h was removed - simdag/simdag.h -> simgrid/simdag.h - msg/datatypes.h was removed - msg/msg.h -> simgrid/msg.h cMSG: * Interface improvement: - Rename MSG_host_is_avail(h) to MSG_host_is_on(h) - Sanitize the interface in MSG_task_ module: - Merge two functions that were close enough but misleading: set_compute_duration(t) -> set_flops_amount(t) get_remaining_computation(t) -> get_flops_amount(t) - set_data_size(t) -> set_bytes_amount(t) get_data_size(t) -> get_bytes_amount(t) - Massive cleanups in the functions related to the energy - MSG_host_get_pstate_number() -> MSG_host_get_nb_pstates() - New: MSG_host_get_pstate() * New example: - msg/energy/onoff: switching hosts on and off jMSG: * Interface improvement: - Rename Host.isAvail() to Host.isOn() - Rename Process.currentProcess() to Process.getCurrentProcess() - Rename Task.setDataSize() to Task.setBytesAmount() - Merge Task.getRemainingDuration() and Task.getComputeDuration() into Task.getFlopsAmount() * Bug fixes: - #18874: Actually allows the GC to reclaim tasks SIMIX: * New functions - SIMIX_process_throw: raises an exception in a remote process * Refactoring: Separate sub-modules - libsmx: the public interface, as libc in a real system - popping: the strange dance that converts a user request into a kernel handling - smx_context_*: the virtualization mechanisms that embed the user code - smx_*: the handling of each simcalls * Interface cleanup: - simcall_host_set_power_peak_at -> simcall_host_set_pstate * Rename smx_action_t into smx_synchro_t, making explicit that these things are used to synchronize processes with their environment. For example, a communication is a sort of synchronization involving the communicating processes (that may block until the exchange) and the platform. The same can be said from computations, etc. * Bug fixes: - #18888: segfault when a process ends before its kill_time SMPI: * New functions - Onesided early support for : MPI_Win_(create, free, fence, get_name, set_name, get_group), MPI_Get, MPI_Put, MPI_Accumulate, MPI_Alloc_mem, MPI_Free_mem. - MPI_Keyval*, MPI_Attr* functions, as well as MPI_Comm_attr*, MPI_Type_attr* variants (C only, no Fortran support yet) - MPI_Type_set_name, MPI_Type_get_name - MPI_*_c2f and MPI_*_f2c functions - MPI_Info_* functions (beware, get_nthkey may not follow the insertion order) - MPI_Pack, MPI_Unpack and MPI_Pack_size functions - Activate a lot of new tests from the mpich 3 testsuite * Features - Constant times can be injected inside MPI_Wtime and MPI_Test through options smpi/wtime and smpi/test - InfiniBand network model added : Based on the works of Jerome Vienne http://mescal.imag.fr/membres/jean-marc.vincent/index.html/PhD/Vienne.pdf - When smpi/display_timing is set, also display global simulation time and application times - Have smpirun, smpicc and friends display the simgrid git hash version on --git-version * Collective communications - SMP-aware algorithms are now dynamically handled. An internal communicator is created for each node, and an external one to handle communications between "leaders" of each node - MVAPICH2 (1.9) collective algorithms selector : normal and SMP algorithms are handled, and selection logic is based on the one used on TACC's Stampede cluster (https://www.tacc.utexas.edu/stampede/). - Support for Rabenseifner Reduce/Allreduce algorithms (https://fs.hlrs.de/projects/par/mpi//myreduce.html) * Replay - Replay now uses algorithms from wanted collective selector - Replay can be used with SMP-aware algorithms - Memory occupation of replay should now be contained (temporary buffers allocated in collective algorithms should be shared between processes) - Replay can now replay several traces at the same time (check examples/smpi/replay_multiple example), to simulate interactions between several applications on a given platform. User can specify the start time of each instance. This should also allow replay + actual applications to run. * Bug fixes - [#17799] : have mpi_group_range_incl and mpi_group_range_excl better test some corner cases - Correctly use loopback on fat-tree clusters - Asynchronous small messages shouldn't trigger deadlocks anymore * Energy/DVFS cleanup and improvement - smpi_set_host_power_peak_at() -> smpi_set_host_pstate() - new: smpi_get_host_pstate() SURF * Bug fixes - "Full" network optimization flag was broken since Surf++ - Better handling of precision flags in maxmin - Fix bug causing sometimes "Impossible" errors - Properly pass cluster properties to included hosts * Improvement of the Energy plugin. - Always update the consumption before returning that value - New property: watt_off to denote the dissipation when the host is off - New functions getWattMinAt and getWattMaxAt to retrieve the dissipation of pstates that we are not currently at. * Java: class NetworkLink renamed to Link * New function: simcall_process_get_kill_time() * Massive rename s/workstation/host/ - That's intrusive, but that's good for the project consistency. Sorry. - Change config option "workstation/model" into "host/model" XBT * New functions - Add a xbt_heap_update function, to avoid costly xbt_heap_remove+xbt_heap_insert use - Add a xbt wrapper for simcall_mutex_trylock (asked in [#17878]) - Add two new log appenders : rollfile and splitfile. Patch by Fabien Chaix. - xbt_dirname and xbt_basename for non-POSIX systems MC * The model checker now runs as a separate process. * The model checker runs must now be launched with the new simgrid-mc program. * Record/Replay: the MC can display a textual representation of a path in the execution graph. It can then be replayed outside of the model checker. -- Mon Oct 12 06:02:41 CEST 2015 Da SimGrid team SimGrid (3.11) stable; urgency=low The Class Release. Tools: * Normalizing pointers addresses tool for better diff between logs Examples: * Add cloud examples using new VMs - examples/msg/cloud/two_tasks_vm.tesh - examples/msg/cloud/simple_vm.tesh - examples/java/cloud/cloud.tesh - examples/java/cloud/migration/migration.tesh * Add java surf examples: - examples/java/surfPlugin/surf_plugin.tesh - examples/java/reservationSurfPlugin/reservation_surf_plugin.tes - examples/java/surfCpuModel/surf_cpu_model.tesh * Add SMPI+MSG example: - examples/smpi/smpi_msg_masterslave/ TeshSuite: * Add tests: - msg process test - msg task destroy cancel test - msg_host on/off test * Move all tests in testsuite to teshsuite (adding tesh files) * Restructure teshsuites - one folder for each kind of test * Restructure AddTests.cmake - unify tests names - structure the order of tests (with sections) MSG: * Add virtual machine - creation of a VM on a PM - migration of a VM from a PM to another PM * New functions - MSG_process_join(msg_process_t process, double timeout) - msg_bar_t MSG_barrier_init(unsigned int count) - int MSG_barrier_wait(msg_bar_t barrier) - void MSG_barrier_destroy(msg_bar_t barrier) - msg_as_t MSG_environment_as_get_by_name(const char * name) * New option "msg/debug_multiple_use" to help debugging when a task is used several times * MSG IO - Improvements and finalization of MSG_storage, MSG_file APIs and their documentation - Increase code coverage in test suites - Bug fixes SIMIX: * Protect context stacks against stack overflow. The number of protected memory pages allocated on the top of each stack (1 by default) can be configured with the new command line option --cfg=contexts/guard_size:. * Simcalls are now generated by a python script that - generates files included by SimGrid - checks that all the functions exist, and proposes prototypes * Clean simcalls - remove sem_destroy, file_set_data, comm_destroy, vm_set_state, host_set_data, host_get_data * New simcalls - simcall_process_join(smx_process_t process, double timeout) * Fix bug where sleeping processing could not be suspended. SURF: * Translate surf models from C to C++ - Generic classes for all models: Model, Resource, Action - A generic interface for each kind of model (CPU, Network, Storage Workstation, WorkstationVM) - C bindings * Translate surf routings from C to C++ * Add callbacks using sigc++ or boost::signals2 - Add callback functions for resource creation/destruction - Add callback functions for action state change - Handle Energy as a plugin * Replace swag by boost::intrusive * Add new routing models for clusters. For documentation, see http://simgrid.gforge.inria.fr/simgrid/latest/doc/platform.html#pf_cluster - tori, with topology="TORUS" and topo_parameters="ndim1,ndim2,...,ndimn" parameters for cluster tag - Fat trees, with topology="FAT_TREE" and topo_parameters="h;m1,...,mh;w1,...,wh;p1,...,ph" parameters for cluster tag - see examples/platforms/torus_cluster.xml and examples/platforms/fat_tree_cluster.xml * More documentation SMPI: * Hostfiles support host:nb_processes construct to deploy several processes on one node. * Collective communication algorithms should not crash if used with improper number of nodes and report the error. * SMPI now partially supports MPI_Topologies : MPI_Cart_create, MPI_Cart_shift, MPI_Cart_rank, MPI_Cart_get, MPI_Cart_coords, MPI_Cartdim_get, MPI_Dims_create, MPI_Cart_sub are supported. * New interface to use SMPI programmatically (still depends on MSG for some parts, see examples/smpi/smpi_msg_masterslave) : - SMPI_app_instance_register(const char *name, xbt_main_func_t code, int num_processes) - SMPI_init() - SMPI_finalize(); * Global variables privatization in MPI executables is now performed at runtime with the option smpi/privatize_global_variables (default:no). Limitations : Linux/BSD only, with mmap enabled. Global variables inside dynamic libraries loaded by the application are not privatized (static linking with these libraries is advised in this case) Tracing: * Options defined in XML work correctly now. Java: * New cmake option, enable_lib_in_jar, to control whether native libraries are copied into simgrid.jar or not (ON by default). Use this option if you want to reduce the size of the installed simgrid.jar, *and* the native libraries are kept installed elsewhere. * Surf binding with SWIG (code generated in maintainer mode only): - plugin to handle callbacks - CPU model only for the moment Build System: * Supernovae build mode is definitively removed. It was used to improve inlining and inter-module optimizations. It is nowadays superseded by link-time optimizations commonly available in compilers. * Update ns-3 find lib. Bindings for ns-3 should work again now. * Add boost dependency for surf++ * Add new macro for tests - ADD_TESH(name ) - ADD_TESH_FACTORIES(name "thread;ucontext;raw" ) XBT: * New functions - xbt_bar_t XBT_barrier_init(unsigned int count) - int XBT_barrier_wait(xbt_bar_t barrier) - void XBT_barrier_destroy(xbt_bar_t barrier) * Make the xbt_os_time module public -- Sat May 31 22:39:38 CEST 2014 Da SimGrid team SimGrid (3.10) stable; urgency=low The Clean Diaper Release, a.k.a. SimGrid is leak-free. Java: * Reintegrate Java to the main archive as desynchronizing these package is not acceptable anymore (Java is now considered stable) * Add explicit synchronization facilities through semaphores * Bug fix: Task.setDataSize() only changed the C world, not the value cached in the Java world MSG: * Dramatically change the way files are handled. API and internals changed, but this part of MSG was not considered as production grade either. * Add explicit synchronization facilities through semaphores * Add a new function MSG_host_get_process_list() * Preliminary DVFS support (see examples/msg/energy/ for details) SMPI: * SMPI is now included directly in the libsimgrid as the windows linker doesn't force us on splitting it anymore. * Improvements of the SMPI replay tool: - Most of the collective communications are now rooted in the same process as in the original application. - Traces now rely on the same MPI data type as the application (MPI_BYTE was used until now). Multiple data types can now be used in a trace. - The replay tool now supports traces produce either by TAU or a modified version of MPE. - Bug Fix: the compute part of the reduce action is now taken into account. - Gatherv collective is now supported - SimGrid (SMPI for now) can generate replay traces as well. Option -trace-ti of smpirun outputs time independent traces for the current run. One file is created per process. If too many processes are simulated, this behavior can be changed to one file for all processes by using the tracing/smpi/format/ti_one_file flag * smpirun generates the host file if needed (with given host count and platform) * Integration of more than 100 STAR-MPI, MPICH, OpenMPI collective algorithms - allows to select one in particular with --cfg=smpi/coll_name:algorithm - allows to use the decision logic of OpenMPI(1.7) or MPICH(3.0.4) by setting --cfg=smpi/coll_selector:(mpich/ompi) * Support for new functions : MPI_Issend, MPI_Ssend, Commutative operations in Reduce * Add a --cfg:tracing/smpi/internals option, to trace internal communications happening inside a collective SMPI call. * Fix the behavior of complex data types handling. * Make MPI_Wtime another synchronization point to take computations into account. * Replace MPICH-1 test suite by the one from MPICH 3.0.4. Can be built using enable_smpi_MPICH3_testsuite flag in cmake. Run with ctest. * Add all missing Fortran bindings, SMPI should work with Fortran 90 (no privatization of global variables yet) * Preliminary DVFS support (see examples/smpi/energy/ for details) Model-Checking; * Verification of liveness properties is now available for SMPI applications (in addition to MSG applications) * Bugged examples using SMPI in examples/smpi/mc/ * Add --cfg=model-check/visited option. Allows the verification of infinite programs. Detection of loops in the execution thanks to the system state comparison and reduction of the state space to explore. Can be combined with DPOR for safety properties. SimDag: * Allow to change SimGrid configuration (see --help) within the code thanks to SD_config() as it can be done in MSG. * Add a new function SD_task_set_amount() upon user request. PLATFORM: * Handle units for values (10ms, 10kiloflops, 10Bps, 1GB, ...) * Remove rule based routing (no more PCRE dependency) * Add a limiter_link option to cluster tag, to specify a maximum reachable bandwidth in fullduplex mode when it is less than twice the nominal bandwidth. * Add a loopback_bw and loopback_lat options to cluster tag. * Fix the peer tag that could not be mixed with other AS within a Vivaldi routing. Now peers are encapsulated in an AS and have their own private router but this is transparent. XBT: * Our own implementation of getline is renamed xbt_getline, and gets used even if the OS provide a getline(). This should reduce the configuration complexity by using the same code on all platforms. * New type: xbt_cfg_elm_boolean. * Allow to use yes/no for boolean configuration options in the command line. * Allow to disable SimGrid cleanups at exit from command line option. There are situations where one may want a simulation to end with an exit. Unfortunately, calling exit may cause SimGrid to segfault, which is quite annoying when scripting around the simulator. Adding a --cfg=clean_atexit:no allows to circumvent this issue. Build System: * Lots of memory leaks were corrected in this release. There is no known memory leaks anymore, in all of our 600+ tests. * New command line option --version, to get SimGrid version information. Packagers may want to add extra words to SIMGRID_VERSION_EXTRA defined in CMakeLists.txt. * Supernovae builds are deprecated, and expected to be removed in the next version of SimGrid. -- Sun Nov 17 00:26:44 CET 2013 Da SimGrid team SimGrid (3.9) stable; urgency=low The Grasgory release: GRAS is really dead now. * Complete overhaul of the internal host structures scheme. GRAS: * If you use GRAS, you should stay at SimGrid 3.5 (at most) since it was considered as experimental and badly maintained since then. * Keeping it was thus a trap to our potential users, that could take it instead of MSG or SMPI by mistake despite is pity state. * GRAS seems to have very few users (if any), and no one volunteered to maintain it further. It also induces a lot of XBT code (for portability sake), that must be maintained too. * For all these reasons, we killed GRAS. If someone wants to revive it in the future, don't cry, our git history still remembers of GRAS. Documentation: * Major overhaul. Merge our documentation again as time proved that splitting it was really not helping our users. * Further improve the developer documentation to help newcomers hacking on SimGrid itself. The user documentation (and in particular, the beginner documentation) is still in a sorry state. SMPI: * Now works on Windows too! * Much more extensive test suite, from MPICH SIMDAG: * Add a new loader (SD_PTG_dotload) that creates a parallel task graph (i.e., a DAG whose nodes are parallel tasks) from a dot file. Creates a dynar of SD_TASK_COMP_PAR_AMDAHL and SD_TASK_COMM_MXN_1D_BLOCK tasks. * Bug fix: let task be scheduled when the last dependency to be solved is a control dependency. * Remove SD_load_environment_script function. Use the C sg_platf function if you want to declare a platform programmatically. MSG: * New function: MSG_process_get_number() * Old function documented: MSG_config() * Remove MSG_load_platform_script function Use the C sg_platf function if you want to declare a platform programmatically. SURF: * Change the default value of the TCP_gamma constant (maximal size of TCP congestion window) to a more realistic 4MiB value. If you notice changes in your simulation results, you can fall back to the previous 20k tiny window by adding --cfg=network/TCP_gamma:20000 on command line. * (Hopefully) fix a bug wrt periodic availability and state traces * Bug fix: use default values at start when first event in availability/state trace is not at time 0. PLATFORM: * remove the "new_" part of function name sg_platf_new_trace_connect (resulting in sg_platf_trace_connect), since it does not create anything new XBT: * Kill synchronized dynars, and xbt_dynar_dopar(). We cannot think of a use case where it's really mandatory, and maintaining it was a pain in our code base. * New: xbt_fifo_search(), search an item with a user-provided comparison function instead of dumb pointer comparison. LUA: * Fix the lua deployment: Use `simgrid.init_application()` before deployment instead of `simgrid.msg_register_application()` after. TRACING: * Transfer the tracing files into the corresponding modules. -- Tue Feb 5 11:31:43 CET 2013 Da SimGrid team SimGrid (3.8.1) stable; urgency=low The "we are told that some people want to also *install* the simgrid framework" release. * Add missing file "tesh.1" to the archive. -- Sat Oct 27 16:12:11 CEST 2012 Da SimGrid team SimGrid (3.8) stable; urgency=low The Psssshiiiit release: SimGrid jumps into the Cloud. MSG: * Add an experimental interface to manipulate VMs. They are mainly process groups with very few intrinsic semantic, but they should allow you to build the semantic you want easily. * New function: MSG_host_set_property_value() * New function: MSG_process_on_exit(). To clean memory in all cases. * Bug fixes that made the host (and link) failures unusable. * Add a way to auto-restart process when the host in which they are executing comes back (ON_FAILURE="RESTART" on deployment file, MSG_process_auto_restart_set). * Use the "msg_" prefix for all datatypes (instead of m_, msg_ and MSG_), please stop using the old ones, they are DEPRECATED. * Deprecate functions MSG_global_init() / MSG_global_init_args() Please use MSG_init() instead. (reducing the amount of entry points in the library helps us). * Make it impossible to link against the wrong version of the lib * Deprecate MSG_clean(). No need to call it anymore. * Function MSG_get_host_number() is not deprecated anymore. Documentation: * Split the doc into a user guide and a reference guide. * Start a developper guide to help people hacking on SimGrid. Cmake: * Enable tracing by default. This modules rocks you should use it. * Remove option custom_flags. Now use environment variables CFLAGS and LDFLAGS. * Use default cmake things to detect lua instead of home grown ones. * New option "enable_mallocators" to disable mallocators, for debugging purpose ("on" by default). SIMIX: * Bug fixes around the resource failures: don't let the processes survive the host they are running onto. * Add an interface to auto-restart processes when the host in which they are executing comes back. * Ensures that SIMIX_clean is called automatically. It's not part of the public interface anymore (bindings should be updated). SimDag: * Bug fix for when SD_Simulate is called with a positive value: be careful when comparing doubles. Sometimes they are different for non significant digits only. * New types of typed tasks. SD_TASK_COMP_PAR_AMDAHL represents a parallel task whose initial work is distributed among host according to the Amdahl's law. Such tasks are created with a parameter alpha that corresponds to the non-parallelizable part of the computation. SD_TASK_COMM_PAR_MXN_1D_BLOCK represents a complex data redistribution between two sets of workstations assuming a 1D block distribution (each workstation owns a similar share of data) on both sides. These tasks can be scheduled with SD_task_schedulel or SD_task_schedulev. Data redistribution will be automatically scheduled once parent and child are both scheduled. The filling of computation_amount and communication_amount structures is now done seamlessly thanks to the chosen assumptions. * New function SD_workstation_dump to display various information * New function SD_task_set_rate to throttle the bandwidth allowed to be used by a SD_TASK_COMM_E2E typed task. This rate depends on both the nominal bandwidth on the route onto which the task is scheduled and the amount of data to transfer. To divide the nominal bandwidth by 2, the rate then has to be : rate = bandwidth/(2*amount) * Compute tasks that have failed can now be rescheduled and executed again (from their beginning) * Increasing source code coverage (src/simdag is now covered at 95.8% on average) SMPI: * Re-implement time-independent trace replay using SMPI (at the smpi_smp_* level) instead of MSG. This should replace examples/msg/actions/actions.c * Implement support of MPI Datatypes (vectors, hvectors, indexed, hindexed and structs) * Implement the exchange of non-contiguous data. [Khalid Hasanov & Jean-Noel Quintin] Thanks for the patch, guys. * Correct behavior of smpi/sender_gap and set its default value to 0 * Add option to asynchronously send small messages to allow better simulation of pt2pt communications. --cfg=smpi/async_small_threshold:value specifies the size in bytes under which messages will be asynchronously sent. * Add support of MPI_Iprobe, MPI_Probe, MPI_Testall, MPI_Wtick functions * SMPI now handles more MPI specific values in input. Closes [#14389] and [#14388] SimGrid: * New C interface to define a platform: XML is now optional. For more info, please check include/simgrid/platf.h. * New interface to define random platforms from the C: For more info, please check include/simgrid/platf_generator.h and examples/msg/masterslave/masterslave_platfgen.c * Export a sg_cmdline dynar containing all the arguments we got from the command line. TRACE: * Two new tracing options for adding comments to trace file so you can track your experiments (see --help-tracing for details). * New option to generate a impoverished trace file (--cfg=tracing/basic:1) * Adding the SimGrid version that generated the trace file as a comment. * Instrumenting other MSG functions (MSG_task_isend_with_matching and MSG_task_dsend) * Fix to avoid key clashes on Paje links * Other minor fixes related to the Paje specification XBT: * Functions xbt_dict_hash() and xbt_dict_hash_ext() are made public, and renamed to xbt_str_hash() and xbt_str_hash_ext(). * New function: xbt_os_timer_resume() to restart a timer w/o resetting it. * Greatly improve the robustness of mmalloc to user errors (such as using an area after freeing it, or freeing it twice) -- Thu Oct 25 17:30:06 CEST 2012 Da SimGrid team SimGrid-java (3.8.1) stable; urgency=low * New module: org.simgrid.trace.Trace (SimGrid trace bindings) Warning: all methods are visible, but only some of them are implemented so far. Check the source (src/jtrace.c) for further information. * New module: org.simgrid.msg.File (SimGrid File management functions) * New Module: org.simgrid.msg.VM (SimGrid interface to mimick IAAS clouds) * Change the meaning of Process.restart: now restart the process from the begining, like MSG_process_restart in C. * Add Process.setAutoRestart: handling of process restart when failed host comes back. * Add Process.getProperty, Host.getProperty, Host.getProperty: allows you to retrieve the properties of the processes/hosts * Deprecate Msg.clean(): you can just forget about it now. * New function Process.getCount(), that only works when compiling with the not yet released version 3.9 of the C library. * New context factory based on Coroutines. It mandates a modified JVM but then, the simulations run about five times faster, and there is no limit to the amount of processes (beside of the available memory). -- 2012-12-04 Da SimGrid team SimGrid (3.7.1) stable; urgency=low MSG: * Restore the prototype of MSG_process_create_with_environment() to the pre-3.7 situation by removing the kill_time argument. * Add a MSG_process_set_kill_time() function instead. SURF: * Fix weird behaviors when dealing with parallel tasks. WINDOWS: * Simgrid is now built as a dll. * Simgrid-java now works on Windows. * Simgrid-Java is now included into Windows package. MacOS: * First pre-build package for MacOSX. Build System: * Fix compilation when using MSG_USE_DEPRECATED. * Fix some compilation issues on Macs and Windows. * Reduce the number of failing tests on exotic systems, like Debian/Hurd. * Environment variables CFLAGS and LDFLAGS are now honored by cmake. We discovered that the Lua console is broken, but we are missing the manpower to fix it right now. The problem existed in 3.7 too, so we are not blocking the release for that. Sorry if you depended on this feature, any help would be really welcome. -- Thu Jun 7 2012 Da SimGrid team SimGrid-java (3.7.1) stable; urgency=low The "Java aint got to be bloated and slow" release Major cleanups: * Various internal cleanups and performance improvement Simulations are expected to run up to twice faster or so * Make Process.kill(process) an instance method, not a static one * User processes are not java.lang.Thread subclasses. This breaks the compatibility (sorry), but previous API was brain-dead, making it impossible to have non-trivial initializations in the process constructor. * Require a full constructor per Process sub-class. Kinda breaks the compatibility (sorry), but this allows a much more efficient way to launch the processes at simulation startup. * Do not embeed our version of semaphores, java 1.5 can be considered as sufficiently prevalent for us to not dupplicate its features. * Lot of bug fixes Extend the API: * Add examples for almost every part of the API We spotted and fixed a lot of bugs in the process * New module: asynchronous communication API * New function: Process.sleep() It takes milliseconds as argument, just as java.lang.Thread.sleep() * New module: org.simgrid.msg.Mutex (SimGrid mutexes) * New module: org.simgrid.msg.RngStream (RngStreams random generators) -- 2012-06-12 Da SimGrid team SimGrid (3.7) stable; urgency=low The "spring cleanups (before the next Big Project kicks in)" release. Models: * We can specify the SMPI latency/bandwidth factor with command line add --cfg=smpi/bw_factor:"threshold0:value0;...;thresholdN:valueN" or add --cfg=smpi/lat_factor:"threshold0:value0;...;thresholdN:valueN" You can also use the "config tag" from platform file by adding this line (see "example/platforms/tag_config.xml" to use "config tag"). Note that the command line supersedes the platform file configuration. * Change the correction factors used in LMM model, according to the latest experiments described in INRIA RR-7821. Accuracy should be improved this way. * Use the partial invalidation optimization by default for the network too. Should produce the exact same results, only faster. * Major cleanup in surf to merge models and split some optimization mechanisms from the core of the models. As a result you can now specify which model to use (e.g., --cfg=network/model:LV08 --cfg=cpu/model:Cas01) and which optimization mode to use (e.g., --cfg=network/optim:lazy --cfg=cpu/optim:TI). Incompatible combinations should err at initialization. See --help-models for the list of all models and optimization modes. * The CLM03 workstation model was dropped for simplicity because it used the deprecated CM02 network model. Use default instead. * Rename the TCP_gamma configuration option to network/TCP_gamma * Rename the coordinates configuration option to network/coordinates, and document it * Use now crosstraffic keyword instead of the terribly misleading fullduplex keyword. It is activated by default now in the current default model, use --cfg=network/crosstraffic:0 to turn it off. * Ongoing refactoring the model parsing to make XML files optional See include/simgrid/platf.h for details (still to be completed) MSG: * Major overhaul of the documentation. Almost instructive now :/ * Deprecate the use of m_channel_t mechanism like MSG_task_{get,put} functions and friends. This interface was considered as deprecated since over 2 years, it's time to inform our users that it is. Switch to MSG_task_{send,recv} instead, or compile SimGrid command line 'cmake -Dcustom_flags="-DMSG_USE_DEPRECATED" .' if you really need to use these (crappy) functions in your code. These functions will be removed soon. Stop using them now. * Deprecate MSG_get_host_{table,number} Implement MSG_hosts_as_dynar() instead. * Implement MSG_processes_as_dynar() (Closes gforge #13642) * Remove the public field msg_host_t->name. Use MSG_host_get_name() Simix: * Stabilize the parallel execution mode of user contexts * Introduce configuration variables to control parallel execution: - contexts/synchro: Synchronization mode to use when running contexts in parallel (either futex, posix or busy_wait) - contexts/parallel_threshold: Minimal number of user contexts that must be part of a scheduling round to switch to parallel execution mode (raw contexts only) * Fix bugs that prevented to use suspend/resume along with synchronization structures. * Fix bugs in process termination that lead to invalid memory access in very specific conditions. SURF: * Introduce a parallel mode for the models (controlled by surf/nthreads configuration item). In our tests, running the models in parallel never lead to any speedups because they are so fast that the gain of computing each model in parallel does not amortizes the synchronization costs, even when ultra fast futexes are used. This is released anyway because YMMV. SimDag: * Performance boost by using a swag internally to compute the set of tasks that are finished and should constitute the return value of SD_simulate. SMPI: * Enable it by default now that it is considered rather stable. TRACE: * Documentation of the tracing functions. * Performance gains when tracing categorized/uncategorized resource utilization by avoiding calls to get route when updating resource variables. LMM constraints are being used instead. * API changed to set task categories. Use MSG_task_set_category instead of TRACE_msg_set_task_category, and SD_task_set_category instead of TRACE_sd_set_task_category. They only work if ENABLE_TRACING is ON. * Bugfix for graphicator, routes not correctly obtained, memory leaks * Examples for link user variables added (see at examples/msg/tracing/) * Deprecated function TRACE_msg_set_process_category completely removed * Trace header updated according to the latest Paje file format * Tracing network lazy updates, no longer obligate users to use full updates * --cfg=tracing/platform:1 also registers power/bandwidth variables * Experimental: let user code declare/set/push/pop application states for hosts * API changed to allow the manual creation of graph configuration files for Triva. See TRACE_get_node_types() and TRACE_get_edge_types(). Lua: * Improve the API of Lua MSG bindings, using the Lua spirit. * Each simulated process now lives in its own Lua world (globals are automatically duplicated). It helps writing simulators. Will allow to run Splay programs within SimGrid in the future. * Add a Chord example in Lua, equivalent to the MSG one. MODEL-CHECKING: * Start the implementation of a solution to express temporal properties, not only local assertions. This is still an experimental work in progress, stay clear from it to be safe. XBT: * Logs: - Add new runtime parameters --help-logs and --help-log-categories to display information about supported logging parameters and categories. - Old deprecated parameters --{gras,surf,msg,simix,xbt}-log=... don't exists anymore. * Mallocators: allow value NULL for the reset function. * Dicts: - New function xbt_dict_new_homogeneous(void(*)(void*)) to create homogeneous dictionaries, where all the elements share the same free function. Non homogeneous dictionaries will be deprecated in the next release. - Dicts of scalar elements (xbt_dicti_*) are deprecated. - Multi-level dictionaries are deprecated. * Dynars: - new function xbt_dynar_search_or_negative() that is useful when you have less than 2 million elements in your dynar and don't want of the extra complexity of catching exceptions when the element is not found. * Portability layer - Make xbt_os_thread module (for thread portability) public. Documentation is still to come, sorry. * mmalloc module: - Cleanups and simplifications to make it maintainable again. - Exotic features (such as memalign and valloc) were removed. - The metadata were extended and improved so that the model-checker becomes able to explore and inspect the heaps. - This may induce a performance drop when enable_model-checking is ON in cmake (even if it's not used in the simulation), but it is necessary at this point to get MC working. Turn model-checking OFF if simulation performance matters to you. Not enabling it at runtime is not enough, disable it in cmake. -- Tue May 15 11:30:19 UTC 2012 Da SimGrid team SimGrid (3.6.2) stable; urgency=low The "Not coding new stuff allows to polish old things" release. General * New bindings to the NS3 packet level simulator (experimental) * Use the raw (efficient) execution contexts instead of the sysv (portable) ones when possible. * libpcre is now mandatory in any cases since not using it led to severe performance loss and possibly other issues * Update the XML platforms: - G5K: include the latest machine in Nancy - GridPP and LCG: new platforms * Documentation was partially updated, at least (more to come) Bug fixes, cosmetics and small improvements * Free terminated processes before the end of the simulation to avoid exhausting the memory of users having very dynamic amount of processes. * Bug fix and cosmetics about canceling non-running tasks * Bug fix about the dot loader's issues when using libcgraph Portability * Create an installer for windows with nsis (amd64 and win32) - Add an hello world project to illustrate simgrid project creation. - Embed libpcre into the Simgrid installer to avoid its compilation burden * The raw execution contexts should work on Apple now * Port to Windows 64 bits - Sysv contexts now have an implementation for this arch - GRAS communication features now support this arch * Drop support for borland compiler on windows - this code was not maintained, and we kinda depend on gcc nowadays * Fix portability issues on kfreebsd/gnu: build error about semaphores * Fix portability issue on unstable ubuntu: linker became picky on argument order -- Wed Oct 5 15:51:01 CEST 2011 Da SimGrid team SimGrid (3.6.1) stable; urgency=low The "Oops, we broke Macs too" release Portability * Fixed contexts detection so that raw ones are used when possible * On Mac, do not use Posix Ucontexts with gcc v4.[1-5] since this leads to a strange error, with user code segfaulting sometimes when the generated code is not perfectly aligned (which is not controllable from the user side, depends on the amount of code) XBT * New macro: CATCH_ANONYMOUS, which is like CATCH(e) but without argument. -- Mon Jun 27 13:59:03 CEST 2011 Da SimGrid team SimGrid-java (3.6) unstable; urgency=low * Initial release. * Split of every thing from simgrid v3.5 into a separate package. -- 2011-10-05 Da SimGrid team SimGrid (3.6) stable; urgency=medium The Summer Release, also known as the "OMG! They Killed Kenny!" version Java and Ruby: * Bindings now constitute their own package, separated from the main one. Rationale: reduce our maintenance nightmare by reducing the module coupling They will soon be released on their own on gforge. * In the meanwhile: svn co svn://scm.gforge.inria.fr/svn/simgrid/contrib/trunk/simgrid-java svn co svn://scm.gforge.inria.fr/svn/simgrid/contrib/trunk/simgrid-ruby GRAS: It is not considered as stable anymore, but experimental. Sorry. * It's not quite deprecated for now because we have no replacement, but it may soon become the case. SMPI * New MPI functions supported: MPI_Comm_disconnect, MPI_Comm_get_name * Fortran: New user-level cache variable to store the rank of the running process. This improves performance by an order of magnitude. * C: New coccinelle script to automatically locate and modify global and local static variables. * Improved SMPI network model with a sender-side gap to account for multiple parallel sends. MSG * New function MSG_comm_get_status(). MSG_comm_test() and MSG_comm_testany() only say if a communication is finished, no matter whether it succeeded or failed. You can call MSG_comm_get_status() to know the status of a finished communication. * New function MSG_task_dsend() to send a task and detach it. When a communication is detached, you are never notified of its success or failure and the memory is released automatically once it is finished. This function is useful when you don't care about the end nor the success of a communication. * Change the prototypes of action replay. Sorry for inconvenience, but this is really more efficient this way (and to adapt your code, you just have to fix the initialization, that shouldn't be too long) * Kill the braindead MSG_task_refcount_dec() function. I guess nobody ever managed to do anything useful with it. * New function MSG_comm_testany(). Similarly to MSG_comm_waitany(), it takes a dynar of communications. It returns immediately and gives the index of a finished communication (if any). * New example: a basic implementation of the Chord P2P algorithm. SURF * New model for multi-core CPUs. You can now use the core attribute to precise the number of cores of a host. This is a basic model. Every process running on the host receives at most the power provided in the DTD (throughput<=power). Total throughput of process cannot exceed power * num_cores. * New peer tag. This peer tag creates a tiny AS comprising a host and a router linked by an up-link and a down-link (possibly asymmetrical). This kind of pattern allows to easily build last-mile model style platforms. Aggregating such patterns in a rule-based AS is thus the technique of choice for modeling large peer-to-peer/volunteer computing/cloud platforms. * New model for Vivaldi routing. We transformed the Vivaldi network model into a Vivaldi routing model (based on the rule-based model). This allows to combine Vivaldi based latencies with last-mile platforms. SIMIX * Added a check for NaN of IEEE754 infinite in the double entries of the smx_user.c file * Introduce a new context factory "raw", highly inspirated from the ucontext factory, but using manually crafted functions in assembly to do the work in an efficient manner. * Allow to change the used context factory at run time, not only at compilation time. Use --cfg=contexts/factory:raw for maximal speed. * Add an option --cfg=contexts/stacksize:N to set the stack size of the user contexts at runtime (only with raw contexts or ucontexts). * Completely rewrote this module to allow parallel execution of user processes. Use --cfg=contexts/nthreads:N to execute user processes with N parallel threads (the default is 1, meaning no parallelism). * Allow to decide dynamically between sequential and parallel modes. When nthreads > 1, you can use --cfg=contexts/threshold:P to run the user processes in parallel only when their number is greater than or equal to P (the default is 2). * Added a check for NaN of IEEE754 infinite in the double entries of the smx_user.c file XBT * New command line option: if you pass --cfg=verbose-exit:0, SimGrid won't output the state of processes when interrupted with Ctrl-C * Add a new function xbt_dynar_to_array that transforms a dynar into a NULL-terminated array. This may solve backward compatibility issues due to the change to return type of SD_simulate. See also: http://lists.gforge.inria.fr/pipermail/simgrid-user/2010-December/002206.html * Add new macros with variable number of arguments. - in xbt/log.h: XBT_DEBUG, XBT_VERB, XBT_INFO, etc. - in xbt/asserts.h: xbt_assert - in xbt/cunit.h: xbt_test_{add,fail,assert,log} - in xbt/ex.h: THROWF and RETHROWF. Define XBT_USE_DEPRECATED if you want to use the old numbered macros like INFO1, INFO2, etc. * Change xbt_die() to accept a format string with arguments, just like printf. * New data structure: xbt_lib_t, like a dict but more general and with better memory handling. INSTR * New configuration options Options triva/categorized and triva/uncategorized can be used to generate graph configuration files for Triva visualization tool. * Configuration option tracing/platform is renamed to tracing/categorized * XBT logging makes tracing error checks easier, new root log hierarchy: instr * New TRACE_user_link_variable interface: User provides the name of the link and the tracing variable to attach to it * the declaration of tracing categories must be done after the environment creation * simpler tracing interface, just one way to declare categories TRACE_category or TRACE_category_with_color, it is up to you * links in the trace file are again identified by their names * trace contains the full platform hierarchy exactly as declared using the ASes * Options tracing/msg/[task|process]:1 groups the process by hosts for both cases, tasks and processes must have names that are unique during the simulation these options generate traces that are suited to gantt-charts, such as the space-time view of Paje * The experimental option tracing/msg/volume is deprecated its functionality may be reincorporated if needed * Buffering The tracing generates a trace file with unordered timestamped events, because of the way the core simulator (surf) works. A script available at the tools directory (fix-paje-trace.sh) can be used to put the events in order. We have changed the tracing so it can generate ordered timestamped events in the final trace, but depending on the simulator (and how much time is simulated) that can lead to a huge memory utilization. It is deactivated by default, but it can be activated using the --cfg=tracing/buffer:1 switch. Build Infrastructure * Define a SIMGRID_VERSION macro in simgrid_config.h. - We are trying hard to keep the API stable, but it may happen that some things change (we're a research project after all, not a nuclear plant operating system). If such things should happen, you could rely on that macro to adapt. - current value: 30600 for 3.06.00, aka 3.6 * Define macro MAKE_SIMGRID_VERSION(major, minor, patch) to help building a number that can be compared with SIMGRID_VERSION. * Add a build option -Denable_debug (set to ON by default): when set to OFF, assertions and verbose/debug logging events are disabled at compile time. -- Tue Jun 21 08:57:43 CEST 2011 Da SimGrid team SimGrid (3.5) stable; urgency=medium Model Checking * New feature to any SimGrid-based simulator: Model-Checking Check SIN#1 for more details. SMPI * New Model SMPI (three-interval linear regression for correction factors) See RR-7426, available at http://hal.inria.fr/inria-00527150 * Ability to use FORTRAN MPI code (through f2c, automatically privatized) * New MPI functions supported: MPI_Get_count(), MPI_Comm_split() * New: RAM folding (see RR-7426 and examples/smpi/NAS/DT-folding) * New: execution sampling (see RR-7426 and examples/smpi/NAS/EP-sampling) * See also src/smpi/README Tracing: Tracing: * Tracing system - Tracing API changes: TRACE_start and TRACE_end should not be called by user-code. They are automatically called by simulators created with SimDAG, MSG and SMPI if the toolkit is compiled with tracing_enabled=ON. Categories declaration and utilization remain the same for MSG and SimDag. - A function was added to the tracing API to declare categories with colors: - TRACE_category_with_color (char *category, char *color) where color must be in the following format "%f %f %f", red, green, blue and red, green, blue are float values in the interval [0, 1] - User can specify NULL as color parameter, or continue calling TRACE_category (cat) On that case, the tracing system will define random colors - The following command-line options are supported: --cfg=tracing/filename:msg.trace --cfg=tracing:1 (activate tracing, needed to use others) --cfg=tracing/platform:1 (categorized resource use) --cfg=tracing/uncategorized:1 (uncategorized resource use) --cfg=tracing/msg/task:1 (task creation) --cfg=tracing/msg/process:1 (process creation, migration) --cfg=tracing/msg/volume:1 (volume of MSG send/recv) --cfg=tracing/smpi:1 (SMPI interface tracing) --cfg=tracing/simdag:1 (allow SimDAG tasks receive categories) - examples of examples/msg/tracing updated * Tracing SimDag - DAXLoader and DOTLoader functions can generate tasks with categories - A new function to attribute a category to SD tasks: TRACE_sd_set_task_category (SD_task_t task, char *category) * Tracing the MPI interface implemented by SMPI - Collective operations are traced with states - Point-to-Point operations are traced with states/links - Tracing activated by a parameter "-trace filename" passed to smpirun during execution (considering that simgrid is compiled with tracing enabled) - To run the simulation with gdb, the simulator accepts --cfg=tracing/smpi:1 to trace SMPI - tesh files to check if smpi tracing is ok - See examples/smpi/NAS/DT-trace * GTNetS tracing re-worked - adaptation to the tracing system of GTNets to cope with modifications regarding the fullduplex mode - new tesh files to check if gtnets tracing is ok MSG * Asynchronous communications through the functions: MSG_task_isend/irecv and MSG_comm_test/wait/waitall * New function: MSG_load_platform_script() to make possible using a lua script instead of XML files to set up platforms * New function: MSG_set_function to associate functions to processes, used when bypassing the parser * New functions: MSG_task_set_name(), MSG_task_set_compute_duration() Platforms: Add some more examples in examples/platforms * Grid'5000: see www.grid5000.fr * *_30000_hosts.xml: various huge files [mainly scalability testing] SURF * Change the XML format. This is a very important modification. SimGrid 3.5 introduces a new hierarchical format based on the notion of Autonomous Systems. Compatibility with old format is ensured through the perl script provided in the install bin directory bin/simgrid_update_xml. It is now possible to build platforms with specific routing mechanism (Full/Dijkstra/DijkstraCache/Floyd) and to easily connect several platforms together. We will try to provide soon set of realistic platforms exploiting these properties (have a look at examples/platforms/ for the moment). * Take the opportunity of the XML format change to be a good XML citizen: rename link:ctn to link_ctn and similar changes (also dealt with by simgrid_update_xml) * Add a new routing scheme (rule-based) using regular expressions. It enables to have an extremely low memory footprint when the underlying routing is simple and can be compactly described. You need to have libpcre4-dev (perl regular expressions) installed if you want to use this routing scheme. * Revive the cluster TAG and allow to easily and efficiently (both in term of memory and speed) connect clusters together. Have a look at teshsuite/simdag/platforms/ to see how this can be done. With this tag, you can create clusters with thousands of tasks at no cost (have a look at examples/platforms/). Note: clusters are implemented as ASes, so there is no need for an enclosing AS tag if you have only one cluster in your platform. * Add new generic functions in the public interface that allows the user to call SURF 'create_resource' methods from your code (same functionality as the XML bypass mechanism but with a much lighter burden). * Add a new model (enabled through command line --cfg=network/model:SMPI) that uses a piecewise linear approximation to produce better results when exchanging small messages. * Add a new parameter to handle correctly full duplex link and account for interferences between uplink and downlink communications (activate with --cfg=fullduplex:1). SIMDAG * Rename the SD_READY (all dependencies are satisfied and task is scheduled) state in SD_RUNNABLE and define a new SD_SCHEDULABLE (all dependencies are satisfied) state. This prevents a confusion between the notion of "ready to schedule" (SD_SCHEDULABLE) used in DAG scheduling and that of "ready to be simulated" (SD_RUNNABLE) used by the simulation kernel. * Change the way a task is considered as ready. Instead of removing dependencies when a task is done, a counter is decreased. This way, it is always possible to reach ancestors thanks to the SD_taks_get_parents function (even after the end of the simulation.) * Change the return type of SD_Simulate from (SD_task_t*) into xbt_dynar_t. This function was in handling a dynar internally and converted it into a NULL terminated array for historical reasons. * New function SD_dotload(char*) to load a DAG described in dot format. This loader and the corresponding examples require the installation of the graphviz library. * Fix a bug in the management of tasks of size 0 in the surf network models. This problem was only visible with SIMDAG and you should thus disregard results produced with earlier versions if you relied on this feature (some tasks were blocked because of this). * Fix a bunch of stuff that prevented to use classical models with SIMDAG even though your applications were doing only point-to-point communications and sequential computations. Now you can really use any model you want (of course, if you create real parallel tasks, which are not implemented in most models beside ptaskL07, this will abort). * Add an example that schedules a DAX on an heterogeneous platform using a Min-Min strategy. * New function SD_workstation_get_current_task() that returns the kind of task currently running on a workstation in the sequential access mode. * Raise some warnings when unexecuted tasks remains at the end of the simulation. This is usually caused by cycles in the DAG. SIMIX * New function: SIMIX_process_set_function() called by MSG_set_function * Change the underlying waiting queue in semaphores so that a process can wait on several of them simultaneously (as in waitany). * Fix the way to handle tokens in semaphores so that all access patterns work: {acquire, acquire_timeout, waitany} / {release, release_forever}. * kill the dirty pimple SIMIX_message_sizes_output() Please use (proper) visualization instead XBT * New data container: setset (set of sets of elements) * New module: mmalloc (mapped malloc, allowing to have several independent segments of malloc) * New function: xbt_dict_cursor_set_data() * New functions: xbt_dynar_sort(), xbt_dynar_compare() * New function: xbt_dynar_is_empty() * New function: xbt_fifo_get_last_item() * Fix xbt_dynar_shrink(): use the right element size. * Fix xbt_dynar_set*(): allow index larger than current size and memset 0 uninitialized areas during expand. * Fix semaphores: previous implementation was severely broken. * Use library init/fini functions for our initialization. - you can use logs and other feature as soon as you want in your code (even before the xbt_init / MSG_init) - xbt_exit is now a no-op and produce a warning when used. GRAS: * Port GRAS to new SIMIX mechanisms. This allows gras users to benefit from the latest improvement to the simulation kernel. * Kill measurement sockets for now. If you rely on them, sorry. This release is not for you. This feature will be reintroduced in the future, but we cannot delay the release any further. * New function: gras_msgtype_get_name(). * Implement gras_agent_spawn in RL too (the prototype changed a bit) * Fix (at last) the pmm example: it should not randomly fail anymore. Build chain: bug fixes and overall polishing * Cmake is now stable enough. Hence, we killed the autotools. * Port to windows ( TM :) * Fix the 'make install' target. No need to use 'make install-simgrid' anymore * Introduce a 'make dist' target compiling a *source* archive 'make package' compiles a binary archive * Compile java files only on need * Add --cd and --setenv command line options to tesh * Out of source builds are not fully supported yet, but we are close * Enable supernovae and optimization flags by default for our users LUA Bindings * Add layer to set up environment directly from lua, without XML. * The effect of gras_stub_generator can be achieved through lua too (check examples/gras/console/ping_generator.lua) -- Wed, 01 Dec 2010 22:09:23 +0100 Da SimGrid team SimGrid (3.4.1) stable; urgency=low The "Polishing easter eggs is probably a good idea" release. This is a bug fixes release only. Java Bindings * Fix a bug preventing the tasks from begin garbage collected. MSG * Fix a bug occuring when a host involved in a communication fails. This was not detected properly by the other peer involved in the communication. Now, it's reported as a network error. SimDag * Warn the user about loop dependencies in data flow of DAX files * Obey the control-flow dependencies of DAX files Cmake * Add option "enable_smpi" allowing to not compile SMPI. Probably useful for the (Mac) users experiencing a build error here * Improve the detection of lua5.1 and ruby1.8 -- Da SimGrid team Tus, 04 May 2010 28 16:11:16 +0100 SimGrid (3.4) stable; urgency=low The "Easter in Cargese" release. Also known as (major changes): * the "se habla Java, Ruby 話せます, fala-se Lua (and deaf-friendly)" ~> bindings were greatly improved ~> new tracing infrastructure for better visualization introduced * the "Welcome to configury modernity" release. ~> we switched from autotools to cmake, and improved our cdash A more detailled list of changes follow (full detail in svn log). Java Bindings: Various Cleanups * (install java-gcj-compat-dev on debian-like to use them) * Remove put/get: no need to export deprecated interface in Java Use send/receive instead. * Cleanup the examples and add a README per directory * Remove example autoDestination (that's the only way to go now) * Remove example explicitDestination (was a plain copy of basic) * Make JniException a runtime exception, so that there is no need to declare the fact that you may encounter such a beast. I guess that nobody will ever want to survive such error. * Create specific errors for each MSG case of failure: host failure, transfer failure, timeout, task cancelled * Cleanup the exceptions that may get thrown by each function * Other internal cleanups in Java bindings. Performance still bad :/ Ruby and Lua Bindings: create them * (install ruby1.8-dev/liblua5.1-0-dev on debian-like to use them) * That's new and great, you should try them out. Same functionalities than Java bindings, only even less polished SimDag: * Kill the useless "rate" argument of SD_task_get_execution_time() Everyone used to provide -1 as a value, it was not used, and the semantic of a possible use wasn't even clear. * SD_SCHED_NO_COST: Constant to use as cost in SD_task_schedule() either as comm costs or compute costs to mean that there is no such thing for that specific task. * Add a SD_task_set_name() function * Fix SD_task_unschedule() on typed tasks * Fix SD_task_get_execution_time() to return seconds, not flop*sec * In DAX loader, accept useless 'level' attributes to since LIGO DAGs have them (seem to be to ease graphical representation). MSG: * Add an example masterslave_mailbox.c using send/receive and not the deprecated put/get interface. * Kill the MSG_paje_output() function. It's a noop since 2 years. * Kill MSG_WARNING and MSG_FATAL return codes: they were not used anywere in source. * Rename MSG_TIMEOUT_FAILURE into MSG_TIMEOUT for sake of logic (declare MSG_USE_DEPRECATED to still have the old name) * Add a MSG_task_set_data() function * About trace replay (see examples/msg/actions): - implement barrier - Allow to work with splitted trace files for each process Give the specific trace file as argument of each process, and call MSG_action_trace_run(NULL) You can still have one merged file for all processes. - Fix implementation of collective operations * Allow task_execute() on 0-sized tasks (closes #10063) SMPI: * This is the first release of SimGrid where SMPI is not considered beta anymore (even if some corners should still be improved) * Port over the new SIMIX_network submodule (internal refactoring) * Basic support to log events as with SMPE (use --cfg=SMPE:1) * Implement more missing elements of the standard: - MPI_COMM_SELF - MPI_MAXLOC MPI_MINLOC + all associated datatype MPI_DOUBLE_INT, MPI_FLOAT_INT, etc. - MPI_Address() MPI_Get_count() MPI_Type_free() MPI_Type_extent() MPI_Scan() MPI_Get_processor_name() - Added implementation of missing case for Alltoall (warning: it's *not* the bruck variant from OpenMPI; based on Alltoallv instead) - SMPI_MPI_Gather() SMPI_MPI_Gatherv() SMPI_MPI_Scatterv() SMPI_MPI_Reduce_scatter() SMPI_MPI_Allgather() SMPI_MPI_Allgatherv() * Bug fixes include: - MPI_Waitsome() was broken - Allow relative includes in smpicc - Command line cfg argument 'reference_speed' was ignored... - Some functions did not properly lead to auto-benching of user code - smpicc passes -O2 by default (just like openmpi one) SIMIX: * add SIMIX_action_suspend() and SIMIX_action_resume() functions * Bug fixes about timeouts during communications * add SIMIX_message_sizes_output() as a pimple to write to file the amount of messages per size. Use gnuplot to get histogram. Pimple because that's the only user-visible function of simix, defined directly in xbt.h (irk, sorry) * About semaphores: - Add a SIMIX_sem_get_capacity() function - Fix interactions with processe resume/suspende - release_forever() was stupidly broken - Fix SIMIX_display_process_status() for processes in a semaphore - Make SIMIX_sem_block_onto() user-visible * Refactoring context stuff: - Use pseudo-OOP for better modularity - reimplement SIMIX_process_kill() without process_schedule() so that the latter can take as invariant that it is called from maestro. - Merge context_start into context_new for sake of simplicity SURF: * Add a Vivaldi network model, coded live during SUD'10 ;) * Rename configuration variables to start a hierarchy: o cpu_model -> cpu/model o network_model -> network/model o workstation_model -> workstation/model * New configuration variables: o network/bandwidth_factor: correction to bandwith o network/latency_factor: correction to latency o netwotk/weight_S: correction to the weight of competing streams * Add a long description to the models, that users can see with such argument on the command line: --cfg=cpu/model:help * --help-models display the long description of all known models XBT: * config: add the ability to set a default value after registration Does not override any previously set value (e.g. from cmd line) * dict: allow to have integer key and data. When so, you need to use the following functions void xbt_dicti_set(xbt_dict_t dict, uintptr_t key, uintptr_t data); uintptr_t xbt_dicti_get(xbt_dict_t dict, uintptr_t key); void xbt_dicti_remove(xbt_dict_t dict, uintptr_t key); In contrary to regular dicts, the key is not malloced before copy. Mixing scalar and regular elements in the same dict is not tested (but may work). * Allow to use xbt_dynar_shrink() to expend the dynar instead Tracing for Visualization: * SimGrid is now instrumented in order to generate a trace file for visualization analysis: to use it, need to compile SimGrid with the "tracing" option enabled, and instrument the program using SimGrid with TRACE_start, TRACE_category, TRACE_msg_set_task_category and TRACE_end (among other functions). * The instrumentation only traces the platform utilization for now * Documentation to use the tracing functions and how to analyze the traces with the Triva tool is written. * More information about: SimGrid FAQ (in the section Tracing Simulations for Visualization) Build system: * We moved to cmake as default build system. Autotools support will be dropped soon. Check the FAQ for more info about how to use it. * Greatly improved our cdash/ctest interactions Check http://cdash.inria.fr/CDash/index.php?project=Simgrid * Added memory checking tests with valgrind; lot of memleak fixing. This may be the first release of simgrid with so few memory issues * Added code coverage tests. Our coverage is still improvable, but at least we see it on cdash. -- Da SimGrid team Wed, 28 Apr 2010 28 17:11:16 +0100 SimGrid (3.3.4) stable; urgency=low The "Desktop Grid needs love too" release (also called Xmas release). Models improvements: * Major speedup in the maxmin system solving by using lazy evaluation Instead of solving completely the maxmin system at each iteration, only invalidate (and recompute) the modified parts. This new feature is enabled in default models but you can try to turn it on with "--cfg:maxmin-selective-update=1" for other models. * Cas01 IMproved as default CPU model This CPU model is the same Cas01 model, but it uses the maxmin-selective-update flag and a heap structure to manage actions on SURF kernel. It reduces the complexity to find the next action to finish and, consequently, it's faster than the old Cas01. This is the new default CPU model (Cas01). * Rename the old Cas01 model to Cas01_fullupdate Keep the old cpu model Cas01 with the new name of Cas01_fullupdate. Use "--cfg=cpu_model:Cas01_fullupdate" to use the old default CPU model. * CpuTI (CPU Trace Integration) A new CPU model whose objective is simulate faster when using availability trace files. Instead of using a full featured, over engineered maxmin system for CPU modeling, this model does the pre-integration of traces files to calculate the amount of CPU power available, and so, executes faster than the old CPU models. Use "--cfg=cpu_model:CpuTI" to change to this CPU model. * Use LV08 as default network model since it gives better accuracy for small messages and shouldn't change things for big ones. Use --cfg=network_model:CM02 to get the previous behavior. ****************************************** *DO NOT MIX 3.3.4 RESULTS WITH OLDER ONES* ****************************************** * The new CPU model may changes simulations! The point is that events occurring at the exact same timestamp are not scheduled in the same order with the old and new version. This may be enough to completely change the execution of simulations in some cases. * The new network model will change simulations! This new model is more realistic than the previous one, so you should consider redoing your old experiments with this model. Sorry for the inconvenience. Build System: * Introduce the supernovae compilation mode When compiled that way, the whole SimGrid (or almost) is put in a single compilation unit and compiled in one shoot. This is to help gcc which has difficulties to inline stuff from one file into another. The speedup seem to be above 15%, althrough more tests are needed on amd64 to confirm that gain. MSG: * Port of MSG's mailbox on top of SIMIX network The put/get mechanism was greatly simplified on the way. SIMIX: * New SIMIX network module. Provides: - Mailbox: rendez-vous mechanism to find with who you want to speak - Synchronous send/recv: easier and hopefully faster since the logic is handled in the maestro process directly now - Asynchronous send/recv: you dreamt of it? It's here now Too bad that nobody cared enough to propagate the change to MSG. * Add semaphores as SIMIX synchronization mechanism. SimDag: * new function SD_daxload(char*) to load a DAX file (see http://vtcpc.isi.edu/pegasus/index.php/WorkflowGenerator) * Introduce typed tasks. Specify its kind and cost at creation. At scheduling, just give where it should be placed, and the cost for each involved resource is automatically computed. Existing constructors so far (more to come of course): - SD_task_create_comm_e2e() for end-to-end communication - SD_task_create_comp_seq() for sequential computation Use SD_task_schedulev() / SD_task_schedulel() to schedule them. * new function SD_task_dump() for debuging display * new function SD_task_dotty(task,FILE*) writing to file the info about the task in dotty format * SD_task_dependency_exists() can now cope with having one of its arguments NULL. If so, it tests whether the other argument has any dependency. * Add getters on list of preceding/following tasks: SD_task_get_parents(task) and SD_task_get_children(task) * Add getters on amount of workstations and list: SD_task_get_workstation_count(t) and SD_task_get_workstation_list(t) * Add getter on task kind: SD_task_get_kind(task) * Update the start_time and finish_time of tasks on completion/failure * Bugfix: Remove task from state swags when destroyed GRAS: * New function: void gras_cpu_burn(double flops) -- a simple CPU burner XBT: * New function: xbt_dynar_dopar(dynar,fun) to map a function over the dynar with one separate thread per value of the dynar. * Change the prototype of xbt_thread_create(), sorry. Added a boolean parameter indicating whether we want to join this thread (used in SG only for now) * Implement xbt_thread_join and xbt_thread_yield in SG also. Bug fixes: * GTNetS wrappers should now be usable again (and betterly tested too) * Fix a major regression from 3.2 where the timeout provided to MSG_task_put_with_timeout() was used as absolute time before which the comm should be done. * Start to fix the tag. - Internal links should be good now (beside of the loopback, which use the private link instead) - paths to the external world is still rather broken - the tag is just broken. Actually that's brain-dead. We need sth like to make it less stupid ** Check your platform with teshsuite/simdag/platforms/flatifier ** * Fix a source-level compatibility glitch from 3.2: after defining MSG_USE_DEPRECATED, you can use the old name MSG_task_put_with_time_out() for MSG_task_put_with_timeout() * Allow to compile from the SVN with automake 1.11 * Fix some problems when using the "start_time" tag in deployment XMLs. * Fix #8569: XBT/synchro.h has redundant declarations * Fix #8563: MSG return values and exceptions Introduce a MSG_TIMEOUT_FAILURE return code and use it consistently. * Integrate patch #8636: Obey DESTDIR when installing documentation. Thanks to Robson Peixoto. * Fix a vicious bug in dictionaries inducing that some elements were not freed on xbt_dict_free() Portability report of this version: * Main portability targets: - linux (ubuntu (804/810/910) /debian (4/5/testing) /fedora (core11)) on (amd64/i386/ia64) - mac leopard on i386 Known problems: http://cdash.inria.fr/CDash/index.php?project=Simgrid but nothing critical. * Other platforms: windows, AIX and others were not tested for this release Timing report of this version: * Lazy evaluation brings arbitrary speedup (ie, speedup depending on scenario parameters). From 8h to a few seconds in desktop grid settings. * Supernovae brings about 25% speedup on i386. -- Da SimGrid team Thu, 24 Dec 2009 19:07:39 +0100 SimGrid (3.3.3) stable; urgency=low The "Need for Speed" release. The timings done to validate the 3.3.2 were faulty. Instead of being 5% faster, it was 15% slower (compared to 3.3.1). The problem was a conversion from a manually handled vector to xbt_dynar_t on the critical path. xbt_dynar_foreach calls functions, inducing stack management crap. We inlined these functions and xbt_dynar_foreach is now breath taking. We also inlined xbt_swag_belong on the way. Here are some approximate speedup measurements (on master/slaves simulations lasting between 10s and 20s each): 3.3.1 -> 3.3.2: about same performance 3.3.2 -> 3.3.3: 40% speedup 3.3.1 -> 3.3.3: 40% speedup 3.3.1 with inline patch -> 3.3.3: 30% speedup Our reading is that the refactoring which occurred in 3.3.2 made us suffer much more from the xbt_dynar_foreach low performance, but once we solved this, this refactoring proved to be very performance effective. From the 40% speedup, somehow, 10% are due to the inlining and 30% to the refactoring. That's a pitty that gcc cannot inline functions placed in other files alone. We have to choose between: - break the encapsulation (by putting private data structures and accessors in headers files to help gcc) - live with low performance - switch to a decent compiler such as icc (not quite possible). -- Da SimGrid team Thu, 20 Aug 2009 21:21:33 +0200 SimGrid (3.3.2) stable; urgency=low The "Simplicity does not preceed complexity, but follows it" release. The main contributors of this release were (lexical order): Silas De Munck, Stéphane Genaud, Martin Quinson, Cristian Rosa. SURF: * Extract the routing logic into its own object. (was dupplicated in network.c and workstation_LV07.c; Allows to implement other ways of storing that info) => kill now useless network_card concept - Use dynar to represent routes (instead of void** + int*) - kill link_set (use surf_network_model->resource_set instead) - Add a command-line option to choose the routing schema to use - Add three new models: * Floyd (shortest path computed at initialization) * Dijikstra (shortest path recomputed all the time) * Cached Dijikstra (shortest path computed on need) All these models where contributed by Silas De Munck, and are described in his ICCS09 paper. * Simplify model declaration (less redirections, less function to write when defining a model) - Factorize stuff between models: - model_init/exit - Set of resources: surf_model_resource_set(model) surf_model_resource_by_name(model, name) - Unify the types of models in s_surf_model_t (using an union) - Embeed fields of common_public directly into s_surf_model_t - Rename model methods: action_free ~> action_unref action_change_state ~> action_state_set action_get_state ~> action_state_get - Change model methods into functions : (model)->common_public->action_use ~> surf_action_ref * Implement a generic resource; use it as ancestor to specific ones (allows to kill duplicated code in models) Drawback: timer command don't need no name nor properties; workstation_CLM03 don't need no properties (but I guess we can live with those few bytes wasted) * Improve the action object model - implement a constructor avoiding dupplicated code about field initialization in generic_action part. * Kill the SDP model: it has an external dependency, is deprecated in flavor of modern lmm models, and didn't compile since a while SIMIX: * Relocation of the context module from XBT to SIMIX. (the context were decoupled from the simix processes, duplicating a lot of code) => a lot of code was factorized - less overhead is introduced during scheduling - simpler API for the context factory - the logic for process creation,destruction and manipulation was simplified * Simplification of the s_smx_process_t data structure. => accesing the simix level data associated to a process is faster now, and the code is a lot more readable. SMPI: * Implement some more MPI primitives: MPI_Bcast, MPI_Waitany, MPI_Waitall, MPI_Reduce, MPI_Allreduce, MPI_Scatter, MPI_Sendrecv, MPI_Alltoall -implementation: Bcast: flat or 2-ary tree (default), Barrier: 4-ary tree, Reduce: flat tree Allreduce: Reduce then Bcast Alltoall: "basic_linear" if data per proc < 3Kb, "otherwise pairwise". Not yet implemented: "Bruck" for data per proc < 200b and comm size > 12 Alltoallv: flat tree, like ompi Scatter: flat tree * Add support for optimized collectives (Bcast is now binomial by default) * Port smpirun and smpicc to OS X SimDag: * Kill SD_link_get_properties: hard to maintain and makes very little sense Shout out if you used it. GRAS: * Display the list of still queued messages in SG mode when existing the process. XBT: * Add xbt_set_get_by_name_or_null() [Silas De Munck] * Add xbt_graph_node_get_outedges() [Silas De Munck] * Add xbt_str_from_file(FILE*) * Add xbt_dict_get_key achieving a linear reverse search * Remove the context module Portability report of this version: * Main portability targets: - Linux(debian)/x86/context - Linux(debian)/x86/pthreads - Linux(debian)/amd64/context - Linux(debian)/amd64/pthreads On these, we still have the eratic breakages of gras/pmm and amok/saturate_sg reported in previous version. We still think that the tests are the cause of the fault, not the tested code. - Mac OSX Leopard/x86/context Still false negative in tesh autotesting. Smpi still fails, but this time because readlink does not accept -f Everything seems to work properly beside of that. * Exotic platforms: - AIX version 5.3 (only tested contexts this time) Smpi still fails there because mktemp is not installed. Everything seems to work properly beside of that. - OpenSolaris 11 I managed to compile it for the first time, but several breakages. Won't delay the release for this exotic platform. * Windows: it's still lagging behind. If you want to help, please stand up. Timing report of this version: This version seem to be more than 5% faster than 3.3.1 (on linux 64bits with contextes). The gain is less than expected, we are investigating this for next release. -- Da SimGrid team Wed, 19 Aug 2009 17:07:12 +0200 SimGrid (3.3.1) stable; urgency=low OVERALL CHANGES: * Implement a --cfg-help to show existing configuration variables * Build chain do not require doxygen in maintainer mode GRAS: * fix a bug on struct sizeof computation, which prevented the exchange of arrays of structs in some conditions - added a regression test about this in datadesc_usage * Allow the exchange of 0-long dynamic vectors. - for that, use -1 as indicator of dynamic size instead of 0 - This implied to change any size from unsigned long to long, reducing a bit communication abilities, but I guess that with 64bits being quite common, this is more than enough. - This also induce a protocol change, thus bumping network protocol version from 0 to 1 (if we have external users, we have to get clean on that point too ;) - added two regression tests about this in datadesc_usage * Be more verbose when propagating local exceptions This helps debugging. * Display the status of simulated processes when receiving SIGINT in simulation mode MSG: * Allow to control the simulation from a trace file. New functions MSG_action_register() and MSG_action_trace_run() The first one allows to associate a function execution to each kind of action while the second one parses a trace file and triggers the corresponding actions within the system. For now, only a toy example is provided in examples/msg/actions * Add an exemple of process migration in examples/msg/migration * Fix a bug in task exchange which broke MSG_task_get_sender() Add a teshsuite regression test for that. [Bug: if MSG_task_get_sender() is called after sender exit, bad things happen] * Fix a bug which prevented suspend/resume to work properly * Display the status of simulated processes when receiving SIGINT This fixes a regression of v3.3. due to the introduction of SIMIX * Bug fixing in failure management: - trace could not start by a failure at time 0 - failure during communications were not working SIMIX: * Add SIMIX_process_set_name() to change the name of the current process in the log messages. * Store smx_hosts in a dict since we only retrieve them by name * Move the configuration infrastructure to surf SIMDAG: * Move the configuration infrastructure to surf SMPI: * Massive internal cleanups: - Store internal structures on processes instead of hosts (allows to have more than one process per host, in addition of being more logical) - Cleanup the initialization/finalization process - Kill a whole bunch of unneeded synchronization: processes run in exclusive manner within the simulator - Move queues from global tables to process data fields * Improve smpirun: - now accept -platform and -hostfile arguments - Pass the right rank value to processes according to the hostfile * Compile the examples by default, and use them as regression tests * Implement MPI_Wtime() * Change the reference speed to a command line option SURF: * TCP_gamma can now be specified as command line option using --cfg=TCP_gamma:10000000.0 * Change the --surf-path cmd line option into --cfg=path: XBT: * Also include strbuff from xbt.h public header * xbt_ex_display(): do not free the exception after displaying This allows to do more with the given exception afterward. Users should call xbt_ex_free() themselves. Portability report of this version: * Main portability targets: - Linux(debian)/x86/context - Linux(debian)/x86/pthreads - Linux(debian)/amd64/context - Linux(debian)/amd64/pthreads These targets fail about 1/10 of times on gras/pmm, but we believe that this is because of the test, not because of simgrid. amok/saturate_sg fails even more rarely, and the test may not be the problem. - Mac OSX Leopard/x86/context The test suite still spits tons of errors because some obscure force prevents us from removing the temporary directories arguing that they still contain some metadata I've never heard of. Smpi fails because seq is not installed. Everything seems to work properly beside of that. * Exotic platforms: - AIX version 5.3 (both contexts and pthread) Smpi still fails there because mktemp is not installed. XML inclusions seems rosty on AIX. * Windows: it's still lagging behind. If you want to help, please stand up. -- Da SimGrid team Sat, 27 Jun 2009 00:14:30 +0200 SimGrid (3.3) stable; urgency=high OVERALL CHANGES: * JAVA BINDINGS for MSG (you dreamt of them? We made them) [Malek Cherier & Mt] * Introduce the SIMIX module: factorize code between MSG and GRAS. [Bruno Donassolo] Until now, GRAS were using MSG as an interface to SURF. It was quite difficult because both interface have several differences (MSG channels vs GRAS sockets were the most notable point). This also opens the gate to SMPI (which should occur soon) and speed up simulations by to 40% (even if it were not the main goal). ************************************** *DO NOT MIX 3.2 RESULTS WITH 3.3 ONES* Simix may changes simulations! ************************************** The point is that events occuring at the exact same timestamp are not scheduled in the same order with the old and new version. This may be enough to completely change the execution of simulations in some cases. Sorry for the inconvenience. * Cleanup and upgrade the XML format to push further scalability issues (check http://hal.inria.fr/inria-00256883/ for more info) * Improve the testing infrastructure with tesh. Now a very large part of the code is tested not only by being run but also by checking that the output match an expected output [Mt]. * Move on to FleXML v1.7 for the embeeded XML parsers. This version is really less memory-demanding, which should allow you to use larger files in SimGrid [AL]. * Inform valgrind about our contextes, so that it becomes usable with the default (and more effecient) version of SimGrid [contributed by Sékou Diakite, many thanks] GRAS: * Introduce a listener thread in charge of receiving incoming messages from the network. It allows to overlap communication and computation but most notably, it removes some stupid deadlocks due to the fact that so far, a process could not send and receive at the same time. This made most non trivial communication schema impossible. * Convert the PIDs from long int to int to match the MSG ones (and linux ones too) [Mt] * New function: gras_agent_spawn() to launch a new process on current host. Only working in simulation for now. [Mt] * New function: gras_os_hostport() returning a constant form (ie, not needing to be freed) of "gras_os_hostname():gras_os_myport()" XBT: * Make the backtrace of exceptions more human readable [Mt] * New module: xbt/str [Mt] a ton of string utility functions (split, join, printf to a newly allocated buffer, trim, etc) * New module: xbt/hash [Mt] SHA1 hashing algorithm (more to come if needed) * New module: xbt/synchro [Mt] synchronization tools (mutex and conditions) working the same way in simulation and in real life (mainly useful for GRAS, but not only). * New module: xbt/queue [Mt] classical producer/consumer synchronization scheme * xbt_dynar_new_sync() creates a synchronized dynar. All access (using the classical functions will get serialized) [Mt] * Make dictionary internal table dynamic. No need to specify its size anymore; functions xbt_dict_new_ext() and xbt_dict_hashsize_set() thus dropped. [Mt]. * Make sure the log channels are organized as a tree under windows (because of ANSI C compatibility issue, any channel were child of root directly) [Mt]. SURF: * Cleaned many thing in surf and fixed a few bugs [AL]. * Add a nice command line configuration mechanism to compose models [AL]. * Add a new model for parallel tasks (ptask_L07) that is less buggy than the previous one (KCCFLN05). It relies on something that looks like a max-min sharing mechanism but cannot be written as such. A new solver was thus designed [AL]. * Add a new solver to lmm. Based on Lagrange optimization and gradient-based descent, it enables to efficiently maximise systems s.a sum f_i(x_i) s.t Ax<= b with A_{i,j}>=0 and f_i a concave function. This solver enables to propose two new network models for TCP Reno and TCP Vegas based on Low's work. These models still need to be fully tested though [Pedro Velho]. SIMDAG [AL]: * Bug fix in SD_simulate. Now the time bound given as argument is used. * Use the new parallel task model (ptask_L07) as default. * Use the SURF command line configuration mechanism. * 0-size tasks (for synchronization) should now work. -- Da SimGrid team Sun Apr 12 05:20:36 CEST 2009 SimGrid (3.2) stable; urgency=high OVERALL CHANGES: * Port to windows. We still experience issues on this platform, but we believe that at least MSG is usable. GRAS API BREAKAGE (for simplification purpose, sorry): * the gras_msgtype_by_name is not used anymore. Instead of gras_msg_send(toserver, gras_msgtype_by_name("request"), &request); you can write (and must) gras_msg_send(toserver, "request", &request); - If you still want to pass a gras_msgtype_t to the function (to cache the type and avoid the lookup time), use the gras_msg_send_() variant. - Impacted functions: gras_cb_register, gras_cb_unregister, gras_msg_send, gras_msg_wait, gras_msg_rpccall, gras_msg_rpc_async_call, gras_msg_wait_ext * The callbacks are now expected to return 0 when everything went well (just like the main() function) GRAS new features and improvements: * New module mechanism where user code can use per process globals [Mt] This is similar to gras_userdata_*() functions, but for libraries. It factorize some code developped over and over in the examples and AMOK. It has still to be documented and used (only amok/peermanagement is converted for now). * Fix a vicious bug in the TCP buffering mechanism which leaded to message loss when they were small enough to fit into the buffer and sent quickly enough so that they can all get received in one shoot. * gras_datadesc_by_name and gras_msgtype_by_name: now raise an exception if not found. Use the *_or_null() variant for the old semantic. * In gras_msg_handle, do not discard messages without callback. They are probably messages to be explicitly awaited later (ie, proofs of mis-synchronization in userland since they are sent before being awaited) No big deal usually. * gras_socket_meas_send/recv: semantic changed! The numerical arguments used to be (1) the total amount of data to send and (2) msg_size. This was changed to (1) msg_size and (2) amount of messages. This was need for the fool willing to send more than MAXINT bytes on quite fat pipes. AMOK: * Do really rename the hostmanagement module to peermanagement. [Mt] Ie, rename functions from amok_hm_* to amok_pm_*. This breaks the API, but this is rather new and this was documented in the module documentation (poor excuses, I admit) * Bandwidth measurement semantic changed! This follows the changes to gras_socket_meas_send/recv explained above. SIMDAG: * A sequential mode has been added to the workstations. When a workstation is in sequential mode, it can execute only one task, and the other tasks are waiting in a FIFO. [Christophe Thiery] SURF: * The KCCFLN05 workstation model now handles parallel tasks. It is the model for SIMDAG. [Christophe Thiery] * Bug fix in the maxmin solver: Some values were close to 0 instead of equal to 0, which caused some bad behaviors in saturated_constraint_set_update. I now use a threshold mechanism like in surf. [AL] XBT: * When running manually src/testall, you select specific units [Mt] testall is the result of our cunit mechanism, and should replace all the scripty thingy around since bash don't run easily on billware. * A mallocator system has been added. [Christophe Thiery] Mallocators allow you to recycle your unused objects instead of freeing them and allocating new ones. Documentation update: * FAQ reworking + New FAQs: - "Valgrind spits tons of errors!" [Mt] - "How to repport bugs" [Mt] - "Cross-compiling a Windows DLL of SimGrid from Linux" [Mt] - "What is the difference between MSG, SimDag, and GRAS?" [Mt] - Communication time measurement within MSG [AL] - I experience weird communication times when I change the latency [AL] * GRAS tutorial [Mt] It contains: - an introduction to the framework and to the used communication model - an initiatic tour introducing the most proheminent features: o Part 1: Bases . Lesson 0: Installing GRAS . Lesson 1: Setting up your own project o Part 2: Message passing . Lesson 2: Exchanging simple messages . Lesson 3: Passing arguments to the processes (in SG) . Lesson 4: Attaching callbacks to messages . Lesson 5: Using globals in processes . Lesson 6: Logging information properly . Lesson 7: Using internal timers . Lesson 8: Handling errors through exceptions . Lesson 9: Exchanging simple data . Lesson 10: Remote Procedure Calling (RPC) . Lesson 11: Explicitely waiting for messages . Recapping of message passing features in GRAS - A HOWTO section containing: o HOWTO design a GRAS application More are due, of course. They will come latter. In the meanwhile, you can check the examples which are still here. -- Da SimGrid team Fri Mar 16 21:11:46 CET 2007 SimGrid (3.1) stable; urgency=high General: * Port to gcc 4.x There was a stack corruption somewhere, visible only when optimizing with these versions. [Vince] SIMDAG: * This is a NEW module! SimDAG (SD for short) is a revival of the old SG module that enabled to play with Directed Acyclic Graphs. It is built directly on top of SURF and provides an API rather close to the old SG. Some old codes using SG are currently under rewrite to check that all needful functions are provided. [Christophe Thiery] SURF: * Complete rewrite of the KCCFLN05 workstation model. It is now an extension of the classical CLM03 model that gracefully handles failures. This is now the default model for MSG and GRAS. It doesn't handle parallel tasks yet though. [AL] * Bug fix: Weights were not correctly set in the network part. WARNING: This may have resulted in incorrect results with simulations where there are more than one flow on a given link. [AL] SURF, MSG, GRAS: * After a (long ?) discussion on simgrid-devel, we have decided that the convention we had on units was stupid. That is why it has been decided to move from (MBits, MFlops, seconds) to (Bits, Flops, seconds). WARNING : This means that all previous platform files will not work as such with this version! A warning is issued to ask users to update their files. [AL] A conversion script can be found in the contrib module of the CVS, under the name contrib/platform_generation/surfxml_update.pl [MQ] MSG,GRAS: * Bug fix: Processes were started in reverse order, wrt deployment file. WARNING: if your code relies on this bug, please fix it. [AL] * Bug fix: Add a test in MSG_task_execute to stop whenever a task is being executed on two different locations. [AL] * Bug fix: Failures are now better supported thanks to Derrick's tests (there was many failure situations I hadn't thought of and that weren't correctly handled). [AL] * New function: MSG_host_is_avail indicates you whether a given m_host_t is up or down. [AL] GRAS: * New! a real RPC mechanism, as it ought to be since too long. [MQ] Exception occurring on server-side are propagated back to client (!). API CHANGE: the callback changed their prototype. Change: int my_handler(gras_socket_t expeditor, void *payload_data) { to: int my_handler(gras_msg_cb_ctx_t ctx , void *payload_data) { gras_socket_t expeditor=gras_msg_cb_ctx_from(ctx); and you're set. * New! function: gras_msg_handleall to deal with all messages arriving within a given period. * New! function: gras_socket_server_range to get a server socket in a range of port numbers (ease to avoid port number conflicts) [MQ] * New! gras processes display their backtrace when they get a SIGUSR1 or when Ctrl-C is pressed. Use Ctrl-C Ctrl-C to exit. Sweet to debug RL processes [MQ] AMOK: * Bandwidth module: - Do not force experiment sizes to be expressed in kb, or it becomes impossible to measure the latency this way (needs one byte-long tests) WARNING: this changes the amok_bw_* function semantic. [MQ] - Implements the link saturation stuff. [MQ] * Peer management module: New! module factorizing code that we wrote over and over [MQ]. XBT: * New module: cunit (my jUnit implementation in ansi C) [MQ] - Test units are placed directly into the library code, they get extracted automatically and placed into the src/testall binary. - Convert most of the XBT tests to this system. * New functions: xbt_dynar_getfirst_as() and xbt_dynar_getlast_as() [MQ] * XML parsing: rewrote parts of flexml to enable multiple xml parsers to live in the same C code. This required to change a little bit the API of surfxml parsing but shouldn't be an issue for end-users. [AL] * New module: sparse graph structure with basic algorithms (this is work in progress and the API is not considered to be frozen yet). [AL] * Display more information on backtraces: source line & function names are now displayed just like valgrind does (rely on addr2line tool) [MQ] * New function: xbt_backtrace_display(). Sweet while debuging [MQ] * Reworked a little bit some #include statements to load only required headers. Some user code that relied on SimGrid to include stdlib or stdio may need to include it by themselves. [AL] * Fixed xbt/log.h. A missing SG_BEGIN_DECL prevented compilation with g++. [AL] * Renamed xbt_host_t into xbt_peer_t since it betterly describes what I meant. This breaks the API of AMOK and of xbt/config. Sorry about this, but I guess that almost nobody used those parts. [MQ] -- Da SimGrid team Fri, 14 Jul 2006 01:32:27 +0200 SimGrid (3.0.1) stable; urgency=low XBT: * Unfortunately, I had missed 5 misnamed functions: xbt_fifo_item_t xbt_fifo_newitem(void); void xbt_fifo_freeitem(xbt_fifo_item_t); xbt_fifo_item_t xbt_fifo_getFirstItem(xbt_fifo_t l); xbt_fifo_item_t xbt_fifo_getNextItem(xbt_fifo_item_t i); xbt_fifo_item_t xbt_fifo_getPrevItem(xbt_fifo_item_t i); They're now deprecated. Please use their new versions: xbt_fifo_item_t xbt_fifo_new_item(void); void xbt_fifo_free_item(xbt_fifo_item_t); xbt_fifo_item_t xbt_fifo_get_first_item(xbt_fifo_t l); xbt_fifo_item_t xbt_fifo_get_next_item(xbt_fifo_item_t i); xbt_fifo_item_t xbt_fifo_get_prev_item(xbt_fifo_item_t i); [AL] * Bugfix: really disconnect fifo items which are remove_item()ed [AL] * Documentation: xbt_log module unmercifully reworked [MQ] * Bugfix: there was a problem with the ending of contexts with the pthread backend. It caused some weird deadlock or behavior depending on the pthread implementation. [AL] * Bugfix: get the exceptions raised in the simulator repport where and why they come from when they are not catched in time [AL, MQ] SURF: * Bugfix: Do repport the error when two non-connected hosts try to exchange data (Thanks to Flavien for stumbling into this one) [AL] SURF: * Add additionnal checkings on communications. Assert that two communicating hosts are connected by a set of links... [AL] MSG: * Add additionnal checkings on channel values in communication [AL] * New: MSG_task_get_source to see on which host a task was generated [HC] * New: int MSG_task_probe_from_host(int channel, m_host_t host): returns the number of tasks waiting to be received on channel and sent by host. [AL] * New: MSG_error_t MSG_task_get_from_host(m_task_t * task, int channel, m_host_t host); waits for the first task coming from a given host.. [AL] GRAS new functionnalities: [MQ] * Enhance the parsing macro to allow the size of multidimentional objects to be given thru annotations. * New example (and documentation): Matrix Multiplication a la RPC (as when I was young!) and fix a bunch of bugs found on the way. GRAS performance improvements: [MQ] [DataDesc] * Reduce the amount of cbps creation/destruction by making it static to datadesc_send/recv() and using a (newly created) cbps_reset (based on dynar_reset ()) [Virtu] * Change libdata to a set so that we can search for stuff by ID (and thus reduce the insane amount of dict lookups) [Transport] * Actually implement gras_datadesc_copy() so that we don't have to mimick RL communication on top of SG since it's so uneffective. It may also be used for inter-thread communication in RL, one day. * Use gras_datadesc_copy() to exchange messages on top of SG Allows to: - improve message exchange performance on top of SG - deprecate transport_plugin_sg.c:gras_trp_sg_chunk_send() & recv() * Don't exchange on the network the size of the used part of buffer, instead, specify the possible buffer size to read(). Advantages: - reduces the amount of read/write calls (one pair per exchange) - reduces the amount of exchanged data (the size) - allows to retrieve all arrived data on receiver side, if we don't need it right now (subsequent read will peek the buffer) - allows the receiver to proceed with the begining of the stream before everything is arrived - make it possible to build an iov transport (using readv/writev) Extra difficulty: - take care of the data with non-stable storage (like stacked data), and bufferize them. * If possible, TCP send uses vector I/O (when writev() is here) - Don't use it for receive since we send data sizes and data on the same stream, so we wouldn't be able to chain large amount of chunks before having to flush the stuff to read the size. * Rework the transport plugin mechanism to simplify it and reduce the amount of pointer dereferencement when searching for the right function to use. * I guess that now, we do almost as few system calls as possible while doing as few data copy as possible. To improve it further, we could try to send all the sizes first and then all the data (to use iov on receiving size), but it's only a partial solution: when you have 2 dimensional data, the sizes of the second dimension is data of the first dimension, so you need 3 streams. I'm not sure the potential performance gains justify the coding burden. -- Da SimGrid team Fri, 21 Oct 2005 14:42:20 +0200 SimGrid (3.00) stable; urgency=high SURF: * New! Give the possibility to hijack the surf parser and thus bypass MSG_create_environment and MSG_launch_application. Have a look at examples/msg/msg_test_surfxml_bypassed.c to see how it can be done. -- Arnaud Legrand Sat, 20 Aug 2005 23:25:25 -0700 SimGrid (2.96) unstable; urgency=low AKA SimGrid 3 rc 2. XBT: * New! Exception handling with setjmp or such (code from OSSP ex) [MQ] This deprecates the xbt_error_t mechanisms. It modifies (simplifies) all XBT and GRAS API. MSG API keeps unchanged (exceptions raised by XBT are catched from within MSG and masked with existing error handling facilities) SURF: * New! Add a FATPIPE model. [AL] * New! Add a parallel task model. [AL] * New! Add automatically a loopback interface (in the default network model) if none was precised. MSG * Bugfix: MSG_process_resume now works with the current running process. [AL] * New! Add MSG_parallel_task_create and MSG_parallel_task_execute. [AL] * Modification of MSG_task_get_compute_duration. Once a task has been processed, the value returned by this function is now equal to 0. [AL] * New! Add double MSG_task_get_remaining_computation(m_task_t task) and MSG_error_t MSG_task_cancel(m_task_t task). Add a state (MSG_TASK_CANCELLED) to MSG_error_t corresponding to the cancelation of a m_task. For now, MSG_task_cancel only works with computation tasks. [AL] * New! Add double MSG_get_host_speed(m_host_t h) that returns the speed of the processor (in Mflop/s) regardless of the current load on the machine. [AL] * API Change: use proper naming convention for MSG_getClock and MSG_process_isSuspended: MSG_get_clock and MSG_process_is_suspended. [AL] * New! Add void MSG_task_set_priority(m_task_t task, double priority). This function changes the priority of a computation task. This priority doesn't affect the transfer rate. A priority of 2 will make a task receive two times more cpu power than the other ones. This function has been added to suit the needs of Nguyen The Loc and hasn't been that much tested yet. So if it fails, please report it and send me your code. [AL] * API Change: removed all functions and types that were marked "deprecated" since many months. Renamed MSG_global_init_args to MSG_global_init. -- Da SimGrid team Mon, 8 Aug 2005 17:58:47 -0700 SimGrid (2.95) unstable; urgency=low XBT * Steal some nice code to GNU pth to fix context detection and usage [AL] * Cleanup in the xbt_config API; add configuration callbacks. [MQ] * Cleanup in the initialization API: the unused "defaultlog" is dead. [MQ] SURF * Bugfix: Allow absolute paths for platform description files [MQ] * Bugfix: do free the variables after use. Leads to drastic performance improvement [AL] * Implement max_duration (ie, timeouts) on resources [AL] MSG * Implement MSG_config to configure MSG at runtime. xbt_cfg test on a real case ;) [MQ] * Implement MSG_channel_select_from() to help GRAS now that SURF provide the needed support (timeouts) [AL] GRAS (new features) * Implement measurement sockets. You can now get the bandwidth between two hosts thanks to AMOK (see below). [MQ] * gras_datadesc_dynar() builds a dynar type descriptor, allowing to send dynar over the network (yeah) [MQ] * Real (even if simplistic) implementation of gras_os_myname() on RL [MQ] * simple/static token-ring example. [Alexandre Colucci and MQ] * Use MSG_channel_select_from() and remove the *slow* hack we had to put in place before [MQ] GRAS (bug fixes) * Differentiate the types "char[22]" and "unsigned char[22]" in automatic type parsing. "short" and "long" modifiers were also ignored; other modifier (such as reference level) are still ignored. [MQ] * Embeed the buffer size within the buffer itself on SG. [MQ] That way, send() are atomic and cannot get intermixed anymore (at least the ones which are less than 100k; bigger messages still have the issue) * Array size pushed by the field, not by the field type (or each and every long int will push stuff to the cbps) [MQ] * use select() to sleep since it allows to portably sleep less than one second. [MQ] GRAS (minor cleanups) * .Makefile.local (generated from gras_stub_generator) |MQ]: - Do clean .o files - Compile with -g * Type Callbacks now receive the gras_datadesc_type_t they work on as argument. * type category 'ignored' killed as it was never used and were difficult to transmit. * whether a type can cycle or not is now a flag, leaving room for more flags (as "ignored", if we feel the need one day ;) * Rename raw sockets to measurement sockets since "raw" has another meaning in networking community. AMOK * Advanced Metacomputing Overlay Kit introduction. It is based over GRAS and offers features not belonging to GRAS but that most applications need. One day, it may be a set of plugins loadable at runtime. * New module: bandwidth bandwidth measurement between arbitrary nodes running this module. [MQ] -- Da SimGrid team Thu, 30 Jun 2005 16:29:20 -0700 SimGrid (2.94) unstable; urgency=low The first beta release of SimGrid 3 ! >>>Arnaud<<< (documentation) * Update the main page and the FAQ. Adding references to gforge. (gras) * Add a gras_os_getpid function. (msg) * Add MSG_task_get_compute_duration() and MSG_task_get_data_size() * Extend the logs so that they also print PID, hostname, date, ... if available. * Convert the MSG example to the use of xbt_logs instead of PRINT_MESSAGE, and kill the old version which were in testsuite/ * Rewrite tools/MSG_visualization/colorize.pl for using with logs instead of PRINT_MESSAGE (xbt) * Add xbt_os_time(). As the rest of xbt/portability, this is not public for users. Instead, each programming environment (GRAS, MSG,...) use it when needed to provide such a feature to users. Don't shortcut the mechanism or you will also shortcut the virtualization you need on the simulator. >>>Martin<<< (infrastructure) * Cleanups in configury with regard to compile optimization/warning flags. Also add -fno-loop-optimize to any powerpc since it's the optimization killing gcc (< 3.4.0). * Doxygen cleanups: move MSG examples, kill the second Doxygen phase needed by MSG examples complications * Borrow configury beautifications from PHP (xbt) * Bugfix: XBT_LOG_NEW_DEFAULT_CATEGORY now compiles without compiler warning (thanks loris for stumbling into this one). * Bugfix: stop loading private headers (gras_config.h) from the public ones (xbt/swag.h). (gras) * Change SIMGRID_INSTALL_PATH to GRAS_ROOT in Makefiles generated for user. * Rename gras_get_my_fqdn to gras_os_myname and implement it in the simulator RL would imply a DNS resolver, which is *hard* to do in a portable way (and therefore delayed). * Implement a real timer mechanism and use it in timing macros. This allows to avoid rounding errors and get a 0.000005 sec precision in timing macros. While I was at it, various cleanups: - allow to declare more than one timed section per file (fix a stupid bug) - move some private declaration to the right place - merge conditional execution and timing macros into emulation module - document the module - make sure the module cleanups its mess on gras_exit * Documentation improvements: - (new) how to compile applications using GRAS - (new) emulation support (timing macros) -- Da SimGrid team Fri, 13 May 2005 10:49:31 +0200 SimGrid (2.93) unstable; urgency=low Alpha 4 on the path to SimGrid 3 (aka the "neuf-trois" version) [Arnaud] - Use Paje properly where used. Still to be sanitized properly. - Portability fix: Add an implementation of the contexts using pthread [Martin] (misc) - Add xbt_procname(): returns the name of the current process. Use it to show the current process's name in all logging. (infrastructure) - fix detection of older flex version and the reaction, since we do depend on modern ones (we use lex_destroy) - Better separation of SG and RL in the libs: remove all simulation code from libgras. As a result, this lib is now only 200k when stripped. Some of the xbt modules may also be duplicated (two sets and such) and should be cleaned/killed before SG3. - Insist on using xlC on AIX because of weird problems involving gcc there. - Cleanup the make remote stuff. This is now done by scripts tools/graspe-{master,slave} (GRAS Platform Expender). This is still mainly for our private use, but we're working on changing them to user tools, too. (gras) - Bugfix: flush the socket on close only if there is some *output*. - Bugfix: flush idempotent when there's nothing to send (don't send size=0) (msg) - Add MSG_task_get_name. The task names are mainly for debugging purpose, but anyway. -- SimGrid team Fri, 4 Mar 2005 14:32:37 -0800 SimGrid (2.92) unstable; urgency=low Alpha 3 on the path to SimGrid 3 [Arnaud] (gras) - New! First try of benchmarking macros. - New! First try so that gras_stub_generator generate deployment and remote compilation helpers. (msg) - Bugfix: Initialization fix in msg_test. [Martin] (surf) - Bugfix: applied patch to lexer so that it doesn't need a huge heap. (xbt) - Bugfix: let dicts work with NULL content (_foreach didn't) and cleanups (gras) - API Change: gras_os_sleep to take the amount of seconds as a double. Accepting an int was error prone since it was the only location where seconds were coded as such. It leaded to damn rounding errors. - Bugfix: Hard to belive that timers ever worked before this. -- SimGrid team Wed, 23 Feb 2005 22:09:21 +0100 SimGrid (2.91) unstable; urgency=low Alpha 2 on the path to SimGrid 3 [Arnaud] (surf) - Bug fix in the lmm_solver. (msg) - New! Interface to Paje (see http://www-id.imag.fr/Logiciels/paje/) through the function MSG_paje_output. - New! Introducing two new functions MSG_process_kill() and MSG_process_killall(). - It is possible to bound the rate of a communication in MSG with MSG_task_put_bounded() (was already in the previous version but I had forgotten to write it in the changelog). - Bug fix to let GRAS run on top of MSG until we move it directly on top of the SURF. [Martin] (infrastructure) - Various cleanups to the autotools stuff - Begin to move Gras examples to examples/gras/ - Let make distcheck work again (yeah!) (documentation) - documentation overhauled using doxygen. gtk-doc-tools is dead in SimGrid now. - Automatically extract all existing logging categories, and add the list to the documentation (long standing one, to say the less) (gras) - Cleanup the known architecture table. Reorder the entries to group what should be, and use a more consistent naming scheme. (some of the test dataset are still to be regenerated) - New! Allow library to register globals on each process just as userdata does. This is implemented using a xbt_dict and not a xbt_set, so we loose the lookup time (for now). Use it in msg and trp. This cleans a lot the internals and helps enforcing privacy of the headers between the gras components. - New! Add a timer mechanism, not unlike cron(8) and at(1). - Bugfix: gras_os_time was delirious in RL. - Bugfix: gras_trp_select/RL don't run into the wall when asked to select onto 0 sockets. - Reenable GRAS now that it works. -- Arnaud Legrand Mon, 14 Feb 2005 14:02:13 -0800 SimGrid (2.90) unstable; urgency=low Alpha 1 on the path to SimGrid 3 * It is a long time since the last release of SimGrid. I'm sorry about that but as I had told you, I was rewriting a lot of things. I apologize to those who had been reporting bugs to me and that I had not answered. If your bug is still in the new version, please tell me. Here is a summary of the main changes. * REVOLUTION 1: The SimGrid project has merged with the GRAS project lead by Martin Quinson. As a consequence SimGrid gains a lot in portability, speed, and a lot more but you'll figure it out later. SimGrid now comprises 3 different projects : MSG, GRAS and SMPI. I wanted to release the new MSG as soon as possible and I have broken GRAS, which is the reason why, for now, only MSG is fully functional. A laconic description of these projects is available in the documentation. * REVOLUTION 2: I have removed SG and I am now using a new simulation kernel optimized for our needs (called SURF but only the developers should use it). Hence, MSG is now roughly 30 times faster and I think that by rewriting a little bit MSG, I could event speed it up a little bit more. Beside the gain in speed, it is also much easier to encode a new platform model with SURF than it was with SG. More to come... * REVOLUTION 3: I have tried to change a little as possible the API of MSG but a few things really had to disappear. The main differences with the previous version are : 1) no more m_links_t and the corresponding functions. Platforms are directly read from a XML description and cannot be hard-coded anymore. The same format is used for application deployment description. The new format is described in the documentation. Have a look in tools/platform_generation. There is a tiny script that converts from the old platform format to the new one. Concerning the application deployment format, parsing the old one is tricky. I think most of you should however be able to convert your files. If it is really an issue, I can write a C code that does the conversion. Let me know. 2) the toolbox tbx does not exist anymore. We now have a library with much more data-structures but without the hash-tables (we have dictionaries that are much faster). -- Arnaud Legrand Mon, 31 Jan 2005 10:45:53 -0800 ***************************************************************************** * Follows the old GRAS changelog. It does not follow the same syntax, but I * * don't feel like converting the oldies. (Mt) * ***************************************************************************** 2005-01-31 Arnaud Version 2.90: "the long awaited one" - Finished rewriting and debugging MSG. Rewrote the documentation. - disable GRAS for now since it needs to be ported to the newest SG 2004-12-16 Martin - Finish the port to windows (using mingw32 for cross-compile) 2004-11-28 Arnaud - Main loop and datastructures of SURF. A cpu resource object is functional. Surf can thus be used to create cpu's with variable performance on which you can execute some actions. 2004-11-15 Martin Quinson - Port to ARM. Simply added the alignment and size descriptions. Should work, but the ARM machines are so slow that I didn't had the opportunity to 'make check' over there yet. 2004-11-15 Arnaud Legrand - Trace manager now written. It uses a heap structure and is therefore expected to be efficient. It may however be speeded up (particularly when many events occur at the same date) by using red and black trees. One day maybe... - Max-min linear system solver written. It uses a sparse matrix structure taking advantage of its expected use. Most operations are O(1) and free/calloc are called as few as possible. The computation of the minimum could however be improved by using a red and black tree (again ! ;). 2004-11-03 Arnaud Legrand - Rename every gras_* function that was in xbt/ to its xbt_ counterpart. - Add a heap and a doubly-linked list to xbt - Added a dichotomy to the dictionaries. make check works as well before so I assume that the patch is correct. I do not know however if things run effectively faster than before now. :) Inclusion of the SimGrid tree in the GRAS one. The archive is renamed to SimGrid, and the version number is bumped to 2.x 2004-10-29 Martin Quinson - Introduction of the remote errors. They are the result of a RMI/RPC on the remote machine. ErrCodes being scalar values, you can't get the host on which those errors did happen. Extending the error mechanism as in Gnome is possible. No idea yet whether it is a good idea. 2004-10-28 Martin Quinson - Interface revolution: the Starred Structure Eradication. I used to do typedef struct {} toto_t; and then handle *toto_t. Arnaud (and Oli) didn't like it, and I surrendered. Now, you have: - ???_t is a valid type (builded with typedef) - s_toto_t is a structure (access to fields with .) - s_toto is a structure needing 'struct' keyword to be used - e_toto_t is an enum - toto_t is an 'object' (struct*) Exemple: typedef struct s_toto {} s_toto_t, *toto_t; typedef enum {} e_toto_t; Moreover, only toto_t (and e_toto_t) are public. The rest (mainly s_toto_t) is private. - While I was at it, all gras__free() functions want a gras__t* so that it can set the variable to NULL. It was so for dicts and sets, it changed for dynars. - Fix a bunch of memleaks in dict_remove - Fix a bug in sg/server_socket opening: it failed all the time. 2004-10-07 Martin Quinson - Speed up dynar lookup operation a bit. gras_dynar_get is dead. Now, you can choose between gras_dynar_get_cpy (the old gras_dynar_get but should be avoided for efficiency reasons) and gras_dynar_get_ptr (which gives you the address of the stored data). gras_dynar_get_as is an helpful macro which allows you to retrieve a copy of the data using an affectation to do the job and not a memcpy. int toto = gras_dynar_get_as(dyn,0,int); rewrites itself to int toto = *(int*)gras_dynar_get_ptr(dyn,0); It does not really speedup the dynar test because they are setting elements all the time (and look them seldom). But the dict does far more lookup than setting. So, this brings the dict_crash test from ~33s to ~25s (200000 elms). 2004-10-05 Martin Quinson - Allow to (en/dis)able the cycle detection at run time. Whether we should check for cycle or not is now a property of each datatype. When you think there may be some cycle, use datadesc_cycle_set. datadesc_cycle_unset allow to remove this property when previously set. Note that the cycle detection is off by default since it impacts the performance. Watch the data you feed GRAS with ;) This property is hereditary. Any element embedded in a structure having it set have it set for the time of this data exchange. You should set it both on sender and receiver side. If you don't set it on sender side, it will enter an endless loop. If you forget on receiver side, the cycles won't be recreated after communication. - Header reorganization. Kill gras_private.h, each submodule must load the headers it needs. 2004-10-04 Martin Quinson - Interface revolution: do not try to survive to malloc failure. Now, gras_malloc and friends call gras_abort() on failure. As a conclusion, malloc_error is not a valid error anymore, and all functions for which it was the only gras_error_t return value are changed. They now return void, or there result directly. This simplify the API a lot. 2004-09-29 Martin Quinson - Re-enable raw sockets. Created by gras_socket_{client,server}_ext; Used with gras_raw_{send,recv} No select possible. It should allow to kill the last bits of gras first version soon. This is not completely satisfactory yet (duplicate code with chunk_{send,recv}; a bit out of the plugin mechanism), but it should work. - Simplify transport plugin (internal) interface by not passing any argument to _server and _client, but embedding them in the socket struct directly. 2004-09-28 Martin Quinson - Finish the port to AIX. autoconf was my problem (segfault within the malloc replacement function. No idea why) 2004-09-16 Martin Quinson - Fix some size_t madness on 64bit architectures. 2004-09-08 Martin Quinson - Reduce the number of system headers loaded, overload some more system calls (such as malloc to cast the result of the system one, and work properly on AIX) - Fix and reintroduce the config support 2004-09-07 Martin Quinson - Source code reorganization to allow Arnaud to surf all over there. - Allow to document the logging categories. - Remove all uppercase from logging categories and useless cleanup in names. 2004-08-18 Martin Quinson Version 0.6.2 (protocol not changed; API changed) - Interface cleanup: gras_msgtype_by_name returns the type (instead of a gras_error_t), and NULL when not found. Functions expecting a msgtype as argument (msg_wait; msg_send) deal with NULL argument by providing a hopefully usefull message. - Portability to prehistoric sparcs again 2004-08-17 Martin Quinson Version 0.6.1 (protocol not changed; ABI not changed) - prealloc some buffers to speed things up 2004-08-11 Martin Quinson Version 0.6 (protocol not changed; ABI expended) - The parsing macro can deal with the references, provided that you add the relevant annotations (using GRAS_ANNOTE(size,field_name)) 2004-08-09 Martin Quinson Version 0.5 (protocol not changed; ABI changed) - Allow to off turn the cycle detection code in data exchange at compilation time. It should be at run time, but I'm short of time (and the config stuff is still broken). That way, we keep dict out of the critical path, which is good because the performance is poor: - search not dichotomial yet - dynar give no way to access their content and memcpy everytime - In composed data description (struct, ref and so on), stop foolness of keeping the subtype's ID, but store the type itself. This keeps sets out of the critical path, which is good since they rely on dynar and dictionnaries. The only loose of that is that we cannot detect the redeclaration of a structure/union with another content (but I'm not sure the code detected well this error before anyway). We still can detect the redefinition discrepancy for the other types. - Use a whole bunch of optimisation flags (plus -fno-strict-aliasing since it breaks the code because of type-punning used all over the place). This breaks on all non-gcc architectures (for now). All those changes (plus the buffer of last time) allow me to gain 2 order of magnitude on cruel tests consisting of 800000 array of integers on two level of a hierarchical structure (200 secondes -> 4 secondes) API change: - the selector of reference must now return the type it points to, not the ID of this type. 2004-08-06 Martin Quinson Version 0.4 (protocol changed; ABI not changed) - Allow to pass --gras-log argument to processes in simulation mode. Really. - New debugging level: trace (under debug) to see effect of GRAS_IN/OUT - Implement a buffer transport, and use it by default (it relies on tcp in real life and on sg in simulation). That's a bit hackish since I had a new field to the structure to store its data without interfering with the subtype ones. Inheritance is tricky in C. And that's a kind of reverse inheritance with one class derivating two classes. Or maybe a game with java interfaces. Anyway, that's damn hard in C (at least). Moreover, I got tired while trying to ensure plugin separation and genericity in SG mode. MSG wants me to do weird things, so let's go for cruel hacks (temporarily of course ;). See comment in transport_private.h:71 - do not include all the _interface headers in private but in the files which really need them (to cut the compilation time when they are modified) 2004-07-26 Martin Quinson Version 0.3 (protocol not changed; ABI changed) - Major overhault of the datadesc interface to simplify it: - shorted the function names: s/gras_datadesc_declare_struct/gras_datadesc_struct/ and so on - add a trivial way to push/pop integers into the cbps without malloc. This allows to make really generic sub_type description, which simply pop their size of the stack. - add a function gras_datadesc_ref_pop_arr() which does what users want most of the time: Declare a dynamic array (which pops its size of the stack) and declare a reference to it. Poor name, but anyway. - kill the post-send callback, add a post-receive one 2004-07-23 Martin Quinson Version 0.2 (protocol changed; ABI changed) - add some testing for cpbs in the test cases, and fix some more bugs. This invalidate again the little64 data file, since I cannot regenerate it myself. - remove an awfull optimization in the logging stuff, allowing me to: - understand it again - learn gcc how to check that the argument match the provided format - fix all errors revealed by gcc after that - internal keys of dict are not \0 terminated. Deal with it properly in loggings instead of segfaulting when the user want to see the logs :-/ 2004-07-22 Martin Quinson - Fix some stupid bug preventing cbps (callback postit) from working 2004-07-21 Martin Quinson - Some documentation cleanups - remove the useless last argument of msgtype_declare - rename the Virtu functions to fit into the 'os' namespace - move headers src/include -> src/include/gras/ and stop fooling with gras -> . symbolic link - make distcheck is now successful 2004-07-19 Martin Quinson Version 0.1.1 - Build shared library also - Install html doc to the right location - stop removing maintainer files in make clean - build tests only on make check 2004-07-13 Martin Quinson version 0.1 - No major issue in previous version => change versioning schema - Re-enable little64 convertion test now that Abdou kindly regenerated the corresponding dataset. 2004-07-11 Martin Quinson version 0.0.4 - Get it working with any kind of structure (we can compute the padding bytes remotely for all the architectures I have access to) - Implement the structure parsing macro (still not quite robust/complete) - Improvement to the remote testing toysuite 2004-07-10 Martin Quinson [autoconf mechanism] - get ride of a bunch of deprecated macros - actually run the test for two-compliment, not only compile it :-/ - test whether the structures get packed (and bail out if yes. Damn. Alignment is a serious matter) - test whether the structures get compacted (but respecting the alignment constraints of each types) - test whether the array fields of structures can straddle alignment boundaries [base] - Damnit, double are bigger than float (typo in creation of 'double' datadesc) (took me 2 hours to find that bug, looking at the wrong place) - Add gras_datadesc_declare_{union,struct}_close(). They must be used before sending/receiving and are used to compute the offsets of fields - Given that padding size depend even on compiler options, keep track of alignment and aligned_size only for the current architecture. Not a big deal since we send structure fields one after the other (seems reasonable). - Add the datastructure used for IEEE paper by the PBIO guys to the test program, let it work on linux/gcc/little32. portability todo. 2004-07-08 Martin Quinson - import and improve remote compilation support from FAST - make sure make check works on half a dozen of machines out there 2004-07-07 Martin Quinson Let's say it's version 0.0.3 ;) - Implement conversions (yuhu!) - Let it work on solaris (beside conversion, of course) - Stupid me, using rand() to generate the conversion datatests in not wise. 2004-07-06 Martin Quinson - Let make dist work, since I'm gonna need it to compile on remote hosts - Let Tests/datadesc_usage write the architecture on which the file was generated as first byte. - Add PowerPC (being also IRIX64), SPARC (also power4) and ALPHA architecture descriptions. - Add datadesc_usage.{i386,ppc,sparc} files being the result of execution on those architectures. - Optimization: send/recv array of scalar in one shoot 2004-07-05 Martin Quinson - YEAH! GRAS/SG and GRAS/RL are both able to run the ping example ! - Plug a whole bunch of memleaks - each process now have to call gras_{init,exit}. One day, their log settings will be separated - Continue the code factorisation between SG, RL and common in Transport. 2004-07-04 Martin Quinson [Transport] - Redistribution between SG and RL. We wanna have to accept in SG, so move accepted related parts of RL in the common part. (more precisely, the dynar of all known sockets is no more a static in transport.c, but part of the process_data) [Core/module.c] [gras_stub_generator] - Bug fix: Do call gras_process_init from gras_init (wasnt called in RL). 2004-07-03 Martin Quinson - Create a new log channel tbx containing dict, set, log, dynar (to shut them all up in one shot) [DataDesc] - Fix the ugly case of reference to dynamic array. - New (semi-public) function gras_datadesc_size to allow the messaging layer to malloc the needed space for the buffer. [Transport] - gras_socket_close now expect the socket to close (and not its address to put NULL in it after it). This is because the socket passed to handlers is one of their argument (=> not writable). [Messaging] - propagate the interface cleanup from last week in datadesc, ie remove a superfluous level of indirection. User pass adress of variable containing data (both when sending and receiving), and not of a variable being a pointer to the data. Let's say that I like it better ;) The price for that is constructs like "int msg=*(int*)payload" in handlers, but it's a fine price, IMHO. [examples/ping] - Let it work in RL (yuhu) 2004-06-21 Martin Quinson [Transport] - porting SG plugin and SG select to new standards (works almost). - plug memleaks and fix bugs around. [DataDesc] - cleanup the prototype of data recv and force users to specify when they want to handle references to objects. Test case working even for cycles. - plug memleaks. Valgrind is perfectly ok with this. 2004-06-12 Martin Quinson [Transport] - cleanup the separation between plugin and main code in plugin creation 2004-06-11 Martin Quinson [Transport] - Reput hook for raw sockets, needed for BW experiments - kill a few lines of dead code [Data description] Interface cleanup - gras_datadesc_by_name returns the searched type or NULL. That way, no variable is needed to use a type desc once, which makes the code clearer. - gras_datadesc_declare_[struct|union]_append_name is removed. The last two parameters were strings (field name, type name), leading to common errors. [Dicos] Interface cleanup - gras_dico_retrieve -> gras_dico_get ; gras_dico_insert -> gras_dico_set This is consistant with the dynar API. 2004-04-21 Martin Quinson [Messaging] - Porting to new standards. [Data description] - interface cleanup. There is no bag anymore, no need to take extra provision to mask the pointers behind "ID". Better splitup of functions between files create/exchange/convert. This is still a bit artificial since convert and receive are so interleaved, but anyway. [Virtu(process)] - add a queued message list to procdata (the ones not matching criteria in msg_wait) - factorize some more code between SG and RL wrt procdata [Tests] - use gras_exit in example to track memleaks - get rid of gs_example now that GS is properly integrated into gras - update run_test to integrate the latest tests (datadesc) [Logging] - rename WARNINGn macros to WARNn since it prooved error-prone 2004-04-19 Martin Quinson [Data description] - register init/exit functions within gras module mechanism - send/receive function. Convertion is not implemented, but short-cutted if not needed. struct/array elements are sent one by one (instead of block-wise), but nobody really cares (yet). Get a prototype before optimizing. - tests (using a file socket) for DD send/receive on: - base types: int, float - array: fixed size, string (ie ref to dynamic string) - structure: homogeneous, heterogeneous - chained list, graph with cycle Believe it or not, valgrind is not too unhappy with the results. The cycle happily segfaults, but the others are ok. And I'm sick of pointers for now. [Transport] [File plugin] - Bugfix when using a filename explicitely (instead of '-') 2004-04-09 Martin Quinson [Transport plugins] - factorize more code between RL and SG in socket creation - Complete the implementation and tests of: o TCP o file (only in RL, and mainly for debugging) I lost 3 days to design a portable address resolver, and then decided that the prototype mainly have to run on my box. Addressing portability too early may be like optimizing too early :-/ [Tests] - use gras_init in the Tests instead of the crappy parse_log_opt (the latter function is removed) [Conditional execution] - New functions: gras_if_RL/gras_if_SG (basic support for this) [Code reorganisation] - Get rid of libgrasutils.a since it makes more trouble than it solves. Build examples against the RL library, since there is no way to disable its creation for now. For information, the beginning of coding on GRAS was back in june 2003. I guess that every line has been rewritten at least twice since then. SimGrid-3.14.159/tools/000755 001750 001750 00000000000 13031031310 016221 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/tools/doxygen/000755 001750 001750 00000000000 13031031276 017711 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/tools/doxygen/xbt_log_extract_hierarchy.pl000755 001750 001750 00000007446 13031031276 025512 0ustar00mquinsonmquinson000000 000000 #!/usr/bin/env perl # Copyright (c) 2008, 2010, 2012-2014. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. use strict; use warnings; my $debug = 0; print "/* Generated file, do not edit */\n"; print "/** \\addtogroup XBT_log_cats\n"; print " \@{\n"; # Search for calls to macros defining new channels, and prepare the tree representation my %ancestor; my %desc; # $ancestor{"toto"} is the ancestor of the toto channel as declared by XBT_LOG_NEW_SUBCATEGORY and # XBT_LOG_NEW_DEFAULT_SUBCATEGORY ie, when the channel toto is initialized (does not work under windows) # $desc{"toto"} is its description my %connected; # $connected{"toto"} is defined if XBT_LOG_CONNECT("toto") is used sub cleanup_ctn { my $ctn = shift; # cleanup the content of a macro call $ctn =~ s/^\s*(.*)\s*$/$1/gs; my @elms; print "ctn=$ctn\n" if $debug > 1; if ($ctn =~ m/^(\w+)\s*,\s*(\w+)\s*,\s*"?([^"]*)"?$/s) { # Perfect, we got 0->name; 1->anc; 2->desc $elms[0] = $1; $elms[1] = $2; $elms[2] = $3; } elsif ($ctn =~ m/^(\w+)\s*,\s*"?([^"]*)"?$/s) { # Mmm. got no ancestor. Add the default one. $elms[0] = $1; $elms[1] = "XBT_LOG_ROOT_CAT"; $elms[2] = $2; } else { die "Unparsable content: $ctn\n"; } $elms[2] =~ s/\\\\/\\/gs; return @elms; } sub parse_file { my $filename = shift; my $data = ""; print "Parse $filename\n" if $debug; open IN, "$filename" || die "Cannot read $filename: $!\n"; while () { $data .= $_; } close IN; # Purge $data from C comments $data =~ s|/\*.*?\*/||sg; # C++ comments are forbiden in SG for portability reasons, but deal with it anyway $data =~ s|//.*$||mg; my $connect_data = $data; # save a copy for second parsing phase while ($data =~ s/^.*?XBT_LOG_NEW(_DEFAULT)?_(SUB)?CATEGORY\(//s) { $data =~ s/([^"]*"[^"]*")\)//s || die "unparsable macro: $data"; my ($name,$anc,$desc) = cleanup_ctn($1); # build the tree, checking for name conflict die "ERROR: Category name conflict: $name used several times (in $ancestor{$name} and $anc, last time in $filename)\n" if defined ($ancestor{$name}) && $ancestor{$name} ne $anc && defined ($desc{$name}) && $desc{$name} ne $desc; $ancestor{$name}=$anc; $desc{$name}=$desc; print " $name -> $anc\n" if $debug; } # Now, look for XBT_LOG_CONNECT calls $data = $connect_data; while ($data =~ s/^.*?XBT_LOG_CONNECT\(//s) { $data =~ s/\s*(\w+)\s*\)//s || die "unparsable macro: $data"; $connected{$1} = 1; } } # Retrieve all the file names, and add their content to $data my $data; open FILES, "find ../src/ ../tools/ ../include/ -name '*.c' -o -name '*.cpp' |" || die "Cannot search for the source file names: $!\n"; while (my $file=) { chomp $file; parse_file($file); } parse_file("../include/xbt/sysdep.h"); close FILES; # Display the tree, looking for disconnected elems my %used; sub display_subtree { my $name=shift; my $indent=shift; $used{$name} = 1; unless ($name eq "XBT_LOG_ROOT_CAT") { # do not display the root print "$indent - $name: ".($desc{$name}|| "(undocumented)")."\n"; } foreach my $cat (grep {$ancestor{$_} eq $name} sort keys %ancestor) { display_subtree($cat,"$indent "); } } display_subtree("XBT_LOG_ROOT_CAT",""); map { if ($_ ne "mc_main") { # This one is not in libsimgrid warn "Category $_ does not seem to be connected. Use XBT_LOG_CONNECT($_).\n"; } } grep {!defined $connected{$_}} sort keys %ancestor; map { warn "Category $_ does not seem to be connected to the root (anc=$ancestor{$_})\n"; } grep {!defined $used{$_}} sort keys %ancestor; print "@}*/\n"; SimGrid-3.14.159/tools/doxygen/fig2dev_postprocessor.pl000755 001750 001750 00000001002 13031031276 024575 0ustar00mquinsonmquinson000000 000000 #! /usr/bin/env perl # Copyright (c) 2010, 2014. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. use strict; my $print = 0; my $line; while($line=) { chomp $line; if($line=~/IMG/) {$print=1;} if($print) {print $line."\n";} if($line=~/\/MAP/) {$print=0;} } #perl -pe 's/imagemap/simgrid_modules/g'| perl -pe 's/ header DEFEVENTS=`cat header | grep Define | awk '{ print $3 }'` GREP="" GREP2="" for i in $DEFEVENTS do GREP="/^$i /d; $GREP" GREP2="-e '^$i ' $GREP2" done GREP="/^%\ /d; /^% /d; /^%E/d; $GREP" cat $TRACE | eval grep $GREP2 > types /bin/sed -e "$GREP" $TRACE > events cat events | sort -n -k 2 -s > events.sorted cat header types events.sorted > $OUTPUT rm types events events.sorted header echo "output: $OUTPUT" SimGrid-3.14.159/tools/CMakeLists.txt000644 001750 001750 00000002033 13031031274 020770 0ustar00mquinsonmquinson000000 000000 set(bin_files ${bin_files} ${CMAKE_CURRENT_SOURCE_DIR}/fix-paje-trace.sh ${CMAKE_CURRENT_SOURCE_DIR}/generate-dwarf-functions ${CMAKE_CURRENT_SOURCE_DIR}/normalize-pointers.py ${CMAKE_CURRENT_SOURCE_DIR}/sg_unit_extractor.pl ${CMAKE_CURRENT_SOURCE_DIR}/sg_xml_unit_converter.py ${CMAKE_CURRENT_SOURCE_DIR}/simgrid_update_xml.pl ${CMAKE_CURRENT_SOURCE_DIR}/doxygen/fig2dev_postprocessor.pl ${CMAKE_CURRENT_SOURCE_DIR}/doxygen/xbt_log_extract_hierarchy.pl ${CMAKE_CURRENT_SOURCE_DIR}/MSG_visualization/colorize.pl ${CMAKE_CURRENT_SOURCE_DIR}/MSG_visualization/trace2fig.pl PARENT_SCOPE) set(txt_files ${txt_files} ${CMAKE_CURRENT_SOURCE_DIR}/simgrid.supp ${CMAKE_CURRENT_SOURCE_DIR}/simgrid2vite.sed PARENT_SCOPE) SimGrid-3.14.159/tools/smpi/000755 001750 001750 00000000000 13031031274 017202 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/tools/smpi/generate_smpi_defines.pl000755 001750 001750 00000004025 13031031274 024062 0ustar00mquinsonmquinson000000 000000 #!/usr/bin/env perl # Copyright 2016 Vincent Danjean # # Call this script like this: # C/C++ : ./generate_smpi_defines.pl ../../include/smpi/smpi.h # FORTRAN: ./generate_smpi_defines.pl -f ../../include/smpi/smpi.h # # It will generate macros that are useful for adding file and line numbers to # traces without obtaining a backtrace (as this would be very slow and make # the simulation/trace unreliable when compared to a "real" trace as obtained # with MPI+TAU). use strict; use warnings; use Getopt::Std; my %options=(); getopts("fc", \%options); # $incall denotes whether we are currently parsing a macro or not... my $incall=0; my $commentChar="//"; if (defined $options{f}) { $commentChar="!" } print "$commentChar This file has been automatically generated by the script\n"; print "$commentChar in tools/smpi/" . __FILE__ ."\n"; print "$commentChar DO NOT EDIT MANUALLY. ALL CHANGES WILL BE OVERWRITTEN!\n"; # Formatting of the output sub output_macro { my $line = shift; my @parts = split (/\s*,\s*/, $line); my $id = $parts[1]; # This is a GCC extension. The last statement is the value of the expression # in parentheses. if (defined $options{f}) { print "#define ". uc($id) ." smpi_trace_set_call_location(__FILE__,__LINE__); call ". lc $id ."\n"; } else { if ($id eq "MPI_Init") { print "#define MPI_Init(...) ({ smpi_process_init(__VA_ARGS__); smpi_trace_set_call_location(__FILE__,__LINE__); MPI_Init(NULL, NULL); })\n"; } else { print "#define $id(...) ({ smpi_trace_set_call_location(__FILE__,__LINE__); $id(__VA_ARGS__); })\n"; } } } my $wholemacro; my $line; while (defined($line = <>)) { chomp($line); if ($line =~ /^MPI_CALL/) { if ($incall) { output_macro($wholemacro); } $incall=1; $wholemacro = $line; } elsif ($incall && $line =~ /^\s+\S/) { # Did we already start parsing an MPI_CALL macro? If so, just concatenate $wholemacro .= ' '.$line; } elsif ($incall) { output_macro($wholemacro); $incall=0; } } SimGrid-3.14.159/tools/normalize-pointers.py000755 001750 001750 00000001545 13031031276 022457 0ustar00mquinsonmquinson000000 000000 #!/usr/bin/env python # Copyright (c) 2013-2014. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. """ Tool for normalizing pointers such as two runs have the same 'addresses' first address encountered will be replaced by 0X0000001, second by 0X0000002, ... """ import sys import re if len(sys.argv) != 2: print "Usage ./normalize-pointers.py " sys.exit(1) f = open(sys.argv[1]) t = f.read() f.close() r = re.compile(r"0x[0-9a-f]+") s = r.search(t) offset = 0 pointers = {} while (s): if s.group() not in pointers: pointers[s.group()] = "0X%07d" % len(pointers) print t[offset:s.start()], print pointers[s.group()], offset = s.end() s = r.search(t, offset) print t[offset:] SimGrid-3.14.159/tools/simgrid.supp000644 001750 001750 00000006423 13031031310 020575 0ustar00mquinsonmquinson000000 000000 # Valgrind suppressions for stuff that we cannot control # Memory leaks in standard tools (e.g. dash, tail, or sort) { Memory leak in /bin tools Memcheck:Leak ... obj:/bin/* } { Memory leak in /usr/bin tools Memcheck:Leak ... obj:/usr/bin/* } # There's a constant leak of 56 bytes in the depths of libc which # manifests, for example, when using backtrace() { Memory leak in libc/dlopen with -pthread Memcheck:Leak fun:malloc fun:_dl_map_object_deps fun:dl_open_worker fun:_dl_catch_error fun:_dl_open fun:do_dlopen fun:_dl_catch_error fun:dlerror_run fun:__libc_dlopen_mode } # Another problem in glibc, where makecontext does not reset the EBP register, # and backtrace goes too far when walking up the stack frames { Invalid read in backtrace, called after makecontext Memcheck:Addr4 fun:backtrace ... fun:makecontext } #There seems to be an issue with libc using an uninitialized value somewhere in dlopen { Invalid read in dl_start Memcheck:Cond fun:index fun:expand_dynamic_string_token ... fun:_dl_start } # 72704 bytes leak from GCC >5.1 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=64535 { Memory leak in dl_init Memcheck:Leak match-leak-kinds:reachable fun:malloc obj:/usr/lib/*/libstdc++.so.* fun:call_init.part.0 ... fun:_dl_init } # SG3.13 : We will remove NetCards soon, so ignore these ones { Memory leak in Netcards Memcheck:Leak ... fun:sg_platf_new_router fun:STag_surfxml_router fun:surf_parse_lex ... fun:parse_platform_file fun:SIMIX_create_environment } { Memory leak in Netcards Memcheck:Leak match-leak-kinds: indirect fun:malloc fun:strdup fun:xbt_strdup ... fun:sg_platf_new_router fun:STag_surfxml_router fun:surf_parse_lex ... fun:parse_platform_file fun:SIMIX_create_environment fun:smpi_main fun:main } #Ignore leaks in SMPI sample codes { Leaks in SMPI sample codes Memcheck:Leak match-leak-kinds: all fun:malloc fun:smpi_simulated_main_ } # Memory leaks appearing to be in libcgraph. They can be seen with the # following simple program: # ,---- # | #include # | #include # | int main(int argc, char *argv[]) # | { # | if (argc == 1) { # | printf("Usage: %s \n", argv[0]); # | return 1; # | } # | Agraph_t *g; # | FILE *inf = fopen(argv[1], "r"); # | g = agread(inf, 0); # | fclose(inf); # | agclose(g); # | return 0; # | } # `---- { Memory leak in libcgraph (1/2) Memcheck:Leak fun:malloc ... obj:/usr/lib/libcgraph.so* fun:aaglex fun:aagparse fun:agconcat } { Memory leak in libcgraph (1/2) Memcheck:Leak fun:calloc ... obj:/usr/lib/libcgraph.so* fun:aagparse fun:agconcat } { Memory leak in libcgraph (2/2) Memcheck:Leak fun:malloc ... fun:agnode obj:/usr/lib/libcgraph.so* fun:aagparse fun:agconcat } # We're not interested by memory leaks in the Lua interpreter { Memory leak in lua Memcheck:Leak ... fun:luaD_precall } # libunwind seems to be using msync poorly, thus triggering these # https://github.com/JuliaLang/julia/issues/4533 { msync unwind Memcheck:Param msync(start) ... obj:*/libpthread*.so ... } SimGrid-3.14.159/tools/stack-cleaner/000755 001750 001750 00000000000 13031031274 020746 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/tools/stack-cleaner/cc000755 001750 001750 00000000105 13031031274 021255 0ustar00mquinsonmquinson000000 000000 #!/bin/sh path="$(dirname $0)" exec "$path"/compiler-wrapper cc "$@" SimGrid-3.14.159/tools/stack-cleaner/compiler-wrapper000755 001750 001750 00000001653 13031031274 024171 0ustar00mquinsonmquinson000000 000000 #!/usr/bin/env python # Compiler wrapper with stack-cleaner support (enabled by default). # Usage: ./compiler-wrapper target-compiler args [-f[no-]stack-cleaner] import sys import os import re compiler = sys.argv[1] enabled = True args = [] for arg in sys.argv[2:]: if arg == "--help": sys.stderr.write( "Compiler wrapper with stack-cleaner.\n" "Usage: {} compiler [-f[no-]stack-cleaner] [options]\n".format(sys.argv[0])) sys.exit(0) elif arg == "-fno-stack-cleaner": enabled = False elif arg == "-fstack-cleaner": enabled = True else: args.append(arg) if enabled: if re.match("^clang", os.path.basename(compiler)): args.insert(0, "-no-integrated-as") args.insert(0, os.path.dirname(sys.argv[0])) args.insert(0, "-B") args.insert(0, compiler) os.execvp(args[0], args) sys.stderr.write("compiler-wrapper: Could not exec\n") os.exit(1) SimGrid-3.14.159/tools/stack-cleaner/fortran000755 001750 001750 00000000113 13031031274 022342 0ustar00mquinsonmquinson000000 000000 #!/bin/sh path="$(dirname $0)" exec "$path"/compiler-wrapper gfortran "$@" SimGrid-3.14.159/tools/stack-cleaner/clean-stack-filter000755 001750 001750 00000003274 13031031274 024352 0ustar00mquinsonmquinson000000 000000 #!/usr/bin/env perl # Transform assembly in order to clean each stack frame for X86_64. use strict; use warnings; $SIG{__WARN__} = sub { die @_ }; # Whether we are still scanning the content of a function: our $scanproc = 0; # Save lines of the function: our $lines = ""; # Size of the stack for this function: our $size = 0; # Counter for assigning unique ids to labels: our $id=0; sub emit_code { my $qsize = $size / 8; my $offset = - $size - 8; if($size != 0) { # This is a crude hack to disable the stack cleaning on the main # stack. It relies on the fact that the main stack is high in # the address space and the other stacks are in the heap (lower). print("\tmovq \$0x7fff00000000, %r11\n"); print("\tcmpq %r11, %rsp\n"); print("\tjae .Lstack_cleaner_done$id\n"); # Loop over the stack frame quadwords and zero it: print("\tmovabsq \$$qsize, %r11\n"); print(".Lstack_cleaner_loop$id:\n"); print("\tmovq \$0, $offset(%rsp,%r11,8)\n"); print("\tsubq \$1, %r11\n"); print("\tjne .Lstack_cleaner_loop$id\n"); print(".Lstack_cleaner_done$id:\n"); } print $lines; $id = $id + 1; $size = 0; $lines = ""; $scanproc = 0; } while (<>) { if ($scanproc) { $lines = $lines . $_; if (m/^[ \t]*.cfi_endproc$/) { emit_code(); } elsif (m/^[ \t]*pushq/) { $size += 8; } elsif (m/^[ \t]*subq[\t *]\$([0-9]*),[ \t]*%rsp$/) { my $val = $1; $val = oct($val) if $val =~ /^0/; $size += $val; emit_code(); } } elsif (m/^[ \t]*.cfi_startproc$/) { print $_; $scanproc = 1; } else { print $_; } } SimGrid-3.14.159/tools/stack-cleaner/README000644 001750 001750 00000001204 13031031274 021623 0ustar00mquinsonmquinson000000 000000 Provides stack-cleaning compilers for x86_64: * as * cc * c++ * fortran Each of them modify the generated/given X86_64 assembly by prepending stack-cleanup code to each function: movq $0x7ffff7ff8000, %r11 cmpq %r11, %rsp jbe .Lstack_cleaner_done0 movq $QSIZE, %r11 .Lstack_cleaner_loop0: movq $0, OFFSET(%rsp,%r11,8) subq $1, %r11 jne .Lstack_cleaner_loop0: .Lstack_cleaner_done0: The modification of the assembly is done by the clean-stack-filter program. If the underlying compiler is clang, it might be necessary to add the -no-integrated-as flag in order to force the usage of an external assembler. SimGrid-3.14.159/tools/stack-cleaner/c++000755 001750 001750 00000000106 13031031274 021241 0ustar00mquinsonmquinson000000 000000 #!/bin/sh path="$(dirname $0)" exec "$path"/compiler-wrapper c++ "$@" SimGrid-3.14.159/tools/stack-cleaner/as000755 001750 001750 00000003070 13031031274 021277 0ustar00mquinsonmquinson000000 000000 #!/usr/bin/env python # Wrapper around the real `as` which adds filtering capabilities. import os import re import subprocess import sys import tempfile # Process the arguments: # * build the argument array for calling the real assembler; # * get the input file name. args = [] input_filename = None args.append('as') i = 1 while i < len(sys.argv): if sys.argv[i] == '-o' or sys.argv[i] == '-I': args.append(sys.argv[i]) if i + 1 >= len(sys.argv): sys.stderr.write("Missing argument\n") sys.exit(1) args.append(sys.argv[i + 1]) i = i + 1 elif re.match('^-', sys.argv[i][0]): args.append(sys.argv[i]) elif input_filename: sys.stdout.write("Too many input files\n") sys.exit(1) else: input_filename = sys.argv[i] i = i + 1 if input_filename == None: sys.stderr.write("Missing input file\n") sys.exit(1) temp_file, temp_filename = tempfile.mkstemp(suffix=".s", prefix="as_wrapper") try: # Generate temporary file with modified assembly code: script_file = os.path.join( os.path.dirname(sys.argv[0]), "clean-stack-filter") input_file = os.open(input_filename, os.O_RDONLY) status = subprocess.call([script_file], stdin=input_file, stdout=temp_file) os.close(input_file) if status != 0: sys.stderr.write("Filtering the assembly code failed.\n") sys.exit(status) # Call the real assembler on this modified assembly code: args.append(temp_filename) sys.exit(subprocess.call(args)) finally: os.remove(temp_filename) SimGrid-3.14.159/tools/sg_unit_extractor.pl000755 001750 001750 00000023430 13031031276 022341 0ustar00mquinsonmquinson000000 000000 #! /usr/bin/env perl # Copyright (c) 2005-2012, 2014. The SimGrid Team. All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. use strict; use Getopt::Long qw(GetOptions); my $progname="sg_unit_extractor"; # Get the args sub usage($) { my $ret; print "USAGE: $progname [--root=part/to/cut] [--outdir=where/to/generate/files] infile [infile+]\n\n"; print "This program is in charge of extracting the unit tests out of the SimGrid source code.\n"; print "See http://simgrid.gforge.inria.fr/doc/latest/inside_tests.html for more details.\n"; exit $ret; } my $outdir=undef; my $root; my $help; Getopt::Long::config('permute','no_getopt_compat', 'no_auto_abbrev'); GetOptions( 'help|h' => sub {usage(0)}, 'root=s' =>\$root, 'outdir=s' =>\$outdir) or usage(1); usage(1) if (scalar @ARGV == 0); map {process_one($_)} @ARGV; sub process_one($) { my $infile = shift; my $outfile; $infile =~ s|src/|| unless (-e $infile); $outfile = $infile; $outfile =~ s/\.c$/_unit.c/; $outfile =~ s/\.cpp$/_unit.cpp/; $outfile =~ s|.*/([^/]*)$|$1| if $outfile =~ m|/|; $outfile = "$outdir$outfile"; print "$progname: processing $infile (generating $outfile)...\n"; # Get the unit data my ($unit_source,$suite_name,$suite_title)=("","",""); my (%tests); # to detect multiple definition my (@tests); # actual content open IN, "$infile" || die "$progname: Cannot open input file '$infile': $!\n"; $infile =~ s|$root|| if defined($root); my $takeit=0; my $line=0; my $beginline=0; while () { $line++; if (m/ifdef +SIMGRID_TEST/) { $beginline = $line; $takeit = 1; next; } if (m/endif.*SIMGRID_TEST/) { $takeit = 0; next } if (m/XBT_TEST_SUITE\(\w*"([^"]*)"\w*, *(.*?)\);/) { #" { die "$progname: Multiple suites in the same file ($infile) are not supported yet\n" if length($suite_name); ($suite_name,$suite_title)=($1,$2); die "$progname: Empty suite name in $infile" unless length($suite_name); die "$progname: Empty suite title in $infile" unless length($suite_title); next; } elsif (m/XBT_TEST_SUITE/) { die "$progname: Parse error: This line seem to be a test suite declaration, but failed to parse it\n$_\n"; } if (m/XBT_TEST_UNIT\(\w*"([^"]*)"\w*,([^,]*),(.*?)\)/) { #"{ die "$progname: multiply defined unit in file $infile: $1\n" if (defined($tests{$1})); my @t=($1,$2,$3); push @tests,\@t; $tests{$1} = 1; } elsif (m/XBT_TEST_UNIT/) { die "$progname: Parse error: This line seem to be a test unit, but failed to parse it\n$_\n"; } $unit_source .= $_ if $takeit; } close IN || die "$progname: cannot close input file '$infile': $!\n"; if ($takeit) { die "$progname: end of file reached in SIMGRID_TEST block.\n". "You should end each of the with a line matching: /endif.*SIMGRID_TEST/\n". "Example:\n". "#endif /* SIMGRID_TEST */\n" } die "$progname: no suite defined in $infile\n" unless (length($suite_name)); # Write the test my ($GENERATED)=("/*******************************/\n". "/* GENERATED FILE, DO NOT EDIT */\n". "/*******************************/\n\n"); $beginline+=2; open OUT,">$outfile" || die "$progname: Cannot open output file '$outfile': $!\n"; print OUT $GENERATED; print OUT "#include \n"; print OUT "#include \"xbt.h\"\n"; print OUT $GENERATED; print OUT "#line $beginline \"$infile\" \n"; print OUT "$unit_source"; print OUT $GENERATED; close OUT || die "$progname: Cannot close output file '$outfile': $!\n"; # write the main skeleton if needed if (! -e "${outdir}simgrid_units_main.c") { open OUT,">${outdir}simgrid_units_main.c" || die "$progname: Cannot open main file '${outdir}simgrid_units_main.c': $!\n"; print OUT $GENERATED; print OUT "#include \n\n"; print OUT "#include \"xbt.h\"\n\n"; print OUT "extern xbt_test_unit_t _xbt_current_unit;\n\n"; print OUT "/* SGU: BEGIN PROTOTYPES */\n"; print OUT "/* SGU: END PROTOTYPES */\n\n"; print OUT $GENERATED; # print OUT "# 93 \"sg_unit_extractor.pl\"\n"; print OUT <) { $newmain .= $_; # print "Look for proto: $_"; last if /SGU: BEGIN PROTOTYPES/; } # search my prototype while () { # print "Seek protos: $_"; last if (/SGU: END PROTOTYPES/ || /SGU: BEGIN FILE $infile/); $newmain .= $_; } if (/SGU: BEGIN FILE $infile/) { # found an old section for this file. Kill it while () { last if /SGU: END FILE/; } $_ = ; # pass extra blank line chomp; die "this line should be blank ($_). Did you edit the file?" if /\W/; } my ($old_)=($_); # add my section $newmain .= " /* SGU: BEGIN FILE $infile */\n"; map { my ($name,$func,$title) = @{$_}; $newmain .= " void $func(void);\n" } @tests; $newmain .= " /* SGU: END FILE */\n\n"; if ($old_ =~ /SGU: BEGIN FILE/ || $old_ =~ /SGU: END PROTOTYPES/) { $newmain .= $old_; } # pass remaining prototypes, search declarations while () { $newmain .= $_ unless /SGU: END PROTOTYPES/; last if /SGU: BEGIN SUITES DECLARATION/; } ### Done with prototypes. And now, the actual code # search my prototype while () { last if (/SGU: END SUITES DECLARATION/ || /SGU: BEGIN FILE $infile/); $newmain .= $_; } if (/SGU: BEGIN FILE $infile/) { # found an old section for this file. Kill it while () { last if /SGU: END FILE/; } $_ = ; # pass extra blank line chomp; die "this line should be blank ($_). Did you edit the file?" if /\W/; } my ($old_)=($_); # add my section $newmain .= " /* SGU: BEGIN FILE $infile */\n"; $newmain .= " suite = xbt_test_suite_by_name(\"$suite_name\",$suite_title);\n"; map { my ($name,$func,$title) = @{$_}; $newmain .= " xbt_test_suite_push(suite, \"$name\", $func, $title);\n"; } @tests; $newmain .= " /* SGU: END FILE */\n\n"; if ($old_ =~ /SGU: BEGIN FILE/ || $old_ =~ /SGU: END SUITES DECLARATION/) { $newmain .= $old_; } # pass the remaining while () { $newmain .= $_; } close IN || die "$progname: Cannot close main file '${outdir}simgrid_units_main.c': $!\n"; # write it back to main open OUT,">${outdir}simgrid_units_main.c" || die "$progname: Cannot open main file '${outdir}simgrid_units_main.c': $!\n"; print OUT $newmain; close OUT || die "$progname: Cannot close main file '${outdir}simgrid_units_main.c': $!\n"; } # end if process_one($) 0; SimGrid-3.14.159/tools/sg_xml_unit_converter.py000644 001750 001750 00000006075 13031031276 023235 0ustar00mquinsonmquinson000000 000000 #!/usr/bin/env python # -*- coding: utf-8 -*- # Copyright (c) 2013-2014. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. # grep -ohrI 'bw=".*"' . | sort | uniq import sys import fnmatch import os from decimal import Decimal import re def to_str(dec): return re.sub(r"(\.\d*?)0*$", r"\1", dec.to_eng_string()).rstrip(".") def format(xml, formats, attrib): res = [] m = re.search(r'%s="(.*?)"' % attrib, xml) while m: b, e = m.span(1) res.append(xml[:b]) val = xml[b:e] xml = xml[e:] try: power = Decimal(val) tmp = to_str(power) for p, f in formats: d = power / p if d >= 1.0: tmp = "%s%s" % (to_str(d), f) break res.append(tmp) except: print "Error with:", val res.append(val) m = re.search(r'%s="(.*?)"' % attrib, xml) res.append(xml) return "".join(res) def formats(list): return sorted(((Decimal(i), j) for i, j in list), key=lambda x: x[0], reverse=True) for root, dirnames, filenames in os.walk(sys.argv[1]): for filename in fnmatch.filter(filenames, '*.xml'): print root, dirnames, filename path = os.path.join(root, filename) xml = open(path).read() power_formats = formats([("1E0", "f"), ("1E3", "kf"), ("1E6", "Mf"), ("1E9", "Gf"), ("1E12", "Tf"), ("1E15", "Pt"), ("1E18", "Ef"), ("1E21", "Zf")]) xml = format(xml, power_formats, "power") bandwidth_formats = formats([("1E0", "Bps"), ("1E3", "kBps"), ("1E6", "MBps"), ("1E9", "GBps"), ("1E12", "TBps")]) xml = format(xml, bandwidth_formats, "bandwidth") xml = format(xml, bandwidth_formats, "bw") xml = format(xml, bandwidth_formats, "bb_bw") xml = format(xml, bandwidth_formats, "bw_in") xml = format(xml, bandwidth_formats, "bw_out") time_formats = formats([("1E-12", "ps"), ("1E-9", "ns"), ("1E-6", "us"), ("1E-3", "ms"), ("1E0", "s"), ("60E0", "m"), ("3600E0", "h"), ("86400E0", "d"), ("604800E0", "w")]) xml = format(xml, time_formats, "latency") xml = format(xml, time_formats, "lat") xml = format(xml, time_formats, "bb_lat") # print xml file = open(path, "w") file.write(xml) file.close() SimGrid-3.14.159/tools/tesh/000755 001750 001750 00000000000 13031031305 017170 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/tools/tesh/IO-orders.tesh000644 001750 001750 00000001573 13031031305 021666 0ustar00mquinsonmquinson000000 000000 p This tests that TESH accepts any order for the input/output p Order: in, out, cmd < < TOTO < > TOTO < $ cat > Test suite from stdin > [(stdin):3] cat > Test suite from stdin OK $ ${bindir:=.}/tesh p Order: out, in, cmd < > TOTO < < TOTO < $ cat > Test suite from stdin > [(stdin):3] cat > Test suite from stdin OK $ ${bindir:=.}/tesh p Order: out, cmd, in < > TOTO < $ cat < < TOTO > Test suite from stdin > [(stdin):2] cat > Test suite from stdin OK $ ${bindir:=.}/tesh p Order: in, cmd, out < < TOTO < $ cat < > TOTO > Test suite from stdin > [(stdin):2] cat > Test suite from stdin OK $ ${bindir:=.}/tesh p Order: cmd, out, in < $ cat < > TOTO < < TOTO > Test suite from stdin > [(stdin):1] cat > Test suite from stdin OK $ ${bindir:=.}/tesh p Order: cmd, in, out < $ cat < < TOTO < > TOTO > Test suite from stdin > [(stdin):1] cat > Test suite from stdin OK $ ${bindir:=.}/tesh SimGrid-3.14.159/tools/tesh/catch-wrong-output.tesh000644 001750 001750 00000000473 13031031305 023633 0ustar00mquinsonmquinson000000 000000 #! ./tesh p This tests whether TESH detects wrong outputs ! expect return 2 < > TOTO < < TUTU < $ cat $ ${bindir:=.}/tesh > Test suite from stdin > [(stdin):3] cat > Output of <(stdin):3> mismatch: >--- expected >+++ obtained >@@ -1 +1 @@ > -TOTO > +TUTU > Test suite `(stdin)': NOK (<(stdin):3> output mismatch) SimGrid-3.14.159/tools/tesh/CMakeLists.txt000644 001750 001750 00000001516 13031031274 021740 0ustar00mquinsonmquinson000000 000000 configure_file("${CMAKE_HOME_DIRECTORY}/tools/tesh/tesh.py" "${CMAKE_BINARY_DIR}/bin/tesh" @ONLY IMMEDIATE) foreach(x setenv set-output-ignore set-output-sort set-return set-timeout background basic bg-basic bg-set-signal catch-return catch-signal catch-timeout catch-wrong-output cd color ignore-regexp IO-bigsize IO-broken-pipe IO-orders) set(tesh_files ${tesh_files} ${CMAKE_CURRENT_SOURCE_DIR}/${x}.tesh) if(NOT enable_memcheck AND NOT WIN32) ADD_TESH(tesh-self-${x} --setenv bindir=${CMAKE_BINARY_DIR}/bin ${CMAKE_HOME_DIRECTORY}/tools/tesh/${x}.tesh) endif() endforeach() set(tesh_files ${tesh_files} PARENT_SCOPE) set(bin_files ${bin_files} ${CMAKE_CURRENT_SOURCE_DIR}/tesh.py ${CMAKE_CURRENT_SOURCE_DIR}/generate_tesh PARENT_SCOPE) SimGrid-3.14.159/tools/tesh/IO-broken-pipe.tesh000644 001750 001750 00000004333 13031031305 022600 0ustar00mquinsonmquinson000000 000000 # TESH autotest: check that the father is protected from broken pipes # # If not, it breaks sometimes (when the child is scheduled before the # father), so we do the test a bunch of times. # < blablablablablablablablablablablablablablablablablablablabla 01 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 02 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 03 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 04 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 05 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 06 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 07 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 08 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 09 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 10 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 11 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 12 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 13 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 14 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 15 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 16 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 17 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 18 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 19 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 20 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 21 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 22 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 23 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 24 $ cmake -E echo < blablablablablablablablablablablablablablablablablablablabla 25 $ cmake -E echo SimGrid-3.14.159/tools/tesh/ignore-regexp.tesh000644 001750 001750 00000000152 13031031305 022626 0ustar00mquinsonmquinson000000 000000 ! ignore .*0x[0-9A-F]+\. $ printf 'word\nMemory address: 0x42AA42.\nanotherword\n' > word > anotherword SimGrid-3.14.159/tools/tesh/IO-bigsize.tesh000644 001750 001750 00001573236 13031031305 022037 0ustar00mquinsonmquinson000000 000000 #! ./tesh # This suite contains two tests: # The first one uses a very big input (150k) to check whether trucated input do work. # The second one uses both a big input and a big output (150k each). # # This checks whether the non-blocking I/O mess is functionnal. # p First, a write test < 000 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 001 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 002 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 003 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 004 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 005 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 006 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 007 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 008 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 009 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 010 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 011 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 012 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 013 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 014 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 015 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 016 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 017 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 018 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 019 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 020 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 021 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 022 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 023 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 024 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 025 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 026 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 027 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 028 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 029 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 030 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 031 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 032 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 033 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 034 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 035 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 036 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 037 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 038 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 039 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 040 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 041 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 042 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 043 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 044 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 045 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 046 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 047 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 048 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 049 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 010 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 051 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 052 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 053 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 054 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 055 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 056 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 057 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 058 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 059 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 060 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 061 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 062 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 063 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 064 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 065 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 066 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 067 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 068 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 069 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 070 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 071 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 072 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 073 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 074 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 075 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 076 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 077 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 078 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 079 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 080 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 081 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 082 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 083 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 084 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 085 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 086 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 087 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 088 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 089 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 090 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 091 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 092 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 093 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 094 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 095 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 096 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 097 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 098 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 099 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 100 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 101 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 102 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 103 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 104 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 105 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 106 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 107 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 108 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 109 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 110 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 111 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 112 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 113 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 114 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 115 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 116 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 117 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 118 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 119 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 120 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 121 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 122 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 123 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 124 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 125 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 126 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 127 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 128 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 129 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 130 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 131 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 132 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 133 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 134 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 135 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 136 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 137 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 138 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 139 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 140 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 141 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 142 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 143 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 144 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 145 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 146 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 147 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 148 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 149 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 110 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 151 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 152 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 153 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 154 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 155 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 156 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 157 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 158 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 159 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 160 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 161 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 162 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 163 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 164 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 165 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 166 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 167 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 168 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 169 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 170 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 171 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 172 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 173 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 174 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 175 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 176 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 177 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 178 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 179 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 180 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 181 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 182 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 183 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 184 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 185 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 186 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 187 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 188 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 189 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 190 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 191 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 192 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 193 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 194 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 195 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 196 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 197 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 198 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 199 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 200 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 201 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 202 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 203 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 204 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 205 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 206 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 207 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 208 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 209 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 210 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 211 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 212 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 213 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 214 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 215 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 216 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 217 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 218 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 219 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 220 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 221 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 222 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 223 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 224 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 225 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 226 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 227 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 228 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 229 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 230 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 231 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 232 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 233 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 234 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 235 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 236 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 237 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 238 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 239 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 240 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 241 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 242 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 243 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 244 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 245 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 246 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 247 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 248 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 249 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 210 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 251 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 252 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 253 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 254 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 255 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 256 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 257 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 258 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 259 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 260 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 261 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 262 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 263 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 264 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 265 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 266 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 267 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 268 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 269 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 270 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 271 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 272 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 273 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 274 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 275 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 276 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 277 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 278 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 279 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 280 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 281 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 282 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 283 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 284 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 285 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 286 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 287 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 288 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 289 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 290 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 291 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 292 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 293 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 294 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 295 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 296 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 297 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 298 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 399 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 300 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 301 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 302 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 303 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 304 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 305 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 306 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 307 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 308 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 309 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 310 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 311 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 312 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 313 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 314 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 315 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 316 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 317 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 318 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 319 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 320 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 321 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 322 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 323 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 324 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 325 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 326 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 327 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 328 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 329 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 330 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 331 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 332 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 333 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 334 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 335 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 336 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 337 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 338 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 339 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 340 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 341 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 342 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 343 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 344 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 345 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 346 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 347 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 348 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 349 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 310 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 351 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 352 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 353 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 354 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 355 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 356 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 357 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 358 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 359 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 360 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 361 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 362 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 363 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 364 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 365 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 366 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 367 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 368 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 369 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 370 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 371 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 372 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 373 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 374 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 375 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 376 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 377 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 378 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 379 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 380 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 381 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 382 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 383 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 384 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 385 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 386 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 387 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 388 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 389 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 390 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 391 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 392 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 393 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 394 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 395 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 396 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 397 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 398 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 399 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 400 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 401 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 402 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 403 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 404 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 405 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 406 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 407 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 408 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 409 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 410 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 411 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 412 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 413 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 414 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 415 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 416 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 417 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 418 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 419 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 420 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 421 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 422 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 423 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 424 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 425 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 426 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 427 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 428 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 429 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 430 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 431 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 432 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 433 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 434 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 435 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 436 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 437 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 438 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 439 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 440 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 441 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 442 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 443 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 444 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 445 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 446 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 447 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 448 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 449 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 410 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 451 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 452 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 453 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 454 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 455 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 456 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 457 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 458 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 459 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 460 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 461 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 462 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 463 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 464 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 465 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 466 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 467 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 468 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 469 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 470 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 471 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 472 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 473 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 474 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 475 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 476 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 477 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 478 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 479 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 480 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 481 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 482 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 483 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 484 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 485 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 486 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 487 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 488 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 489 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 490 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 491 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 492 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 493 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 494 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 495 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 496 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 497 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 498 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 499 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 500 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 501 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 502 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 503 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 504 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 505 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 506 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 507 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 508 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 509 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 510 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 511 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 512 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 513 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 514 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 515 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 516 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 517 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 518 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 519 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 520 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 521 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 522 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 523 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 524 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 525 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 526 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 527 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 528 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 529 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 530 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 531 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 532 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 533 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 534 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 535 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 536 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 537 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 538 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 539 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 540 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 541 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 542 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 543 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 544 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 545 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 546 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 547 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 548 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 549 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 510 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 551 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 552 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 553 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 554 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 555 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 556 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 557 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 558 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 559 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 560 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 561 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 562 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 563 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 564 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 565 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 566 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 567 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 568 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 569 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 570 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 571 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 572 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 573 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 574 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 575 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 576 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 577 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 578 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 579 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 580 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 581 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 582 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 583 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 584 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 585 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 586 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 587 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 588 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 589 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 590 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 591 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 592 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 593 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 594 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 595 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 596 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 597 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 598 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 599 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 600 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 601 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 602 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 603 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 604 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 605 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 606 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 607 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 608 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 609 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 610 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 611 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 612 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 613 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 614 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 615 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 616 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 617 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 618 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 619 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 620 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 621 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 622 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 623 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 624 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 625 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 626 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 627 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 628 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 629 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 630 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 631 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 632 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 633 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 634 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 635 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 636 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 637 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 638 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 639 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 640 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 641 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 642 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 643 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 644 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 645 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 646 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 647 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 648 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 649 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 610 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 651 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 652 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 653 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 654 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 655 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 656 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 657 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 658 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 659 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 660 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 661 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 662 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 663 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 664 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 665 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 666 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 667 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 668 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 669 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 670 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 671 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 672 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 673 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 674 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 675 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 676 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 677 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 678 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 679 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 680 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 681 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 682 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 683 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 684 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 685 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 686 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 687 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 688 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 689 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 690 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 691 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 692 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 693 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 694 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 695 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 696 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 697 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 698 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 699 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 700 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 701 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 702 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 703 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 704 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 705 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 706 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 707 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 708 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 709 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 710 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 711 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 712 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 713 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 714 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 715 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 716 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 717 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 718 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 719 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 720 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 721 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 722 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 723 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 724 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 725 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 726 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 727 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 728 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 729 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 730 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 731 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 732 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 733 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 734 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 735 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 736 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 737 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 738 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 739 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 740 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 741 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 742 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 743 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 744 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 745 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 746 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 747 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 748 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 749 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 710 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 751 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 752 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 753 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 754 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 755 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 756 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 757 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 758 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 759 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 760 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 761 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 762 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 763 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 764 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 765 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 766 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 767 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 768 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 769 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 770 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 771 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 772 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 773 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 774 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 775 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 776 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 777 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 778 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 779 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 780 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 781 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 782 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 783 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 784 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 785 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 786 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 787 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 788 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 789 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 790 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 791 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 792 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 793 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 794 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 795 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 796 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 797 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 798 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 799 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 800 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 801 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 802 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 803 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 804 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 805 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 806 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 807 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 808 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 809 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 810 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 811 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 812 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 813 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 814 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 815 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 816 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 817 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 818 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 819 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 820 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 821 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 822 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 823 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 824 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 825 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 826 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 827 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 828 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 829 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 830 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 831 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 832 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 833 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 834 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 835 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 836 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 837 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 838 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 839 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 840 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 841 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 842 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 843 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 844 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 845 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 846 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 847 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 848 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 849 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 810 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 851 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 852 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 853 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 854 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 855 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 856 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 857 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 858 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 859 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 860 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 861 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 862 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 863 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 864 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 865 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 866 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 867 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 868 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 869 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 870 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 871 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 872 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 873 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 874 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 875 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 876 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 877 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 878 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 879 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 880 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 881 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 882 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 883 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 884 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 885 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 886 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 887 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 888 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 889 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 890 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 891 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 892 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 893 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 894 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 895 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 896 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 897 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 898 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 899 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 900 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 901 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 902 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 903 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 904 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 905 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 906 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 907 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 908 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 909 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 910 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 911 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 912 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 913 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 914 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 915 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 916 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 917 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 918 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 919 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 920 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 921 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 922 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 923 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 924 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 925 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 926 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 927 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 928 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 929 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 930 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 931 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 932 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 933 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 934 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 935 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 936 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 937 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 938 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 939 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 940 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 941 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 942 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 943 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 944 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 945 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 946 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 947 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 948 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 949 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 910 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 951 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 952 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 953 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 954 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 955 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 956 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 957 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 958 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 959 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 960 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 961 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 962 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 963 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 964 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 965 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 966 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 967 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 968 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 969 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 970 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 971 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 972 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 973 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 974 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 975 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 976 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 977 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 978 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 979 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 980 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 981 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 982 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 983 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 984 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 985 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 986 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 987 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 988 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 989 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 990 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 991 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 992 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 993 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 994 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 995 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 996 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 997 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 998 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 999 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA $ sed -n '$=' > 1000 p And now, a read/write test < 000 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 001 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 002 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 003 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 004 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 005 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 006 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 007 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 008 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 009 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 010 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 011 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 012 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 013 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 014 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 015 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 016 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 017 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 018 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 019 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 020 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 021 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 022 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 023 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 024 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 025 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 026 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 027 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 028 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 029 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 030 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 031 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 032 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 033 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 034 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 035 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 036 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 037 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 038 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 039 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 040 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 041 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 042 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 043 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 044 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 045 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 046 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 047 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 048 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 049 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 010 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 051 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 052 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 053 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 054 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 055 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 056 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 057 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 058 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 059 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 060 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 061 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 062 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 063 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 064 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 065 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 066 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 067 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 068 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 069 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 070 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 071 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 072 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 073 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 074 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 075 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 076 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 077 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 078 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 079 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 080 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 081 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 082 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 083 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 084 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 085 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 086 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 087 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 088 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 089 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 090 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 091 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 092 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 093 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 094 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 095 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 096 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 097 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 098 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 099 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 100 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 101 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 102 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 103 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 104 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 105 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 106 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 107 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 108 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 109 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 110 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 111 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 112 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 113 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 114 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 115 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 116 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 117 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 118 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 119 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 120 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 121 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 122 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 123 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 124 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 125 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 126 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 127 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 128 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 129 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 130 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 131 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 132 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 133 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 134 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 135 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 136 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 137 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 138 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 139 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 140 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 141 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 142 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 143 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 144 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 145 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 146 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 147 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 148 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 149 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 110 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 151 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 152 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 153 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 154 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 155 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 156 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 157 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 158 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 159 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 160 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 161 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 162 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 163 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 164 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 165 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 166 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 167 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 168 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 169 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 170 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 171 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 172 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 173 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 174 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 175 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 176 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 177 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 178 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 179 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 180 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 181 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 182 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 183 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 184 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 185 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 186 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 187 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 188 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 189 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 190 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 191 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 192 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 193 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 194 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 195 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 196 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 197 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 198 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 199 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 200 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 201 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 202 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 203 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 204 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 205 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 206 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 207 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 208 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 209 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 210 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 211 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 212 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 213 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 214 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 215 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 216 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 217 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 218 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 219 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 220 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 221 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 222 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 223 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 224 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 225 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 226 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 227 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 228 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 229 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 230 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 231 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 232 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 233 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 234 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 235 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 236 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 237 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 238 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 239 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 240 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 241 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 242 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 243 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 244 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 245 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 246 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 247 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 248 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 249 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 210 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 251 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 252 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 253 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 254 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 255 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 256 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 257 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 258 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 259 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 260 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 261 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 262 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 263 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 264 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 265 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 266 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 267 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 268 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 269 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 270 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 271 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 272 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 273 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 274 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 275 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 276 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 277 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 278 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 279 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 280 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 281 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 282 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 283 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 284 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 285 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 286 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 287 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 288 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 289 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 290 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 291 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 292 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 293 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 294 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 295 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 296 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 297 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 298 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 399 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 300 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 301 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 302 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 303 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 304 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 305 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 306 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 307 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 308 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 309 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 310 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 311 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 312 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 313 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 314 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 315 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 316 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 317 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 318 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 319 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 320 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 321 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 322 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 323 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 324 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 325 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 326 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 327 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 328 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 329 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 330 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 331 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 332 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 333 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 334 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 335 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 336 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 337 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 338 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 339 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 340 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 341 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 342 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 343 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 344 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 345 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 346 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 347 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 348 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 349 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 310 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 351 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 352 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 353 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 354 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 355 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 356 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 357 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 358 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 359 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 360 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 361 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 362 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 363 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 364 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 365 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 366 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 367 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 368 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 369 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 370 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 371 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 372 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 373 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 374 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 375 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 376 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 377 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 378 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 379 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 380 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 381 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 382 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 383 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 384 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 385 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 386 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 387 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 388 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 389 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 390 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 391 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 392 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 393 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 394 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 395 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 396 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 397 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 398 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 399 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 400 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 401 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 402 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 403 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 404 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 405 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 406 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 407 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 408 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 409 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 410 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 411 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 412 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 413 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 414 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 415 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 416 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 417 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 418 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 419 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 420 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 421 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 422 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 423 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 424 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 425 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 426 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 427 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 428 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 429 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 430 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 431 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 432 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 433 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 434 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 435 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 436 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 437 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 438 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 439 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 440 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 441 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 442 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 443 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 444 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 445 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 446 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 447 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 448 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 449 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 410 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 451 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 452 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 453 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 454 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 455 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 456 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 457 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 458 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 459 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 460 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 461 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 462 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 463 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 464 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 465 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 466 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 467 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 468 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 469 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 470 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 471 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 472 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 473 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 474 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 475 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 476 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 477 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 478 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 479 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 480 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 481 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 482 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 483 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 484 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 485 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 486 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 487 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 488 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 489 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 490 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 491 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 492 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 493 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 494 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 495 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 496 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 497 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 498 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 499 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 500 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 501 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 502 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 503 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 504 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 505 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 506 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 507 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 508 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 509 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 510 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 511 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 512 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 513 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 514 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 515 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 516 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 517 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 518 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 519 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 520 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 521 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 522 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 523 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 524 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 525 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 526 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 527 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 528 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 529 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 530 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 531 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 532 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 533 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 534 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 535 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 536 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 537 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 538 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 539 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 540 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 541 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 542 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 543 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 544 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 545 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 546 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 547 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 548 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 549 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 510 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 551 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 552 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 553 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 554 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 555 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 556 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 557 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 558 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 559 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 560 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 561 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 562 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 563 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 564 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 565 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 566 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 567 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 568 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 569 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 570 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 571 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 572 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 573 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 574 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 575 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 576 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 577 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 578 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 579 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 580 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 581 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 582 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 583 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 584 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 585 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 586 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 587 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 588 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 589 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 590 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 591 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 592 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 593 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 594 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 595 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 596 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 597 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 598 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 599 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 600 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 601 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 602 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 603 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 604 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 605 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 606 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 607 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 608 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 609 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 610 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 611 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 612 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 613 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 614 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 615 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 616 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 617 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 618 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 619 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 620 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 621 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 622 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 623 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 624 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 625 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 626 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 627 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 628 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 629 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 630 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 631 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 632 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 633 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 634 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 635 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 636 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 637 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 638 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 639 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 640 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 641 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 642 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 643 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 644 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 645 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 646 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 647 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 648 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 649 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 610 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 651 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 652 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 653 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 654 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 655 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 656 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 657 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 658 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 659 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 660 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 661 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 662 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 663 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 664 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 665 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 666 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 667 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 668 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 669 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 670 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 671 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 672 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 673 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 674 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 675 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 676 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 677 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 678 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 679 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 680 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 681 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 682 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 683 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 684 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 685 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 686 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 687 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 688 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 689 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 690 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 691 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 692 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 693 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 694 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 695 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 696 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 697 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 698 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 699 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 700 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 701 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 702 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 703 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 704 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 705 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 706 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 707 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 708 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 709 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 710 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 711 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 712 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 713 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 714 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 715 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 716 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 717 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 718 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 719 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 720 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 721 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 722 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 723 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 724 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 725 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 726 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 727 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 728 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 729 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 730 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 731 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 732 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 733 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 734 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 735 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 736 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 737 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 738 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 739 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 740 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 741 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 742 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 743 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 744 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 745 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 746 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 747 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 748 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 749 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 710 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 751 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 752 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 753 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 754 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 755 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 756 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 757 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 758 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 759 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 760 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 761 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 762 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 763 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 764 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 765 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 766 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 767 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 768 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 769 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 770 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 771 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 772 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 773 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 774 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 775 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 776 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 777 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 778 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 779 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 780 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 781 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 782 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 783 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 784 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 785 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 786 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 787 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 788 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 789 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 790 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 791 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 792 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 793 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 794 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 795 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 796 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 797 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 798 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 799 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 800 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 801 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 802 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 803 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 804 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 805 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 806 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 807 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 808 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 809 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 810 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 811 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 812 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 813 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 814 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 815 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 816 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 817 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 818 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 819 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 820 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 821 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 822 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 823 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 824 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 825 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 826 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 827 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 828 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 829 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 830 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 831 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 832 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 833 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 834 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 835 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 836 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 837 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 838 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 839 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 840 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 841 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 842 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 843 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 844 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 845 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 846 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 847 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 848 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 849 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 810 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 851 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 852 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 853 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 854 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 855 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 856 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 857 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 858 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 859 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 860 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 861 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 862 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 863 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 864 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 865 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 866 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 867 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 868 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 869 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 870 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 871 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 872 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 873 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 874 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 875 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 876 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 877 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 878 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 879 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 880 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 881 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 882 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 883 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 884 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 885 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 886 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 887 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 888 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 889 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 890 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 891 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 892 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 893 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 894 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 895 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 896 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 897 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 898 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 899 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 900 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 901 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 902 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 903 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 904 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 905 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 906 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 907 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 908 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 909 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 910 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 911 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 912 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 913 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 914 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 915 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 916 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 917 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 918 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 919 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 920 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 921 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 922 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 923 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 924 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 925 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 926 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 927 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 928 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 929 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 930 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 931 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 932 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 933 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 934 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 935 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 936 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 937 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 938 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 939 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 940 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 941 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 942 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 943 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 944 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 945 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 946 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 947 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 948 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 949 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 910 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 951 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 952 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 953 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 954 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 955 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 956 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 957 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 958 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 959 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 960 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 961 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 962 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 963 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 964 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 965 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 966 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 967 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 968 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 969 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 970 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 971 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 972 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 973 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 974 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 975 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 976 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 977 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 978 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 979 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 980 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 981 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 982 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 983 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 984 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 985 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 986 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 987 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 988 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 989 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 990 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 991 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 992 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 993 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 994 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 995 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 996 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 997 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 998 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA < 999 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA $ cat > 000 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 001 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 002 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 003 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 004 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 005 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 006 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 007 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 008 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 009 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 010 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 011 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 012 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 013 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 014 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 015 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 016 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 017 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 018 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 019 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 020 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 021 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 022 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 023 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 024 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 025 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 026 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 027 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 028 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 029 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 030 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 031 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 032 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 033 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 034 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 035 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 036 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 037 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 038 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 039 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 040 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 041 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 042 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 043 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 044 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 045 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 046 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 047 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 048 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 049 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 010 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 051 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 052 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 053 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 054 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 055 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 056 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 057 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 058 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 059 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 060 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 061 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 062 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 063 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 064 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 065 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 066 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 067 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 068 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 069 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 070 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 071 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 072 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 073 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 074 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 075 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 076 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 077 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 078 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 079 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 080 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 081 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 082 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 083 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 084 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 085 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 086 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 087 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 088 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 089 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 090 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 091 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 092 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 093 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 094 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 095 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 096 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 097 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 098 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 099 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 100 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 101 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 102 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 103 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 104 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 105 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 106 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 107 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 108 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 109 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 110 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 111 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 112 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 113 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 114 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 115 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 116 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 117 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 118 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 119 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 120 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 121 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 122 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 123 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 124 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 125 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 126 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 127 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 128 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 129 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 130 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 131 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 132 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 133 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 134 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 135 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 136 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 137 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 138 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 139 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 140 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 141 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 142 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 143 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 144 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 145 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 146 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 147 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 148 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 149 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 110 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 151 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 152 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 153 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 154 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 155 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 156 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 157 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 158 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 159 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 160 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 161 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 162 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 163 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 164 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 165 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 166 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 167 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 168 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 169 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 170 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 171 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 172 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 173 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 174 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 175 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 176 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 177 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 178 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 179 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 180 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 181 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 182 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 183 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 184 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 185 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 186 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 187 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 188 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 189 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 190 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 191 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 192 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 193 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 194 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 195 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 196 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 197 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 198 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 199 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 200 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 201 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 202 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 203 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 204 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 205 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 206 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 207 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 208 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 209 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 210 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 211 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 212 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 213 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 214 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 215 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 216 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 217 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 218 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 219 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 220 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 221 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 222 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 223 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 224 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 225 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 226 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 227 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 228 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 229 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 230 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 231 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 232 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 233 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 234 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 235 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 236 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 237 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 238 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 239 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 240 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 241 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 242 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 243 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 244 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 245 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 246 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 247 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 248 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 249 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 210 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 251 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 252 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 253 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 254 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 255 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 256 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 257 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 258 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 259 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 260 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 261 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 262 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 263 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 264 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 265 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 266 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 267 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 268 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 269 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 270 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 271 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 272 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 273 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 274 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 275 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 276 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 277 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 278 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 279 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 280 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 281 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 282 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 283 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 284 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 285 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 286 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 287 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 288 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 289 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 290 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 291 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 292 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 293 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 294 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 295 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 296 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 297 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 298 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 399 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 300 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 301 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 302 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 303 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 304 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 305 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 306 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 307 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 308 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 309 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 310 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 311 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 312 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 313 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 314 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 315 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 316 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 317 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 318 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 319 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 320 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 321 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 322 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 323 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 324 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 325 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 326 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 327 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 328 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 329 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 330 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 331 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 332 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 333 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 334 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 335 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 336 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 337 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 338 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 339 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 340 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 341 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 342 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 343 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 344 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 345 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 346 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 347 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 348 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 349 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 310 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 351 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 352 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 353 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 354 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 355 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 356 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 357 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 358 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 359 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 360 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 361 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 362 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 363 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 364 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 365 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 366 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 367 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 368 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 369 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 370 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 371 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 372 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 373 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 374 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 375 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 376 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 377 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 378 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 379 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 380 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 381 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 382 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 383 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 384 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 385 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 386 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 387 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 388 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 389 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 390 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 391 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 392 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 393 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 394 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 395 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 396 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 397 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 398 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 399 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 400 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 401 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 402 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 403 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 404 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 405 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 406 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 407 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 408 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 409 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 410 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 411 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 412 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 413 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 414 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 415 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 416 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 417 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 418 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 419 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 420 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 421 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 422 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 423 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 424 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 425 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 426 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 427 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 428 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 429 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 430 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 431 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 432 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 433 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 434 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 435 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 436 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 437 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 438 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 439 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 440 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 441 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 442 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 443 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 444 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 445 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 446 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 447 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 448 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 449 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 410 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 451 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 452 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 453 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 454 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 455 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 456 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 457 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 458 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 459 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 460 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 461 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 462 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 463 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 464 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 465 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 466 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 467 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 468 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 469 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 470 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 471 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 472 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 473 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 474 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 475 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 476 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 477 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 478 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 479 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 480 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 481 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 482 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 483 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 484 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 485 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 486 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 487 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 488 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 489 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 490 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 491 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 492 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 493 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 494 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 495 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 496 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 497 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 498 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 499 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 500 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 501 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 502 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 503 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 504 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 505 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 506 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 507 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 508 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 509 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 510 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 511 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 512 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 513 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 514 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 515 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 516 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 517 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 518 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 519 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 520 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 521 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 522 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 523 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 524 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 525 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 526 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 527 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 528 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 529 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 530 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 531 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 532 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 533 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 534 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 535 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 536 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 537 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 538 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 539 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 540 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 541 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 542 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 543 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 544 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 545 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 546 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 547 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 548 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 549 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 510 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 551 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 552 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 553 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 554 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 555 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 556 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 557 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 558 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 559 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 560 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 561 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 562 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 563 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 564 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 565 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 566 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 567 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 568 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 569 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 570 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 571 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 572 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 573 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 574 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 575 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 576 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 577 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 578 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 579 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 580 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 581 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 582 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 583 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 584 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 585 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 586 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 587 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 588 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 589 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 590 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 591 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 592 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 593 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 594 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 595 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 596 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 597 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 598 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 599 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 600 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 601 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 602 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 603 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 604 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 605 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 606 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 607 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 608 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 609 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 610 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 611 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 612 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 613 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 614 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 615 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 616 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 617 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 618 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 619 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 620 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 621 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 622 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 623 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 624 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 625 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 626 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 627 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 628 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 629 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 630 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 631 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 632 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 633 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 634 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 635 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 636 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 637 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 638 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 639 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 640 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 641 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 642 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 643 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 644 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 645 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 646 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 647 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 648 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 649 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 610 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 651 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 652 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 653 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 654 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 655 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 656 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 657 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 658 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 659 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 660 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 661 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 662 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 663 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 664 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 665 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 666 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 667 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 668 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 669 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 670 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 671 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 672 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 673 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 674 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 675 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 676 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 677 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 678 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 679 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 680 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 681 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 682 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 683 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 684 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 685 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 686 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 687 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 688 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 689 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 690 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 691 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 692 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 693 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 694 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 695 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 696 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 697 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 698 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 699 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 700 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 701 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 702 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 703 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 704 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 705 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 706 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 707 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 708 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 709 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 710 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 711 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 712 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 713 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 714 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 715 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 716 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 717 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 718 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 719 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 720 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 721 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 722 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 723 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 724 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 725 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 726 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 727 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 728 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 729 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 730 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 731 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 732 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 733 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 734 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 735 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 736 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 737 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 738 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 739 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 740 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 741 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 742 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 743 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 744 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 745 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 746 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 747 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 748 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 749 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 710 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 751 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 752 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 753 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 754 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 755 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 756 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 757 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 758 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 759 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 760 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 761 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 762 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 763 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 764 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 765 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 766 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 767 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 768 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 769 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 770 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 771 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 772 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 773 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 774 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 775 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 776 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 777 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 778 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 779 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 780 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 781 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 782 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 783 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 784 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 785 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 786 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 787 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 788 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 789 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 790 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 791 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 792 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 793 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 794 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 795 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 796 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 797 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 798 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 799 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 800 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 801 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 802 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 803 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 804 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 805 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 806 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 807 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 808 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 809 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 810 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 811 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 812 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 813 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 814 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 815 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 816 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 817 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 818 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 819 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 820 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 821 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 822 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 823 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 824 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 825 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 826 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 827 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 828 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 829 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 830 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 831 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 832 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 833 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 834 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 835 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 836 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 837 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 838 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 839 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 840 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 841 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 842 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 843 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 844 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 845 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 846 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 847 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 848 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 849 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 810 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 851 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 852 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 853 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 854 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 855 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 856 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 857 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 858 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 859 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 860 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 861 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 862 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 863 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 864 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 865 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 866 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 867 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 868 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 869 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 870 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 871 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 872 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 873 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 874 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 875 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 876 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 877 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 878 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 879 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 880 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 881 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 882 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 883 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 884 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 885 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 886 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 887 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 888 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 889 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 890 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 891 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 892 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 893 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 894 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 895 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 896 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 897 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 898 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 899 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 900 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 901 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 902 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 903 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 904 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 905 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 906 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 907 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 908 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 909 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 910 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 911 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 912 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 913 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 914 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 915 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 916 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 917 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 918 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 919 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 920 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 921 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 922 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 923 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 924 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 925 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 926 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 927 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 928 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 929 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 930 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 931 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 932 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 933 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 934 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 935 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 936 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 937 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 938 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 939 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 940 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 941 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 942 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 943 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 944 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 945 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 946 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 947 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 948 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 949 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 910 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 951 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 952 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 953 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 954 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 955 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 956 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 957 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 958 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 959 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 960 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 961 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 962 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 963 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 964 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 965 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 966 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 967 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 968 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 969 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 970 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 971 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 972 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 973 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 974 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 975 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 976 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 977 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 978 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 979 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 980 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 981 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 982 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 983 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 984 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 985 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 986 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 987 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 988 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 989 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 990 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 991 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 992 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 993 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 994 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 995 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 996 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 997 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 998 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA > 999 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA SimGrid-3.14.159/tools/tesh/catch-return.tesh000644 001750 001750 00000000443 13031031305 022455 0ustar00mquinsonmquinson000000 000000 #! ./tesh # This suite builds and uses a program returning 1. # tesh must detect this condition and report the issue. ! expect return 2 < $ perl -e "exit 1" $ ${bindir:=.}/tesh > Test suite from stdin > [(stdin):1] perl -e "exit 1" > Test suite `(stdin)': NOK (<(stdin):1> returned code 1) SimGrid-3.14.159/tools/tesh/set-timeout.tesh000644 001750 001750 00000000237 13031031305 022336 0ustar00mquinsonmquinson000000 000000 #! ./tesh # This suite must be functional because we changed the timeout value to 10 # before sleeping 6 secs. ! timeout 10 $ sleep 6 ! timeout no $ sleep 1SimGrid-3.14.159/tools/tesh/basic.tesh000644 001750 001750 00000002745 13031031305 021146 0ustar00mquinsonmquinson000000 000000 #! ./tesh p This is a basic test < TOTO \ TUTU $ mkfile tmp_fich $ cat tmp_fich > TOTO TUTU ! expect return 2 ! output ignore < $ cat tmp_fich < > TUTU TOTO $ ${bindir:=.}/tesh $ rm tmp_fich p And now, some multilines examples p Really basic multiline < a < b < c < d $ mkfile tmp_fich $ cat tmp_fich > a > b > c > d $ sed -n '$=' tmp_fich > 4 $ rm tmp_fich p Now, check for spaces in input p * leading and trailing spaces... < a < b < c $ sed 's/ /_/g' > __a > _b_ > c__ p * empty lines... < a < < c $ sed '2s/^/b/' > a > b > c < < b < c $ sed '1s/^/a/' > a > b > c p Now that input should be good, check for spaces in output p * leading spaces... < _x $ sed 's/_/ /' > x ! expect return 2 ! output ignore < < _x < $ sed 's/_/ /' < > x $ ${bindir:=.}/tesh ! expect return 2 ! output ignore < < x < $ cat < > x $ ${bindir:=.}/tesh p * trailing spaces... < x_ $ sed 's/_/ /' > x ! expect return 2 ! output ignore < < x_ < $ sed 's/_/ /' < > x $ ${bindir:=.}/tesh ! expect return 2 ! output ignore < < x < $ cat < > x $ ${bindir:=.}/tesh p * empty lines... < a < < c $ cat > a > > c ! expect return 2 ! output ignore < < a < < < < c < $ cat < > a < > c $ ${bindir:=.}/tesh ! expect return 2 ! output ignore < < a < < c < $ cat < > a < > < > c $ ${bindir:=.}/tesh < < b < c $ cat > > b > c ! expect return 2 ! output ignore < < < < b < < c < $ cat < > b < > c $ ${bindir:=.}/tesh ! expect return 2 ! output ignore < < b < < c < $ cat < > < > b < > c $ ${bindir:=.}/tesh SimGrid-3.14.159/tools/tesh/bg-set-signal.tesh000644 001750 001750 00000000654 13031031305 022516 0ustar00mquinsonmquinson000000 000000 #! ./tesh # This suite builds and uses a program raising a segfault, ie a program dying # of SIGSEV. tesh must detect this condition and report the issue. $ cmake -E remove_directory temp_testdir-bg-set-signal $ mkdir temp_testdir-bg-set-signal $ cd temp_testdir-bg-set-signal < kill 'SEGV', $$; $ mkfile segfault.pl ! expect signal SIGSEGV $ perl segfault.pl $ cd .. $ cmake -E remove_directory temp_testdir-bg-set-signal SimGrid-3.14.159/tools/tesh/setenv.tesh000644 001750 001750 00000000644 13031031305 021365 0ustar00mquinsonmquinson000000 000000 #! ./tesh # This suite builds and uses a program returning 1. # tesh is instructed of this return code and must not whine. $ cmake -E remove_directory temp_testdir_setenv $ mkdir temp_testdir_setenv $ cd temp_testdir_setenv < print "tesh_test_toto=$ENV{tesh_test_toto}"; $ mkfile testenv.pl ! setenv tesh_test_toto=blah $ perl testenv.pl > tesh_test_toto=blah $ cd .. $ cmake -E remove_directory temp_testdir_setenv SimGrid-3.14.159/tools/tesh/set-return.tesh000644 001750 001750 00000000311 13031031305 022160 0ustar00mquinsonmquinson000000 000000 #! ./tesh # This suite builds and uses a program returning 1. # tesh is instructed of this return code and must not whine. ! expect return 1 $ perl -e "exit 1" ! expect return 42 $ perl -e "exit 42" SimGrid-3.14.159/tools/tesh/set-output-sort.tesh000644 001750 001750 00000005257 13031031305 023204 0ustar00mquinsonmquinson000000 000000 #! ./tesh p This tests whether TESH correctly sorts command output < p Test sorting and filtering of output < < $ true < < ! output sort < $ true < < $ printf 'profiling: foo\n' < < $ printf 'profiling: foo' < < ! output sort < $ printf 'profiling: foo\n' < < ! output sort < $ printf 'profiling: foo' < < $ printf 'a\nb\nc\nd\n' < > a < > b < > c < > d < < $ printf 'a\nb\nc\nd' < > a < > b < > c < > d < < ! output sort < $ printf 'c\nd\nb\na\n' < > a < > b < > c < > d < < ! output sort < $ printf 'c\nd\nb\na' < > a < > b < > c < > d < < $ printf 'a\nprofiling: foo\nprofiling: bar\nb\nc\nd\nprofiling: baz\n' < > a < > b < > c < > d < < $ printf 'a\nprofiling: foo\nprofiling: bar\nb\nc\nd\nprofiling: baz' < > a < > b < > c < > d < < ! output sort < $ printf 'c\nprofiling: foo\nprofiling: bar\nd\nb\na\nprofiling: baz\n' < > a < > b < > c < > d < < ! output sort < $ printf 'c\nprofiling: foo\nprofiling: bar\nd\nb\na\nprofiling: baz' < > a < > b < > c < > d $ ${bindir:=.}/tesh --ignore-jenkins > Ignore all cruft seen on SimGrid's continous integration servers > Test suite from stdin > [(stdin):1] Test sorting and filtering of output > [(stdin):3] true > [(stdin):6] true > [(stdin):8] printf 'profiling: foo\n' > [(stdin):10] printf 'profiling: foo' > [(stdin):13] printf 'profiling: foo\n' > [(stdin):16] printf 'profiling: foo' > [(stdin):18] printf 'a\nb\nc\nd\n' > [(stdin):24] printf 'a\nb\nc\nd' > [(stdin):31] printf 'c\nd\nb\na\n' > [(stdin):38] printf 'c\nd\nb\na' > [(stdin):44] printf 'a\nprofiling: foo\nprofiling: bar\nb\nc\nd\nprofiling: baz\n' > [(stdin):50] printf 'a\nprofiling: foo\nprofiling: bar\nb\nc\nd\nprofiling: baz' > [(stdin):57] printf 'c\nprofiling: foo\nprofiling: bar\nd\nb\na\nprofiling: baz\n' > [(stdin):64] printf 'c\nprofiling: foo\nprofiling: bar\nd\nb\na\nprofiling: baz' > Test suite from stdin OK p Check the Right Prefix Length (19) for "output sort" ! output sort 19 < 12345678901234567 B line < 12345678901234567 A line $ cat > 12345678901234567 A line > 12345678901234567 B line ! output sort 19 < 123456789012345678 B line < 123456789012345678 A line $ cat > 123456789012345678 B line > 123456789012345678 A line ! expect return 2 ! output ignore < ! output sort 19 < < 123456789012345678 B line < < 123456789012345678 A line < $ cat < > 123456789012345678 A line < > 123456789012345678 B line $ ${bindir:=.}/tesh p Check user-defined prefix length for "output sort" ! output sort 5 < 000 B line < 000 A line $ cat > 000 A line > 000 B line ! output sort 4 < 000 B line < 000 A line $ cat > 000 B line > 000 A line ! expect return 2 ! output ignore < ! output sort 4 < < 000 B line < < 000 A line < $ cat < > 000 A line < > 000 B line $ ${bindir:=.}/tesh SimGrid-3.14.159/tools/tesh/color.tesh000644 001750 001750 00000000504 13031031305 021172 0ustar00mquinsonmquinson000000 000000 ! output display $ printf "I \033[0;31mlove\033[0m tesh\n" > I love tesh ! output display $ echo -e "\e[1;4mBold and Underlined\e[0m" > Bold and Underlined ! output display $ echo -e "Default \e[94mLight blue\e[0m" > Default Light blue ! output display $ echo -e "\e[38;5;82mHello \e[38;5;198mWorld\e[0m" > Hello World SimGrid-3.14.159/tools/tesh/cd.tesh000644 001750 001750 00000001222 13031031305 020440 0ustar00mquinsonmquinson000000 000000 #! ./tesh # This example uses the cd command $ cmake -E remove_directory testdir_temp-cd $ mkdir testdir_temp-cd $ cd testdir_temp-cd # Check that there is nothing in the current dir (which must be testdir_temp-cd) $ ls # Check that tesh detects properly cd to non-existing directories ! expect return 4 < $ cd toto > Test suite from stdin > Chdir to toto failed: No such file or directory > Test suite `(stdin)': NOK (system error) $ ${bindir:=.}/tesh # The next command checks that there is a testdir_temp-cd in the upper directory, # ie that mkdir and cd both worked. $ test -e ../testdir_temp-cd $ cd .. $ cmake -E remove_directory testdir_temp-cd SimGrid-3.14.159/tools/tesh/catch-signal.tesh000644 001750 001750 00000001275 13031031305 022417 0ustar00mquinsonmquinson000000 000000 #! ./tesh # This suite builds and uses a program raising a segfault, ie a program dying # of SIGSEV. tesh must detect this condition and report the issue. $ cmake -E remove_directory temp_testdir-catch-signal $ mkdir temp_testdir-catch-signal $ cd temp_testdir-catch-signal < kill 'SEGV', $$; $ mkfile segfault.pl p Check that we notice when SEGV is raised ! expect signal SIGSEGV $ perl segfault.pl p Check that we return the expected return value on SEGV ! expect return 11 < $ perl segfault.pl $ ${bindir:=.}/tesh > Test suite from stdin > [(stdin):1] perl segfault.pl > Test suite `(stdin)': NOK (<(stdin):1> got signal SIGSEGV) $ cd .. $ cmake -E remove_directory temp_testdir-catch-signal SimGrid-3.14.159/tools/tesh/tesh.py000755 001750 001750 00000041545 13031031276 020530 0ustar00mquinsonmquinson000000 000000 #! @PYTHON_EXECUTABLE@ # -*- coding: utf-8 -*- """ tesh -- testing shell ======================== Copyright (c) 2012-2016. The SimGrid Team. All rights reserved. This program is free software; you can redistribute it and/or modify it under the terms of the license (GNU LGPL) which comes with this package. #TODO: child of child of child that printfs. Does it work? #TODO: a child dies after its parent. What happen? #TODO: regular expression in output #ex: >> Time taken: [0-9]+s #TODO: linked regular expression in output #ex: # >> Bytes sent: ([0-9]+) # >> Bytes recv: \1 # then, even better: # ! expect (\1 > 500) # TODO: If the output is sorted, we should report it to the users. Corresponding perl chunk # print "WARNING: Both the observed output and expected output were sorted as requested.\n"; # print "WARNING: Output were only sorted using the $sort_prefix first chars.\n" # if ( $sort_prefix > 0 ); # print "WARNING: Use to sort by simulated date and process ID only.\n"; # # print "----8<--------------- Begin of unprocessed observed output (as it should appear in file):\n"; # map {print "> $_\n"} @{$cmd{'unsorted got'}}; # print "--------------->8---- End of the unprocessed observed output.\n"; """ import sys, os import shlex import re import difflib import signal import argparse if sys.version_info[0] == 3: import subprocess import _thread else: raise "This program is expected to run with Python3 only" ############## # # Utilities # # # Singleton metaclass that works in Python 2 & 3 # http://stackoverflow.com/questions/6760685/creating-a-singleton-in-python class _Singleton(type): """ A metaclass that creates a Singleton base class when called. """ _instances = {} def __call__(cls, *args, **kwargs): if cls not in cls._instances: cls._instances[cls] = super(_Singleton, cls).__call__(*args, **kwargs) return cls._instances[cls] class Singleton(_Singleton('SingletonMeta', (object,), {})): pass SIGNALS_TO_NAMES_DICT = dict((getattr(signal, n), n) \ for n in dir(signal) if n.startswith('SIG') and '_' not in n ) #exit correctly def exit(errcode): #If you do not flush some prints are skipped sys.stdout.flush() #os._exit exit even when executed within a thread os._exit(errcode) def fatal_error(msg): print("[Tesh/CRITICAL] "+str(msg)) exit(1) #Set an environment variable. # arg must be a string with the format "variable=value" def setenv(arg): print("[Tesh/INFO] setenv "+arg) t = arg.split("=") os.environ[t[0]] = t[1] #os.putenv(t[0], t[1]) does not work #see http://stackoverflow.com/questions/17705419/python-os-environ-os-putenv-usr-bin-env #http://stackoverflow.com/questions/30734967/how-to-expand-environment-variables-in-python-as-bash-does def expandvars2(path): return re.sub(r'(? 1 and line[-2] == "\\": txt = txt[0:-1] line = next(self.f) self.linenumber += 1 txt += line[0:-1] return txt #keep the state of tesh (mostly configuration values) class TeshState(Singleton): def __init__(self): self.threads = [] self.args_suffix = "" self.ignore_regexps_common = [] self.wrapper = None self.keep = False def add_thread(self, thread): self.threads.append(thread) def join_all_threads(self): for t in self.threads: t.acquire() t.release() #Command line object class Cmd(object): def __init__(self): self.input_pipe = [] self.output_pipe_stdout = [] self.output_pipe_stderr = [] self.timeout = 5 self.args = None self.linenumber = -1 self.background = False self.cwd = None self.ignore_output = False self.expect_return = 0 self.output_display = False self.sort = -1 self.ignore_regexps = TeshState().ignore_regexps_common def add_input_pipe(self, l): self.input_pipe.append(l) def add_output_pipe_stdout(self, l): self.output_pipe_stdout.append(l) def add_output_pipe_stderr(self, l): self.output_pipe_stderr.append(l) def set_cmd(self, args, linenumber): self.args = args self.linenumber = linenumber def add_ignore(self, txt): self.ignore_regexps.append(re.compile(txt)) def remove_ignored_lines(self, lines): for ign in self.ignore_regexps: lines = [l for l in lines if not ign.match(l)] return lines def _cmd_mkfile(self, argline): filename = argline[len("mkfile "):] file = open(filename, "w") if file is None: fatal_error("Unable to create file "+filename) file.write("\n".join(self.input_pipe)) file.write("\n") file.close() def _cmd_cd(self, argline): args = shlex.split(argline) if len(args) != 2: fatal_error("Too many arguments to cd") try: os.chdir(args[1]) print("[Tesh/INFO] change directory to "+args[1]) except FileNotFoundError: print("Chdir to "+args[1]+" failed: No such file or directory") print("Test suite `"+FileReader().filename+"': NOK (system error)") exit(4) #Run the Cmd if possible. # Return False if nothing has been ran. def run_if_possible(self): if self.can_run(): if self.background: #Python threads loose the cwd self.cwd = os.getcwd() lock = _thread.allocate_lock() lock.acquire() TeshState().add_thread(lock) _thread.start_new_thread( Cmd._run, (self, lock) ) else: self._run() return True else: return False def _run(self, lock=None): #Python threads loose the cwd if self.cwd is not None: os.chdir(self.cwd) self.cwd = None #retrocompatibility: support ${aaa:=.} variable format def replace_perl_variables(m): vname = m.group(1) vdefault = m.group(2) if vname in os.environ: return "$"+vname else: return vdefault self.args = re.sub(r"\${(\w+):=([^}]*)}", replace_perl_variables, self.args) #replace bash environment variables ($THINGS) to their values self.args = expandvars2(self.args) if re.match("^mkfile ", self.args) is not None: self._cmd_mkfile(self.args) if lock is not None: lock.release() return if re.match("^cd ", self.args) is not None: self._cmd_cd(self.args) if lock is not None: lock.release() return if TeshState().wrapper is not None: self.timeout *= 20 self.args = TeshState().wrapper + self.args elif re.match(".*smpirun.*", self.args) is not None: self.args = "sh " + self.args self.args += TeshState().args_suffix print("["+FileReader().filename+":"+str(self.linenumber)+"] "+self.args) args = shlex.split(self.args) #print (args) try: proc = subprocess.Popen(args, bufsize=1, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True) except OSError as e: if e.errno == 8: e.strerror += "\nOSError: [Errno 8] Executed scripts should start with shebang line (like #!/bin/sh)" raise e cmdName = FileReader().filename+":"+str(self.linenumber) try: (stdout_data, stderr_data) = proc.communicate("\n".join(self.input_pipe), self.timeout) except subprocess.TimeoutExpired: print("Test suite `"+FileReader().filename+"': NOK (<"+cmdName+"> timeout after "+str(self.timeout)+" sec)") exit(3) if self.output_display: print(stdout_data) #remove text colors ansi_escape = re.compile(r'\x1b[^m]*m') stdout_data = ansi_escape.sub('', stdout_data) #print ((stdout_data, stderr_data)) if self.ignore_output: print("(ignoring the output of <"+cmdName+"> as requested)") else: stdouta = stdout_data.split("\n") while len(stdouta) > 0 and stdouta[-1] == "": del stdouta[-1] stdouta = self.remove_ignored_lines(stdouta) #the "sort" bash command is case unsensitive, # we mimic its behaviour if self.sort == 0: stdouta.sort(key=lambda x: x.lower()) self.output_pipe_stdout.sort(key=lambda x: x.lower()) elif self.sort > 0: stdouta.sort(key=lambda x: x[:self.sort].lower()) self.output_pipe_stdout.sort(key=lambda x: x[:self.sort].lower()) diff = list(difflib.unified_diff(self.output_pipe_stdout, stdouta,lineterm="",fromfile='expected', tofile='obtained')) if len(diff) > 0: print("Output of <"+cmdName+"> mismatch:") for line in diff: print(line) print("Test suite `"+FileReader().filename+"': NOK (<"+cmdName+"> output mismatch)") if lock is not None: lock.release() if TeshState().keep: f = open('obtained','w') obtained = stdout_data.split("\n") while len(obtained) > 0 and obtained[-1] == "": del obtained[-1] obtained = self.remove_ignored_lines(obtained) for line in obtained: f.write("> "+line+"\n") f.close() print("Obtained output kept as requested: "+os.path.abspath("obtained")) exit(2) #print ((proc.returncode, self.expect_return)) if proc.returncode != self.expect_return: if proc.returncode >= 0: print("Test suite `"+FileReader().filename+"': NOK (<"+cmdName+"> returned code "+str(proc.returncode)+")") if lock is not None: lock.release() exit(2) else: print("Test suite `"+FileReader().filename+"': NOK (<"+cmdName+"> got signal "+SIGNALS_TO_NAMES_DICT[-proc.returncode]+")") if lock is not None: lock.release() exit(-proc.returncode) if lock is not None: lock.release() def can_run(self): return self.args is not None ############## # # Main # # if __name__ == '__main__': parser = argparse.ArgumentParser(description='tesh -- testing shell', add_help=True) group1 = parser.add_argument_group('Options') group1.add_argument('teshfile', nargs='?', help='Name of teshfile, stdin if omitted') group1.add_argument('--cd', metavar='some/directory', help='ask tesh to switch the working directory before launching the tests') group1.add_argument('--setenv', metavar='var=value', action='append', help='set a specific environment variable') group1.add_argument('--cfg', metavar='arg', help='add parameter --cfg=arg to each command line') group1.add_argument('--log', metavar='arg', help='add parameter --log=arg to each command line') group1.add_argument('--ignore-jenkins', action='store_true', help='ignore all cruft generated on SimGrid continous integration servers') group1.add_argument('--wrapper', metavar='arg', help='Run each command in the provided wrapper (eg valgrind)') group1.add_argument('--keep', action='store_true', help='Keep the obtained output when it does not match the expected one') try: options = parser.parse_args() except: exit(1) if options.cd is not None: os.chdir(options.cd) if options.ignore_jenkins: print("Ignore all cruft seen on SimGrid's continous integration servers") TeshState().ignore_regexps_common = [ re.compile("^profiling:"), re.compile(".*WARNING: ASan doesn\'t fully support"), re.compile("Unable to clean temporary file C:.*")] if options.teshfile is None: f = FileReader(None) print("Test suite from stdin") else: f = FileReader(options.teshfile) print("Test suite '"+f.abspath+"'") if options.setenv is not None: for e in options.setenv: setenv(e) if options.cfg is not None: TeshState().args_suffix += " --cfg="+options.cfg if options.log is not None: TeshState().args_suffix += " --log="+options.log if options.wrapper is not None: TeshState().wrapper = options.wrapper if options.keep: TeshState().keep = True #cmd holds the current command line # tech commands will add some parameters to it # when ready, we execute it. cmd = Cmd() line = f.readfullline() while line is not None: #print(">>============="+line+"==<<") if len(line) == 0: #print ("END CMD block") if cmd.run_if_possible(): cmd = Cmd() elif line[0] == "#": pass elif line[0:2] == "p ": print("["+str(FileReader())+"] "+line[2:]) elif line[0:2] == "< ": cmd.add_input_pipe(line[2:]) elif line[0:1] == "<": cmd.add_input_pipe(line[1:]) elif line[0:2] == "> ": cmd.add_output_pipe_stdout(line[2:]) elif line[0:1] == ">": cmd.add_output_pipe_stdout(line[1:]) elif line[0:2] == "$ ": if cmd.run_if_possible(): cmd = Cmd() cmd.set_cmd(line[2:], f.linenumber) elif line[0:2] == "& ": if cmd.run_if_possible(): cmd = Cmd() cmd.set_cmd(line[2:], f.linenumber) cmd.background = True elif line[0:15] == "! output ignore": cmd.ignore_output = True #print("cmd.ignore_output = True") elif line[0:16] == "! output display": cmd.output_display = True cmd.ignore_output = True elif line[0:15] == "! expect return": cmd.expect_return = int(line[16:]) #print("expect return "+str(int(line[16:]))) elif line[0:15] == "! expect signal": sig = line[16:] #get the signal integer value from the signal module if sig not in signal.__dict__: fatal_error("unrecognized signal '"+sig+"'") sig = int(signal.__dict__[sig]) #popen return -signal when a process ends with a signal cmd.expect_return = -sig elif line[0:len("! timeout ")] == "! timeout ": if "no" in line[len("! timeout "):]: cmd.timeout = None else: cmd.timeout = int(line[len("! timeout "):]) elif line[0:len("! output sort")] == "! output sort": if len(line) >= len("! output sort "): sort = int(line[len("! output sort "):]) else: sort = 0 cmd.sort = sort elif line[0:len("! setenv ")] == "! setenv ": setenv(line[len("! setenv "):]) elif line[0:len("! ignore ")] == "! ignore ": cmd.add_ignore(line[len("! ignore "):]) else: fatal_error("UNRECOGNIZED OPTION") line = f.readfullline() cmd.run_if_possible() TeshState().join_all_threads() if f.filename == "(stdin)": print("Test suite from stdin OK") else: print("Test suite `"+f.filename+"' OK") SimGrid-3.14.159/tools/tesh/background.tesh000644 001750 001750 00000000507 13031031305 022176 0ustar00mquinsonmquinson000000 000000 $ cmake -E remove bgtesh.pl bgtesh.data < use strict; < sleep(1); < open (FILE, ") { < print; < } < close(FILE); < exit 0; < $ mkfile bgtesh.pl & perl bgtesh.pl > TOTO < TOTO $ mkfile bgtesh.data $ sleep 2 $ cmake -E remove bgtesh.pl bgtesh.data SimGrid-3.14.159/tools/tesh/generate_tesh000755 001750 001750 00000000376 13031031276 021750 0ustar00mquinsonmquinson000000 000000 #!/bin/sh # Generate a tesh file from a given command (simple case). command="$1" output=$(sh -c "$command" 2>&1) return=$? tesh="#! ./tesh ! expect return $return $ $command $(echo "$output" | sed 's/^/> /' ) " echo "$tesh" echo "$tesh" | tesh 1>&2 SimGrid-3.14.159/tools/tesh/catch-timeout.tesh000644 001750 001750 00000000435 13031031305 022625 0ustar00mquinsonmquinson000000 000000 #! ./tesh # This suite must be functional because we changed the timeout value to 10 # before sleeping 6 secs. ! expect return 3 < ! timeout 1 < $ sleep 6 > Test suite from stdin > [(stdin):2] sleep 6 > Test suite `(stdin)': NOK (<(stdin):2> timeout after 1 sec) $ ${bindir:=.}/tesh SimGrid-3.14.159/tools/tesh/set-output-ignore.tesh000644 001750 001750 00000000400 13031031305 023461 0ustar00mquinsonmquinson000000 000000 #! ./tesh p This tests whether TESH accepts to ignore command output < ! output ignore < > TOTO < < TUTU < $ cat $ ${bindir:=.}/tesh > Test suite from stdin > [(stdin):4] cat > (ignoring the output of <(stdin):4> as requested) > Test suite from stdin OK SimGrid-3.14.159/tools/tesh/bg-basic.tesh000644 001750 001750 00000000216 13031031305 021523 0ustar00mquinsonmquinson000000 000000 #! ./tesh p This is a basic test < TOTO \ TUTU & cat > TOTO TUTU p And now, some multilines examples < a < b < c < d & cat > a > b > c > dSimGrid-3.14.159/tools/simgrid2vite.sed000644 001750 001750 00000000255 13031031310 021330 0ustar00mquinsonmquinson000000 000000 /0 1 0 MPI/i \0 ROOT 0 ROOT s/0 1 0 MPI/0 1 ROOT MPI/ s/4 3 0 1 1 MPI_LINK/4 3 ROOT 1 1 MPI_LINK/ s/1 0 \"/1 root \"/g /6 0 1 1/i \6 0 root ROOT 0 "root" s/0 PTP/root PTP/g SimGrid-3.14.159/tools/lualib.patch000644 001750 001750 00000003434 13031031273 020526 0ustar00mquinsonmquinson000000 000000 This patch is to be applied to the Lua 5.3 source file to get a shared library. This is because the authors of Lua don't bother distributing a working build system for their software, so we have to make one... As unfortunate as it may be, there is nothing else we can do. -- Da SimGrid team. diff --git a/Makefile b/Makefile index 5ee5601..93830a3 100644 --- a/Makefile +++ b/Makefile @@ -52,7 +52,7 @@ R= $V.1 all: $(PLAT) $(PLATS) clean: - cd src && $(MAKE) $@ + cd src && $(MAKE) $@ V=$(V) R=$(R) test: dummy src/lua -v diff --git a/src/Makefile b/src/Makefile index d71c75c..64bda79 100644 --- a/src/Makefile +++ b/src/Makefile @@ -7,7 +7,7 @@ PLAT= none CC= gcc -std=gnu99 -CFLAGS= -O2 -Wall -Wextra -DLUA_COMPAT_5_2 $(SYSCFLAGS) $(MYCFLAGS) +CFLAGS= -O2 -Wall -Wextra -DLUA_COMPAT_5_2 $(SYSCFLAGS) $(MYCFLAGS) -fPIC LDFLAGS= $(SYSLDFLAGS) $(MYLDFLAGS) LIBS= -lm $(SYSLIBS) $(MYLIBS) @@ -29,6 +29,7 @@ MYOBJS= PLATS= aix bsd c89 freebsd generic linux macosx mingw posix solaris LUA_A= liblua.a +LUA_SO= liblua.so CORE_O= lapi.o lcode.o lctype.o ldebug.o ldo.o ldump.o lfunc.o lgc.o llex.o \ lmem.o lobject.o lopcodes.o lparser.o lstate.o lstring.o ltable.o \ ltm.o lundump.o lvm.o lzio.o @@ -43,7 +44,7 @@ LUAC_T= luac LUAC_O= luac.o ALL_O= $(BASE_O) $(LUA_O) $(LUAC_O) -ALL_T= $(LUA_A) $(LUA_T) $(LUAC_T) +ALL_T= $(LUA_A) $(LUA_T) $(LUAC_T) $(LUA_SO) ALL_A= $(LUA_A) # Targets start here. @@ -59,6 +60,11 @@ $(LUA_A): $(BASE_O) $(AR) $@ $(BASE_O) $(RANLIB) $@ +$(LUA_SO): $(CORE_O) $(LIB_O) + $(CC) -shared -ldl -Wl,-soname,$(LUA_SO).$(V) -o $@.$(R) $? -lm $(MYLDFLAGS) + ln -sf $(LUA_SO).$(R) $(LUA_SO).$(V) + ln -sf $(LUA_SO).$(R) $(LUA_SO) + $(LUA_T): $(LUA_O) $(LUA_A) $(CC) -o $@ $(LDFLAGS) $(LUA_O) $(LUA_A) $(LIBS) SimGrid-3.14.159/tools/generate-dwarf-functions000755 001750 001750 00000003465 13031031276 023073 0ustar00mquinsonmquinson000000 000000 #!/bin/sh # Generate files from a given dwarf.h # Usage: tools/generate-dwarf-functions /usr/include/dwarf.h cat - > src/mc/mc_dwarf_tagnames.cpp < #include #include namespace simgrid { namespace dwarf { /** \brief Get the name of a dwarf tag (DW_TAG_*) from its code * * \param tag tag code (see the DWARF specification) * \return name of the tag */ XBT_PRIVATE const char *tagname(int tag) { switch (tag) { $(cat "$1" | grep DW_TAG_ | sed 's/.*\(DW_TAG_[^ ]*\) = \(0x[0-9a-f]*\).*/ case \2: return "\1";/') case DW_TAG_invalid: return "DW_TAG_invalid"; default: return "DW_TAG_unknown"; } } } } EOF cat - > src/mc/mc_dwarf_attrnames.cpp << EOF /* Copyright (c) 2014-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* Warning: autogenerated, do not edit! */ #include #include namespace simgrid { namespace dwarf { /** \brief Get the name of an attribute (DW_AT_*) from its code * * \param attr attribute code (see the DWARF specification) * \return name of the attribute */ XBT_PRIVATE const char *attrname(int attr) { switch (attr) { $(cat "$1" | grep DW_AT_ | sed 's/.*\(DW_AT_[^ ]*\) = \(0x[0-9a-f]*\).*/ case \2: return "\1";/') default: return "DW_AT_unknown"; } } } } EOF SimGrid-3.14.159/tools/graphicator/000755 001750 001750 00000000000 13031031306 020531 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/tools/graphicator/CMakeLists.txt000644 001750 001750 00000001373 13031031274 023301 0ustar00mquinsonmquinson000000 000000 add_executable (graphicator graphicator.c) target_link_libraries(graphicator simgrid) set_target_properties(graphicator PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin) ADD_TESH(graphicator --setenv srcdir=${CMAKE_HOME_DIRECTORY} --setenv bindir=${CMAKE_BINARY_DIR}/bin --cd ${CMAKE_HOME_DIRECTORY}/tools/graphicator graphicator.tesh) ## Clean generated files get_directory_property(extra_clean_files ADDITIONAL_MAKE_CLEAN_FILES) set_directory_properties(PROPERTIES ADDITIONAL_MAKE_CLEAN_FILES "${extra_clean_files};${CMAKE_CURRENT_BINARY_DIR}/simgrid.trace;") set(tesh_files ${tesh_files} ${CMAKE_CURRENT_SOURCE_DIR}/graphicator.tesh PARENT_SCOPE) set(tools_src ${tools_src} ${CMAKE_CURRENT_SOURCE_DIR}/graphicator.c PARENT_SCOPE) SimGrid-3.14.159/tools/graphicator/graphicator.c000644 001750 001750 00000001430 13031031306 023176 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/msg.h" #include "xbt/graph.h" XBT_LOG_NEW_DEFAULT_CATEGORY(graphicator, "Graphicator Logging System"); int main(int argc, char **argv) { XBT_LOG_CONNECT(graphicator); MSG_init(&argc, argv); xbt_assert(argc == 3, "Usage: %s ", argv[0]); MSG_create_environment(argv[1]); int status = TRACE_platform_graph_export_graphviz (argv[2]); xbt_assert(status != 0, "%s expects --cfg=tracing:yes --cfg=tracing/platform:yes", argv[0]); return 0; } SimGrid-3.14.159/tools/graphicator/graphicator.tesh000644 001750 001750 00000000476 13031031305 023727 0ustar00mquinsonmquinson000000 000000 #! ./tesh $ ${bindir:=.}/graphicator ${srcdir:=.}/teshsuite/simdag/platforms/one_cluster.xml --cfg=tracing:yes --cfg=tracing/platform:yes test.dot > [0.000000] [xbt_cfg/INFO] Configuration change: Set 'tracing' to 'yes' > [0.000000] [xbt_cfg/INFO] Configuration change: Set 'tracing/platform' to 'yes' $ rm test.dot SimGrid-3.14.159/tools/MSG_visualization/000755 001750 001750 00000000000 13031031276 021643 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/tools/MSG_visualization/colorize.pl000755 001750 001750 00000006231 13031031276 024033 0ustar00mquinsonmquinson000000 000000 #!/usr/bin/env perl # Copyright (c) 2005, 2007, 2010, 2014. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. $col_white = "\033[00m"; $col_black = "\033[30m"; $col_red = "\033[31m"; $col_green = "\033[32m"; $col_yellow = "\033[33m"; $col_blue = "\033[34m"; $col_purple = "\033[35m"; $col_cyan = "\033[36m"; $col_ltgray = "\033[37m"; $col_darkgray = "\033[30m"; $col_norm = $col_white; $col_background = "\033[07m"; $col_brighten = "\033[01m"; $col_underline = "\033[04m"; $col_blink = "\033[05m"; # Customize colors here... $col_default = $col_ltgray; my (@coltab) = ( $col_green, $col_yellow, $col_purple, $col_cyan, $col_red, $col_blue, $col_background . $col_green, $col_background . $col_yellow, $col_background . $col_purple, $col_background . $col_cyan, $col_background . $col_red, $col_background . $col_blue, $col_background . $col_magenta, ); my %pid; # Get options while (($_ = $ARGV[0]) =~ /^-/) { shift; if (/-location/i) { $opt_print_location = 1; shift; } elsif (/-h(elp)?$|-u(sage)?$/i) { print< where is a text file of values or '-' for STDIN Options: () denote short version -location Print the location in the code of the message. EOH ; exit; } } sub pidcolor { my $pid = shift; unless (defined($pid{$pid})) { # first time we see this pid. Affect it a color $pid{$pid}=(scalar keys %pid) % (scalar @coltab); } return $coltab[$pid{$pid}]; } sub print_line { my($host,$procname,$pid,$date,$location,$xbt_channel,$message)=@_; print $col_norm; printf "[% 10.6f]",$date; print pidcolor($pid); if(defined($location)) { printf "[%10s:%-10s] %s ",$host,$procname,$location; } else { printf "[%10s:%-10s]",$host,$procname; } print " $message"; print $col_norm."\n"; } # Read the messages and do the job while (<>) { $orgline = $thisline = $_; # Typical line [Gatien:slave:(9) 11.243148] msg/gos.c:137: [msg_gos/DEBUG] Action terminated if ($thisline =~ /^\[(.+):([^:]+):\((\d+)\) ([\d\.]*)\] ([^\[]*) \[([^\[]*)\] (.*)$/) { $host=$1; $procname=$2; $pid=$3; $date=$4; if($opt_print_location) { $location=$5; $location =~ s/:$//; } else { $location = undef; } $xbt_channel=$6; $message=$7; print_line($host,$procname,$pid,$date,$location,$xbt_channel,$message); # Typical line [Boivin:slave:(2) 9.269357] [pmm/INFO] ROW: step(2)<>myrow(0). Receive data from TeX } elsif ($thisline =~ /^\[([^:]+):([^:]+):\((\d+)\) ([\d\.]*)\] \[([^\[]*)\] (.*)$/) { $host=$1; $procname=$2; $pid=$3; $date=$4; $xbt_channel=$5; $message=$6; print_line($host,$procname,$pid,$date,undef,$xbt_channel,$message); } elsif ( $thisline =~ /^==(\d+)== (.*)$/) { # take care of valgrind outputs print pidcolor($1)."$2\n"; } else { print $col_default. $orgline; } } print $col_norm; SimGrid-3.14.159/tools/MSG_visualization/trace2fig.pl000755 001750 001750 00000026417 13031031276 024063 0ustar00mquinsonmquinson000000 000000 #!/usr/bin/env perl # Copyright (c) 2006-2007, 2014. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. use strict; use warnings; #use Data::Dumper; use XFig; use POSIX; my($grid_Y_size)=225; my($grid_X_size)=100550; # Can be changed to improve readability in function of the total execution time my($color_suspended)=1; my($color_compute)=2; my($color_wait_for_recpt)=3; my($color_communicate)=4; # Determine the order of the colors in the legend my(@color_list)=($color_compute,$color_communicate,$color_wait_for_recpt,$color_suspended); sub read_cat { my(%Cat); my($filename)=@_; my($line); open INPUT, $filename; while (defined($line=)) { chomp $line; if($line =~ /^7\s/) { my($event,$date,$id,$type,$father,@name) = split(/\s+/,$line); $Cat{$id}{name}="@name "; $Cat{$id}{name}=~s/\"//g; $Cat{$id}{father}=$father; $Cat{$id}{type}=$type; $Cat{$id}{date}=$date; } } close INPUT; return \%Cat; } sub read_event { my($filename,$Cat)=@_; my($line); open INPUT, $filename; while (defined($line=)) { chomp $line; if($line =~ /^11\s/) { my($event,$date,$type,$id,$state) = split(/\s+/,$line); push @{$$Cat{$id}{state}}, [$date,$state]; } if($line =~ /^12\s/) { my($event,$date,$type,$id) = split(/\s+/,$line); push @{$$Cat{$id}{state}}, [$date]; } } close INPUT; } sub read_link { my($filename)=@_; my($line); my(%link); open INPUT, $filename; while (defined($line=)) { chomp $line; if($line =~ /^16\s/) { my($event,$date,$type,$father,$channel,$src,$key,$trash) = split(/\t+/,$line); my($numkey)=hex "$key"; while (defined($link{$numkey})) {$numkey++;} $link{$numkey}{src}=$src; $link{$numkey}{src_date}=$date; } if($line =~ /^17\s/) { my($event,$date,$type,$father,$channel,$dst,$key,$trash) = split(/\t+/,$line); my($numkey)=hex "$key"; while (defined($link{$numkey}{dst})) {$numkey++;} $link{$numkey}{dst}=$dst; $link{$numkey}{dst_date}=$date; } } close INPUT; return \%link; } sub build_cat_tree { my($root,$Cat)=@_; my(@childs)=(); my($cat); foreach $cat (keys %$Cat) { if($$Cat{$cat}{father} eq $root) { push @childs, build_cat_tree($cat,$Cat); } # print "$$Cat{$cat}{name}\t\t $Cat{$cat}{father}\n"; } return [$root,@childs]; } sub build_cat_list { my($tree,$cat_list)=@_; my($root) = shift @$tree; my($u); push @$cat_list,$root; foreach $u (@$tree) { build_cat_list($u,$cat_list); } unshift @$tree, $root; } sub cat_sorting_function { my($cat1,$cat2,$Cat)=@_; if (!defined($$Cat{$cat1}{state})) { if (!defined($$Cat{$cat2}{state})) { return 0; } else { return 1; } } if (!defined($$Cat{$cat2}{state})) { return -1; } my($comp) = $$Cat{$$Cat{$cat1}{'father'}}{'name'} cmp $$Cat{$$Cat{$cat2}{'father'}}{'name'}; if ($comp == 0) { return $$Cat{$cat1}{'name'} cmp $$Cat{$cat2}{'name'}; } else { return $comp; } } sub update_host_Y { my($host,$i) = @_; if (!defined($$host{'Y_min_host'})) { $$host{'Y_min_host'} = $i; } else { if ($$host{'Y_min_host'} > $i) { $$host{'Y_min_host'} = $i; } } if (!defined($$host{'Y_max_host'})) { $$host{'Y_max_host'} = $i+1; } else { if ($$host{'Y_max_host'} < $i+1) { $$host{'Y_max_host'} = $i+1; } } } sub set_cat_position { my($Cat,$cat_list)=@_; my($i)=0; my($cat); foreach $cat (sort {cat_sorting_function($a,$b,$Cat)} @$cat_list) { if(defined($$Cat{$cat}{state})) { update_host_Y($$Cat{$$Cat{$cat}{'father'}},$i); $$Cat{$cat}{Y_min} = $i; $$Cat{$cat}{Y_max} = $i+1; $i++; } } } sub create_fig { my($filename)=shift; my($fig)=new XFig; $fig->{object} = 'compound'; # Compound $fig->{elements} = []; $fig->{version} = 3.2; $fig->{orientation} = 'Landscape'; $fig->{justification} = 'Center'; $fig->{units} = 'Metric'; $fig->{papersize} = 'A4'; $fig->{magnification} = '100.00'; $fig->{multiplepage} = 'Single'; $fig->{transparent} = '-2'; $fig->{resolution} = '1200'; $fig->{coordsystem} = '2'; $fig->{filename} = $filename; return $fig; } sub draw_cat { my($fig,$Cat,$Link)=@_; my($cat,$e,$link); my($max_string_length)=0; foreach $cat (keys %$Cat) { next unless (defined($$Cat{$cat}{Y_min}) && defined($$Cat{$cat}{Y_max})); my($text) = new XFig ('text'); # $text->{'text'} = "$$Cat{$$Cat{$cat}{father}}{name}"."$$Cat{$cat}{name}"; my($printed_name)= $$Cat{$cat}{name}; $printed_name =~ s/\d+ \(0\)\s*$//; if (length($printed_name) > $max_string_length) { $max_string_length = length($printed_name); } $text->{'text'} = "$printed_name"; $text->{'y'} = ($$Cat{$cat}{Y_min}+$$Cat{$cat}{Y_max})/2*$grid_Y_size+68; $text->{'x'} = -100; $text->{'subtype'} = 2; $fig->add ($text); } my($max_date)=0; foreach $cat (keys %$Cat) { next unless (defined($$Cat{$cat}{Y_min}) && defined($$Cat{$cat}{Y_max})); my(@states)=(); my($e); foreach $e (@{$$Cat{$cat}{state}}) { my($new_date,$state) = ($$e[0],$$e[1]); if ($new_date > $max_date) { $max_date = $new_date; } if(defined($state)) { push @states, $e; } else { my($old_event) = pop @states; my($old_date) = $$old_event[0]; $state = $$old_event[1]; # LM: I added the next line because of "undefined values"... # normally, I think that this should not happen, but this part of code is a bit too cryptic to me next unless (defined($state)); my($line) = new XFig ('polyline'); $line->{'depth'} = 50; # line $line->{'subtype'} = 1; # line $line->{'points'} = [ [$old_date*$grid_X_size, $$Cat{$cat}{Y_min}*$grid_Y_size], [$new_date*$grid_X_size, $$Cat{$cat}{Y_min}*$grid_Y_size], [$new_date*$grid_X_size, $$Cat{$cat}{Y_max}*$grid_Y_size], [$old_date*$grid_X_size, $$Cat{$cat}{Y_max}*$grid_Y_size], [$old_date*$grid_X_size, $$Cat{$cat}{Y_min}*$grid_Y_size] ]; $line->{'areafill'} = 20; if($state eq "S") { $line->{'fillcolor'} = $color_suspended; } elsif ($state eq "E") { $line->{'fillcolor'} = $color_compute; } elsif ($state eq "B") { $line->{'fillcolor'} = $color_wait_for_recpt; } elsif ($state eq "C") { $line->{'fillcolor'} = $color_communicate; } $fig->add ($line); } } } foreach $link (keys %$Link) { my($line) = new XFig ('polyline'); my($src_date)=$$Link{$link}{src_date}; my($src)=$$Link{$link}{src}; my($dst_date)=$$Link{$link}{dst_date}; my($dst)=$$Link{$link}{dst}; $line->{'subtype'} = 1; # line print STDERR "$link: $src ($src_date) -> $dst ($dst_date)\n"; print STDERR "$$Cat{$src}{name} -> $$Cat{$dst}{name}\n"; $line->{'points'} = [ [$src_date*$grid_X_size, ($$Cat{$src}{Y_min}+$$Cat{$src}{Y_max})/2*$grid_Y_size], [$dst_date*$grid_X_size, ($$Cat{$dst}{Y_min}+$$Cat{$dst}{Y_max})/2*$grid_Y_size] ]; $line->{'forwardarrow'} = ['1', '1', '1.00', '60.00', '120.00']; $fig->add ($line); } # Host visualization my($max_Y)= 0; my($index_fill)=0; my($width_of_one_letter)=80; my($min_x_for_host)=-400 - $max_string_length*$width_of_one_letter; my($host_text_x)= $min_x_for_host + 200; foreach $cat (keys %$Cat) { next unless (defined($$Cat{$cat}{Y_min_host}) && defined($$Cat{$cat}{Y_max_host})); my($line) = new XFig ('polyline'); $line->{'depth'} = 150; $line->{'subtype'} = 1; # line $line->{'points'} = [ [$min_x_for_host, $$Cat{$cat}{Y_min_host}*$grid_Y_size], [$max_date*$grid_X_size+150, $$Cat{$cat}{Y_min_host}*$grid_Y_size], [$max_date*$grid_X_size+150, $$Cat{$cat}{Y_max_host}*$grid_Y_size], [$min_x_for_host, $$Cat{$cat}{Y_max_host}*$grid_Y_size] ]; $line->{'areafill'} = 4+3*($index_fill % 2); $line->{'fillcolor'} = 0; $line->{'thickness'} = 0; $index_fill++; $fig->add ($line); my($text) = new XFig ('text'); $text->{'text'} = "$$Cat{$cat}{name}"; $text->{'angle'} = 3.14159265/2; $text->{'x'} = $host_text_x; $text->{'y'} = ($$Cat{$cat}{Y_min_host}+$$Cat{$cat}{Y_max_host})/2*$grid_Y_size; $text->{'subtype'} = 1; $text->{'font_size'} = 30; $fig->add ($text); if ($max_Y<$$Cat{$cat}{Y_max_host}) { $max_Y = $$Cat{$cat}{Y_max_host}; } } # Legend: my($i)=1; my($color); foreach $color (@color_list) { my($min_x)=0; my($min_Y)=($max_Y+1)*$grid_Y_size; my($width)=1700; my($height)=$grid_Y_size; my($line) = new XFig ('polyline'); $line->{'depth'} = 50; $line->{'subtype'} = 1; # line $line->{'points'} = [ [$min_x,$min_Y + ($i-1)*$height ], [$min_x + $width,$min_Y + ($i-1)*$height], [$min_x + $width,$min_Y+$height + ($i-1)*$height], [$min_x,$min_Y+$height + ($i-1)*$height], [$min_x,$min_Y+ ($i-1)*$height]]; $line->{'areafill'} = 20; $line->{'fillcolor'} = $color; $fig->add ($line); my($text) = new XFig ('text'); if ($color==$color_suspended) { $text->{'text'} = "Suspended"; } if ($color==$color_compute) { $text->{'text'} = "Computing"; } if ($color==$color_wait_for_recpt) { $text->{'text'} = "Waiting for reception"; } if ($color==$color_communicate) { $text->{'text'} = "Communicating"; } $text->{'y'} = $min_Y + ($i-0.5)*$height +68; $text->{'x'} = 50; $text->{'subtype'} = 0; $fig->add ($text); $i++; } # Time axis my($line) = new XFig ('polyline'); $line->{'depth'} = 0; $line->{'subtype'} = 1; # line $line->{'points'} = [ [0,0],[$max_date * $grid_X_size+150,0] ]; $line->{'forwardarrow'} = ['1', '1', '1.00', '60.00', '120.00']; $fig->add ($line); my($digits)=POSIX::floor(log($max_date)/log(10)); my($exponent) = 10**$digits; my($mantissa)= $max_date / $exponent; my($incr); if ($mantissa<2) { $incr = $exponent/10; } elsif ($mantissa<5) { $incr = $exponent/4; } else { $incr = $exponent/2; } print "$max_date $digits $exponent $mantissa $incr\n"; my($x); for($x=0; $x < $max_date; $x += $incr) { print "$x\n"; $line = new XFig ('polyline'); $line->{'depth'} = 0; $line->{'subtype'} = 1; # line $line->{'points'} = [ [$x * $grid_X_size,0],[$x * $grid_X_size, -100] ]; $line->{'forwardarrow'} = 0; $fig->add ($line); my($text) = new XFig ('text'); $text->{'text'} = "$x"; $text->{'y'} = -200; $text->{'x'} = $x * $grid_X_size; $text->{'subtype'} = 1; $fig->add ($text); } # Empty line so that the text of the time axis can be seen on the pdf $line = new XFig ('polyline'); $line->{'depth'} = 999; $line->{'subtype'} = 1; # line $line->{'thickness'} = 0; $line->{'points'} = [ [0,0],[0, -400] ]; $fig->add ($line); } sub main { my($Cat) = read_cat($ARGV[0]); my($cat_tree)=build_cat_tree("0",$Cat); read_event($ARGV[0],$Cat); my($Link)=read_link($ARGV[0]); # print Dumper($cat_tree); # print Dumper($Cat); my($cat_list)=[]; build_cat_list($cat_tree,$cat_list); shift @$cat_list; shift @$cat_list; # print "@$cat_list \n"; set_cat_position($Cat,$cat_list); my($fig)=create_fig("toto.fig"); draw_cat($fig,$Cat,$Link); $fig->writefile (); system "fig2dev -L pdf toto.fig toto.pdf"; } main; SimGrid-3.14.159/tools/cmake/000755 001750 001750 00000000000 13031031274 017312 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/tools/cmake/MakeLibWin.cmake000644 001750 001750 00000001267 13031031273 022303 0ustar00mquinsonmquinson000000 000000 ### Make Libs add_library(simgrid SHARED ${simgrid_sources}) if(MSVC) set_target_properties(simgrid PROPERTIES COMPILE_FLAGS "/DDLL_EXPORT" VERSION ${libsimgrid_version} ) else() set_target_properties(simgrid PROPERTIES COMPILE_FLAGS "-DDLL_EXPORT" LINK_FLAGS "-shared" VERSION ${libsimgrid_version} PREFIX "lib" SUFFIX ".dll" IMPORT_PREFIX "lib" IMPORT_SUFFIX ".dll") set(SIMGRID_DEP "-lm") if (HAVE_PTHREAD) set(SIMGRID_DEP "${SIMGRID_DEP} -lpthread") endif() if (HAVE_BOOST_CONTEXTS) set(SIMGRID_DEP "${SIMGRID_DEP} ${Boost_CONTEXT_LIBRARY}") endif() endif() target_link_libraries(simgrid ${SIMGRID_DEP}) SimGrid-3.14.159/tools/cmake/Tests.cmake000644 001750 001750 00000012551 13031031273 021421 0ustar00mquinsonmquinson000000 000000 IF(enable_smpi AND NOT WIN32) execute_process(COMMAND chmod a=rwx ${CMAKE_BINARY_DIR}/bin/smpicc) execute_process(COMMAND chmod a=rwx ${CMAKE_BINARY_DIR}/bin/smpicxx) execute_process(COMMAND chmod a=rwx ${CMAKE_BINARY_DIR}/bin/smpiff) execute_process(COMMAND chmod a=rwx ${CMAKE_BINARY_DIR}/bin/smpif90) execute_process(COMMAND chmod a=rwx ${CMAKE_BINARY_DIR}/bin/smpirun) ENDIF() SET(TESH_OPTION "--ignore-jenkins") SET(TESH_COMMAND "${PYTHON_EXECUTABLE}" ${CMAKE_BINARY_DIR}/bin/tesh) IF(enable_memcheck) INCLUDE(FindValgrind) ENDIF() #some tests may take forever on non futexes systems, using busy_wait with n cores < n workers # default to posix for these tests if futexes are not supported IF(NOT HAVE_FUTEX_H) SET(CONTEXTS_SYNCHRO --cfg contexts/synchro:posix) ENDIF() MACRO(ADD_TESH NAME) SET(ARGT ${ARGV}) LIST(REMOVE_AT ARGT 0) IF(WIN32) STRING(REPLACE "§" "\;" ARGT "${ARGT}") ENDIF() if(TESH_WRAPPER) ADD_TEST(${NAME} ${TESH_COMMAND} --wrapper "${TESH_WRAPPER}" ${TESH_OPTION} ${ARGT}) else() ADD_TEST(${NAME} ${TESH_COMMAND} ${TESH_OPTION} ${ARGT}) endif() ENDMACRO() MACRO(ADD_TESH_FACTORIES NAME FACTORIES) SET(ARGR ${ARGV}) LIST(REMOVE_AT ARGR 0) # remove name FOREACH(I ${FACTORIES}) # remove all factories LIST(REMOVE_AT ARGR 0) ENDFOREACH() FOREACH(FACTORY ${FACTORIES}) if ((${FACTORY} STREQUAL "thread" AND HAVE_THREAD_CONTEXTS) OR (${FACTORY} STREQUAL "boost" AND HAVE_BOOST_CONTEXTS) OR (${FACTORY} STREQUAL "raw" AND HAVE_RAW_CONTEXTS) OR (${FACTORY} STREQUAL "ucontext" AND HAVE_UCONTEXT_CONTEXTS)) ADD_TESH("${NAME}-${FACTORY}" "--cfg" "contexts/factory:${FACTORY}" ${ARGR}) ENDIF() ENDFOREACH() ENDMACRO() IF(enable_java) IF(WIN32) SET(TESH_CLASSPATH "${CMAKE_BINARY_DIR}/examples/java/\;${CMAKE_BINARY_DIR}/teshsuite/java/\;${SIMGRID_JAR}") STRING(REPLACE "\;" "§" TESH_CLASSPATH "${TESH_CLASSPATH}") ELSE() SET(TESH_CLASSPATH "${CMAKE_BINARY_DIR}/examples/java/:${CMAKE_BINARY_DIR}/teshsuite/java/:${SIMGRID_JAR}") ENDIF() ENDIF() IF(HAVE_MC) ADD_TESH_FACTORIES(mc-bugged1 "ucontext;raw" --setenv bindir=${CMAKE_BINARY_DIR}/examples/msg/mc --cd ${CMAKE_HOME_DIRECTORY}/examples/msg/mc bugged1.tesh) ADD_TESH_FACTORIES(mc-bugged2 "ucontext;raw" --setenv bindir=${CMAKE_BINARY_DIR}/examples/msg/mc --cd ${CMAKE_HOME_DIRECTORY}/examples/msg/mc bugged2.tesh) IF(HAVE_UCONTEXT_CONTEXTS AND SIMGRID_PROCESSOR_x86_64) # liveness model-checking works only on 64bits (for now ...) ADD_TESH(mc-bugged1-liveness-ucontext --setenv bindir=${CMAKE_BINARY_DIR}/examples/msg/mc --cd ${CMAKE_HOME_DIRECTORY}/examples/msg/mc bugged1_liveness.tesh) ADD_TESH(mc-bugged1-liveness-ucontext-sparse --setenv bindir=${CMAKE_BINARY_DIR}/examples/msg/mc --cd ${CMAKE_HOME_DIRECTORY}/examples/msg/mc bugged1_liveness_sparse.tesh) ADD_TESH(mc-bugged1-liveness-visited-ucontext --setenv bindir=${CMAKE_BINARY_DIR}/examples/msg/mc --cd ${CMAKE_HOME_DIRECTORY}/examples/msg/mc bugged1_liveness_visited.tesh) ADD_TESH(mc-bugged1-liveness-visited-ucontext-sparse --setenv bindir=${CMAKE_BINARY_DIR}/examples/msg/mc --cd ${CMAKE_HOME_DIRECTORY}/examples/msg/mc bugged1_liveness_visited_sparse.tesh) IF(HAVE_C_STACK_CLEANER) # This test checks if the stack cleaner is making a difference: ADD_TEST(mc-bugged1-liveness-stack-cleaner ${CMAKE_HOME_DIRECTORY}/examples/msg/mc/bugged1_liveness_stack_cleaner ${CMAKE_HOME_DIRECTORY}/examples/msg/mc/ ${CMAKE_BINARY_DIR}/examples/msg/mc/) ENDIF() ENDIF() ENDIF() IF(enable_smpi_MPICH3_testsuite AND SMPI_FORTRAN AND HAVE_THREAD_CONTEXTS) ADD_TEST(test-smpi-mpich3-thread-f77 ${CMAKE_COMMAND} -E chdir ${CMAKE_BINARY_DIR}/teshsuite/smpi/mpich3-test/f77/ ${PERL_EXECUTABLE} ${CMAKE_HOME_DIRECTORY}/teshsuite/smpi/mpich3-test/runtests ${TESH_OPTION} -mpiexec=${CMAKE_BINARY_DIR}/smpi_script/bin/smpirun -srcdir=${CMAKE_HOME_DIRECTORY}/teshsuite/smpi/mpich3-test/f77/ -tests=testlist -privatization=${HAVE_PRIVATIZATION} -execarg=--cfg=contexts/stack_size:8000 -execarg=--cfg=contexts/factory:thread -execarg=--cfg=smpi/privatize-global-variables:${HAVE_PRIVATIZATION}) SET_TESTS_PROPERTIES(test-smpi-mpich3-thread-f77 PROPERTIES PASS_REGULAR_EXPRESSION "tests passed!") ADD_TEST(test-smpi-mpich3-thread-f90 ${CMAKE_COMMAND} -E chdir ${CMAKE_BINARY_DIR}/teshsuite/smpi/mpich3-test/f90/ ${PERL_EXECUTABLE} ${CMAKE_HOME_DIRECTORY}/teshsuite/smpi/mpich3-test/runtests ${TESH_OPTION} -mpiexec=${CMAKE_BINARY_DIR}/smpi_script/bin/smpirun -srcdir=${CMAKE_HOME_DIRECTORY}/teshsuite/smpi/mpich3-test/f90/ -tests=testlist -privatization=${HAVE_PRIVATIZATION} -execarg=--cfg=smpi/privatize-global-variables:${HAVE_PRIVATIZATION} -execarg=--cfg=contexts/factory:thread) SET_TESTS_PROPERTIES(test-smpi-mpich3-thread-f90 PROPERTIES PASS_REGULAR_EXPRESSION "tests passed!") ENDIF() IF(HAVE_LUA) # Tests testing simulation from C but using lua for platform files. Executed like this # ~$ ./masterslave platform.lua deploy.lua ADD_TESH(lua-platform-masterslave --setenv srcdir=${CMAKE_HOME_DIRECTORY}/teshsuite/lua --cd ${CMAKE_BINARY_DIR}/examples/lua ${CMAKE_HOME_DIRECTORY}/teshsuite/lua/lua_platforms.tesh) SET_TESTS_PROPERTIES(lua-platform-masterslave PROPERTIES ENVIRONMENT "LUA_CPATH=${CMAKE_BINARY_DIR}/examples/lua/?.so") ENDIF() ADD_TEST(testall ${CMAKE_BINARY_DIR}/testall) SimGrid-3.14.159/tools/cmake/Distrib.cmake000644 001750 001750 00000033337 13031031273 021724 0ustar00mquinsonmquinson000000 000000 ######################################### ### Fill in the "make install" target ### ######################################### # doc file(MAKE_DIRECTORY ${CMAKE_HOME_DIRECTORY}/doc/html/) install(DIRECTORY "${CMAKE_HOME_DIRECTORY}/doc/html/" DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/doc/simgrid/html/) # binaries if(enable_smpi) install(PROGRAMS ${CMAKE_BINARY_DIR}/bin/smpicc ${CMAKE_BINARY_DIR}/bin/smpicxx ${CMAKE_BINARY_DIR}/bin/smpirun DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/bin/) if(SMPI_FORTRAN) install(PROGRAMS ${CMAKE_BINARY_DIR}/bin/smpif90 ${CMAKE_BINARY_DIR}/bin/smpiff DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/bin/) endif() endif() install(PROGRAMS ${CMAKE_BINARY_DIR}/bin/tesh DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/bin/) install(PROGRAMS ${CMAKE_BINARY_DIR}/bin/graphicator DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/bin/) install(PROGRAMS ${CMAKE_HOME_DIRECTORY}/tools/MSG_visualization/colorize.pl DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/bin/ RENAME simgrid-colorizer) add_custom_target(simgrid-colorizer ALL COMMENT "Install ${CMAKE_BINARY_DIR}/bin/colorize" COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_HOME_DIRECTORY}/tools/MSG_visualization/colorize.pl ${CMAKE_BINARY_DIR}/bin/colorize) install(PROGRAMS ${CMAKE_HOME_DIRECTORY}/tools/simgrid_update_xml.pl DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/bin/ RENAME simgrid_update_xml) add_custom_target(simgrid_update_xml ALL COMMENT "Install ${CMAKE_BINARY_DIR}/bin/simgrid_update_xml" COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_HOME_DIRECTORY}/tools/simgrid_update_xml.pl ${CMAKE_BINARY_DIR}/bin/simgrid_update_xml) # libraries install(TARGETS simgrid DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/lib/) if(enable_java) set(SIMGRID_JAR_TO_INSTALL "${SIMGRID_JAR}") install(TARGETS simgrid-java DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/lib/) install(FILES ${SIMGRID_JAR_TO_INSTALL} DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/java/ RENAME simgrid.jar) endif() # include files set(HEADERS ${headers_to_install} ${generated_headers_to_install}) foreach(file ${HEADERS}) get_filename_component(location ${file} PATH) string(REPLACE "${CMAKE_CURRENT_BINARY_DIR}/" "" location "${location}") install(FILES ${file} DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/${location}) endforeach(file ${HEADERS}) # example files foreach(file ${examples_to_install}) string(REPLACE "${CMAKE_CURRENT_SOURCE_DIR}/examples/" "" file ${file}) get_filename_component(location ${file} PATH) install(FILES "examples/${file}" DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/doc/simgrid/examples/${location}) endforeach(file ${examples_to_install}) ########################################### ### Fill in the "make uninstall" target ### ########################################### add_custom_target(uninstall COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_INSTALL_PREFIX}/doc/simgrid COMMAND ${CMAKE_COMMAND} -E echo "uninstall doc ok" COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/lib/libsimgrid* COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/lib/lua/5.1/simgrid* COMMAND ${CMAKE_COMMAND} -E echo "uninstall lib ok" COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/bin/smpicc COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/bin/smpicxx COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/bin/smpiff COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/bin/smpif90 COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/bin/smpirun COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/bin/tesh COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/bin/simgrid-colorizer COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/bin/simgrid_update_xml COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/bin/graphicator COMMAND ${CMAKE_COMMAND} -E echo "uninstall bin ok" COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_INSTALL_PREFIX}/include/instr COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_INSTALL_PREFIX}/include/msg COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_INSTALL_PREFIX}/include/simdag COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_INSTALL_PREFIX}/include/smpi COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_INSTALL_PREFIX}/include/simix COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_INSTALL_PREFIX}/include/surf COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_INSTALL_PREFIX}/include/xbt COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_INSTALL_PREFIX}/include/mc COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_INSTALL_PREFIX}/include/simgrid COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/include/simgrid.h COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/include/simgrid_config.h COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/include/xbt.h COMMAND ${CMAKE_COMMAND} -E echo "uninstall include ok" COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/share/man/man1/simgrid_update_xml.1 COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/share/man/man1/tesh.1 COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/share/man/man1/smpicc.1 COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/share/man/man1/smpicxx.1 COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/share/man/man1/smpirun.1 COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/share/man/man1/smpiff.1 COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/share/man/man1/smpif90.1 COMMAND ${CMAKE_COMMAND} -E echo "uninstall man ok" WORKING_DIRECTORY "${CMAKE_INSTALL_PREFIX}") if(HAVE_LUA) add_custom_command(TARGET uninstall COMMAND ${CMAKE_COMMAND} -E echo "uninstall binding lua ok" COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_INSTALL_PREFIX}/lib/lua/5.1/simgrid.${LIB_EXE} WORKING_DIRECTORY "${CMAKE_HOME_DIRECTORY}/") endif() ################################################################ ## Build a sain "make dist" target to build a source package ### ## containing only the files that I explicitely state ### ## (instead of any cruft laying on my disk as CPack does) ### ################################################################ # This is the complete list of what will be added to the source archive set(source_to_pack ${headers_to_install} ${source_of_generated_headers} ${BINDINGS_SRC} ${JEDULE_SRC} ${JMSG_C_SRC} ${JMSG_JAVA_SRC} ${JSURF_SWIG_SRC} ${JSURF_SWIG_SRC_EXTRA} ${JSURF_C_SRC} ${LUA_SRC} ${MC_SRC_BASE} ${MC_SRC} ${MC_SIMGRID_MC_SRC} ${MSG_SRC} ${S4U_SRC} ${NS3_SRC} ${PLUGINS_SRC} ${RNGSTREAM_SRC} ${SIMDAG_SRC} ${SIMGRID_SRC} ${SIMIX_SRC} ${SMPI_SRC} ${SURF_SRC} ${TRACING_SRC} ${XBT_RL_SRC} ${XBT_SRC} ${EXTRA_DIST} ${CMAKE_SOURCE_FILES} ${CMAKEFILES_TXT} ${DOC_FIGS} ${DOC_IMG} ${DOC_SOURCES} ${DOC_TOOLS} ${PLATFORMS_EXAMPLES} ${README_files} ${bin_files} ${examples_src} ${tesh_files} ${teshsuite_src} ${testsuite_src} ${tools_src} ${txt_files} ${xml_files} ) ########################################## ### Fill in the "make dist-dir" target ### ########################################## add_custom_target(dist-dir COMMENT "Generating the distribution directory" COMMAND ${CMAKE_COMMAND} -E remove_directory ${PROJECT_NAME}-${release_version}/ COMMAND ${CMAKE_COMMAND} -E remove ${PROJECT_NAME}-${release_version}.tar.gz COMMAND ${CMAKE_COMMAND} -E make_directory ${PROJECT_NAME}-${release_version} COMMAND ${CMAKE_COMMAND} -E make_directory ${PROJECT_NAME}-${release_version}/doc/html/ COMMAND ${CMAKE_COMMAND} -E copy_directory ${CMAKE_HOME_DIRECTORY}/doc/html/ ${PROJECT_NAME}-${release_version}/doc/html/) add_dependencies(dist-dir maintainer_files) set(dirs_in_tarball "") foreach(file ${source_to_pack}) #message(${file}) # This damn prefix is still set somewhere (seems to be in subdirs) string(REPLACE "${CMAKE_HOME_DIRECTORY}/" "" file "${file}") # Create the directory on need get_filename_component(file_location ${file} PATH) string(REGEX MATCH ";${file_location};" OPERATION "${dirs_in_tarball}") if(NOT OPERATION) set(dirs_in_tarball "${dirs_in_tarball};${file_location};") add_custom_command( TARGET dist-dir COMMAND ${CMAKE_COMMAND} -E make_directory ${PROJECT_NAME}-${release_version}/${file_location}/) endif() # Actually copy the file add_custom_command( TARGET dist-dir COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_HOME_DIRECTORY}/${file} ${PROJECT_NAME}-${release_version}/${file_location}) endforeach(file ${source_to_pack}) add_custom_command( TARGET dist-dir COMMAND ${CMAKE_COMMAND} -E echo "${GIT_VERSION}" > ${PROJECT_NAME}-${release_version}/.gitversion) ###################################### ### Fill in the "make dist" target ### ###################################### add_custom_target(dist COMMENT "Removing the distribution directory" DEPENDS ${CMAKE_BINARY_DIR}/${PROJECT_NAME}-${release_version}.tar.gz COMMAND ${CMAKE_COMMAND} -E echo ${PROJECT_NAME}-${release_version} > ${CMAKE_BINARY_DIR}/VERSION COMMAND ${CMAKE_COMMAND} -E remove_directory ${PROJECT_NAME}-${release_version}/) add_custom_command( OUTPUT ${CMAKE_BINARY_DIR}/${PROJECT_NAME}-${release_version}.tar.gz COMMENT "Compressing the archive from the distribution directory" COMMAND ${CMAKE_COMMAND} -E tar cf ${PROJECT_NAME}-${release_version}.tar ${PROJECT_NAME}-${release_version}/ COMMAND gzip -9v ${PROJECT_NAME}-${release_version}.tar COMMAND ${CMAKE_COMMAND} -E remove_directory ${PROJECT_NAME}-${release_version}/) add_dependencies(dist dist-dir) if(NOT enable_maintainer_mode) add_custom_target(echo-dist COMMAND ${CMAKE_COMMAND} -E echo "WARNING: ----------------------------------------------------" COMMAND ${CMAKE_COMMAND} -E echo "WARNING: Distrib is generated without option maintainer mode " COMMAND ${CMAKE_COMMAND} -E echo "WARNING: ----------------------------------------------------") add_dependencies(dist echo-dist) endif() ########################################### ### Fill in the "make distcheck" target ### ########################################### set(CMAKE_BINARY_TEST_DIR ${CMAKE_BINARY_DIR}) # Allow to test the "make dist" add_custom_target(distcheck COMMAND ${CMAKE_COMMAND} -E echo "XXX compare archive with git repository" COMMAND ${CMAKE_HOME_DIRECTORY}/tools/internal/check_dist_archive -batch ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}.tar.gz COMMAND ${CMAKE_COMMAND} -E echo "XXX remove old copy" COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version} COMMAND ${CMAKE_COMMAND} -E echo "XXX Untar distrib" COMMAND ${CMAKE_COMMAND} -E tar xf ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}.tar.gz ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version} COMMAND ${CMAKE_COMMAND} -E echo "XXX create build and install subtrees" COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}/_build COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}/_inst COMMAND ${CMAKE_COMMAND} -E echo "XXX Configure" COMMAND ${CMAKE_COMMAND} -E chdir ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}/_build ${CMAKE_COMMAND} -DCMAKE_INSTALL_PREFIX=${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}/_inst -Denable_lto=OFF .. COMMAND ${CMAKE_COMMAND} -E echo "XXX Build" COMMAND ${CMAKE_COMMAND} -E chdir ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}/_build ${CMAKE_MAKE_PROGRAM} -j 4 COMMAND ${CMAKE_COMMAND} -E echo "XXX Test" COMMAND ${CMAKE_COMMAND} -E chdir ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}/_build ctest --output-on-failure -j 4 COMMAND ${CMAKE_COMMAND} -E echo "XXX Install" COMMAND ${CMAKE_COMMAND} -E chdir ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}/_build ${CMAKE_MAKE_PROGRAM} install COMMAND ${CMAKE_COMMAND} -E create_symlink ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}/_inst/lib/libsimgrid.so ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}/_inst/lib/libsimgridtest.so COMMAND ${CMAKE_COMMAND} -E echo "XXX Install with documentation" COMMAND ${CMAKE_COMMAND} -E chdir ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version}/_build ${CMAKE_MAKE_PROGRAM} install COMMAND ${CMAKE_COMMAND} -E echo "XXX Remove temp directories" COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_BINARY_TEST_DIR}/${PROJECT_NAME}-${release_version} ) add_dependencies(distcheck dist) ####################################### ### Fill in the "make check" target ### ####################################### if(enable_memcheck) add_custom_target(check COMMAND ctest -D ExperimentalMemCheck) else() add_custom_target(check COMMAND make test) endif() ####################################### ### Fill in the "make xxx-clean" target ### ####################################### add_custom_target(maintainer-clean COMMAND ${CMAKE_COMMAND} -E remove -f src/config_unit.c COMMAND ${CMAKE_COMMAND} -E remove -f src/cunit_unit.c COMMAND ${CMAKE_COMMAND} -E remove -f src/dict_unit.c COMMAND ${CMAKE_COMMAND} -E remove -f src/dynar_unit.c COMMAND ${CMAKE_COMMAND} -E remove -f src/ex_unit.c COMMAND ${CMAKE_COMMAND} -E remove -f src/set_unit.c COMMAND ${CMAKE_COMMAND} -E remove -f src/simgrid_units_main.c COMMAND ${CMAKE_COMMAND} -E remove -f src/swag_unit.c COMMAND ${CMAKE_COMMAND} -E remove -f src/xbt_str_unit.c COMMAND ${CMAKE_COMMAND} -E remove -f src/xbt_strbuff_unit.c COMMAND ${CMAKE_COMMAND} -E remove -f src/xbt_synchro_unit.c WORKING_DIRECTORY "${CMAKE_HOME_DIRECTORY}") include(CPack) SimGrid-3.14.159/tools/cmake/GCCFlags.cmake000644 001750 001750 00000022577 13031031273 021701 0ustar00mquinsonmquinson000000 000000 ## ## This file is in charge of setting our paranoid flags with regard to warnings and optimization. ## ## It is only used for gcc and clang. MSVC builds don't load this file. ## ## These flags do break some classical CMake tests, so you don't ## want to do so before the very end of the configuration. ## ## Other compiler flags (C/C++ standard version) are tested and set ## by the beginning of the configuration, directly in ~/CMakeList.txt set(warnCFLAGS "") set(optCFLAGS "") set(warnCXXFLAGS "") if(enable_compile_warnings) set(warnCFLAGS "-fno-common -Wall -Wunused -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wcomment -Wformat -Wwrite-strings -Wno-unused-function -Wno-unused-parameter -Wno-strict-aliasing -Wno-format-nonliteral") if(CMAKE_COMPILER_IS_GNUCC) set(warnCFLAGS "${warnCFLAGS} -Wclobbered -Wno-error=clobbered -Wno-unused-local-typedefs -Wno-error=attributes") endif() set(warnCXXFLAGS "${warnCFLAGS} -Wall -Wextra -Wunused -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wcomment -Wformat -Wwrite-strings -Wno-unused-function -Wno-unused-parameter -Wno-strict-aliasing -Wno-format-nonliteral") if(CMAKE_COMPILER_IS_GNUCXX) set(warnCXXFLAGS "${warnCXXFLAGS} -Wclobbered -Wno-error=clobbered -Wno-unused-local-typedefs -Wno-error=attributes") endif() if (CMAKE_CXX_COMPILER_ID MATCHES "Clang") # don't care about class that become struct, avoid issue of empty C structs # size (coming from libunwind.h) set(warnCXXFLAGS "${warnCXXFLAGS} -Wno-mismatched-tags -Wno-extern-c-compat") endif() # the one specific to C but refused by C++ set(warnCFLAGS "${warnCFLAGS} -Wmissing-prototypes") set(CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} -Wall") set(CMAKE_JAVA_COMPILE_FLAGS "-Xlint") endif() # NDEBUG gives a lot of "initialized but unused variables" errors. Don't die anyway. if(enable_compile_warnings AND enable_debug) set(warnCFLAGS "${warnCFLAGS} -Werror") set(warnCXXFLAGS "${warnCXXFLAGS} -Werror") endif() # Activate the warnings on #if FOOBAR when FOOBAR has no value # It breaks on FreeBSD within Boost headers, so activate this only in Pure Hardcore debug mode. if(enable_maintainer_mode) set(warnCFLAGS "${warnCFLAGS} -Wundef") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wundef") endif() # Se the optimisation flags # NOTE, we should CMAKE_BUILD_TYPE for this if(enable_compile_optimizations) set(optCFLAGS "-O3 -funroll-loops -fno-strict-aliasing ") else() set(optCFLAGS "-O0 ") endif() if(enable_compile_optimizations AND CMAKE_COMPILER_IS_GNUCC AND (NOT enable_model-checking)) # This is redundant (already in -03): set(optCFLAGS "${optCFLAGS} -finline-functions ") endif() # Configure LTO # NOTE, cmake 3.0 has a INTERPROCEDURAL_OPTIMIZATION target # property for this (http://www.cmake.org/cmake/help/v3.0/prop_tgt/INTERPROCEDURAL_OPTIMIZATION.html) if(enable_lto) # User wants LTO. Try if we can do that set(enable_lto OFF) if(enable_compile_optimizations AND CMAKE_COMPILER_IS_GNUCC AND (NOT enable_model-checking)) # On windows, we need 4.8 or higher to enable lto because of http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50293 # We are experiencing assertion failures even with 4.8 on MinGW. # Push the support forward: will see if 4.9 works when we test it. # # On Linux, we got the following with GCC 4.8.4 on Centos and Ubuntu # lto1: internal compiler error: in output_die, at dwarf2out.c:8478 # Please submit a full bug report, with preprocessed source if appropriate. # So instead, we push the support forward if ( (CMAKE_C_COMPILER_VERSION VERSION_GREATER "4.8.5") AND (LINKER_VERSION VERSION_GREATER "2.22")) set(enable_lto ON) endif() endif() if(enable_lto) message("-- LTO seems usable.") else() if(NOT enable_compile_optimizations) message("-- LTO disabled: Compile-time optimizations turned off.") else() if(enable_model-checking) message("-- LTO disabled when compiling with model-checking.") else() message("-- LTO does not seem usable -- try updating your build chain.") endif() endif() endif() else() message("-- LTO disabled on the command line.") endif() if(enable_lto) # User wants LTO, and it seems usable. Go for it set(optCFLAGS "${optCFLAGS} -flto ") # See https://gcc.gnu.org/wiki/LinkTimeOptimizationFAQ#ar.2C_nm_and_ranlib: # "Since version 4.9 gcc produces slim object files that only contain # the intermediate representation. In order to handle archives of # these objects you have to use the gcc wrappers: # gcc-ar, gcc-nm and gcc-ranlib." if(${CMAKE_C_COMPILER_ID} STREQUAL "GNU" AND CMAKE_C_COMPILER_VERSION VERSION_GREATER "4.8") set (CMAKE_AR gcc-ar) set (CMAKE_RANLIB gcc-ranlib) endif() endif() if(enable_model-checking AND enable_compile_optimizations) # Forget it, do not optimize the code (because it confuses the MC): set(optCFLAGS "-O0 ") # But you can still optimize this: foreach(s src/xbt/mmalloc/mm.c src/xbt/log.c src/xbt/xbt_log_appender_file.c src/xbt/xbt_log_layout_format.c src/xbt/xbt_log_layout_simple.c src/xbt/dict.cpp src/xbt/dict_elm.c src/xbt/dict_cursor.c src/xbt/dynar.cpp src/xbt/fifo.c src/xbt/heap.c src/xbt/swag.c src/xbt/str.c src/xbt/strbuff.c src/xbt/snprintf.c src/xbt/queue.c src/xbt/xbt_os_time.c src/xbt/xbt_os_thread.c src/xbt/backtrace_linux.c ${MC_SRC_BASE} ${MC_SRC}) set (mcCFLAGS "-O3 -funroll-loops -fno-strict-aliasing") if(CMAKE_COMPILER_IS_GNUCC) set (mcCFLAGS "${mcCFLAGS} -finline-functions") endif() set_source_files_properties(${s} PROPERTIES COMPILE_FLAGS ${mcCFLAGS}) endforeach() endif() if(NOT enable_debug) set(CMAKE_C_FLAGS "-DNDEBUG ${CMAKE_C_FLAGS}") set(CMAKE_CXX_FLAGS "-DNDEBUG ${CMAKE_CXX_FLAGS}") endif() set(CMAKE_C_FLAGS "${warnCFLAGS} ${CMAKE_C_FLAGS} ${optCFLAGS}") set(CMAKE_CXX_FLAGS "${warnCXXFLAGS} ${CMAKE_CXX_FLAGS} ${optCFLAGS}") # Try to make Mac a bit more complient to open source standards if(CMAKE_SYSTEM_NAME MATCHES "Darwin") set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -D_XOPEN_SOURCE=700 -D_DARWIN_C_SOURCE") endif() set(TESH_OPTION "") if(enable_coverage) find_program(GCOV_PATH gcov) if(GCOV_PATH) set(COVERAGE_COMMAND "${GCOV_PATH}" CACHE TYPE FILEPATH FORCE) set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DCOVERAGE") set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fprofile-arcs -ftest-coverage") set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fprofile-arcs -ftest-coverage") set(CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} -fprofile-arcs -ftest-coverage") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fprofile-arcs -ftest-coverage") set(TESH_OPTION --enable-coverage) add_definitions(-fprofile-arcs -ftest-coverage) endif() endif() if(enable_address_sanitizer) set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fsanitize=address -fno-omit-frame-pointer") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fsanitize=address -fno-omit-frame-pointer") set(CMAKE_C_LINK_FLAGS "${CMAKE_C_LINK_FLAGS} -fsanitize=address") set(TESH_OPTION --enable-sanitizers) endif() if(enable_thread_sanitizer) set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fsanitize=thread -fno-omit-frame-pointer -no-pie") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fsanitize=thread -fno-omit-frame-pointer -no-pie") set(CMAKE_C_LINK_FLAGS "${CMAKE_C_LINK_FLAGS} -fsanitize=thread -no-pie") endif() if(enable_undefined_sanitizer) set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fsanitize=undefined -fno-omit-frame-pointer") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fsanitize=undefined -fno-omit-frame-pointer") set(CMAKE_C_LINK_FLAGS "${CMAKE_C_LINK_FLAGS} -fsanitize=undefined") endif() if(NOT $ENV{CFLAGS} STREQUAL "") message(STATUS "Add CFLAGS: \"$ENV{CFLAGS}\" to CMAKE_C_FLAGS") set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} $ENV{CFLAGS}") endif() if(NOT $ENV{CXXFLAGS} STREQUAL "") message(STATUS "Add CXXFLAGS: \"$ENV{CXXFLAGS}\" to CMAKE_CXX_FLAGS") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} $ENV{CXXFLAGS}") endif() if(NOT $ENV{LDFLAGS} STREQUAL "") message(STATUS "Add LDFLAGS: \"$ENV{LDFLAGS}\" to CMAKE_C_LINK_FLAGS") set(CMAKE_C_LINK_FLAGS "${CMAKE_C_LINK_FLAGS} $ENV{LDFLAGS}") endif() if(MINGW) # http://stackoverflow.com/questions/10452262/create-64-bit-jni-under-windows # We don't want to ship libgcc_s_seh-1.dll nor libstdc++-6.dll set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -static-libgcc") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -static-libgcc -static-libstdc++") set(CMAKE_SHARED_LIBRARY_LINK_C_FLAGS "${CMAKE_SHARED_LIBRARY_LINK_C_FLAGS} -static-libgcc") set(CMAKE_SHARED_LIBRARY_LINK_CXX_FLAGS "${CMAKE_SHARED_LIBRARY_LINK_CXX_FLAGS} -static-libgcc -static-libstdc++") # JNI searches for stdcalls set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wl,--add-stdcall-alias") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wl,--add-stdcall-alias") set(CMAKE_SHARED_LIBRARY_LINK_C_FLAGS "${CMAKE_SHARED_LIBRARY_LINK_C_FLAGS} -Wl,--add-stdcall-alias") set(CMAKE_SHARED_LIBRARY_LINK_CXX_FLAGS "${CMAKE_SHARED_LIBRARY_LINK_CXX_FLAGS} -Wl,--add-stdcall-alias") # Specify the data model that we are using (yeah it may help Java) if(CMAKE_SIZEOF_VOID_P EQUAL 4) # 32 bits set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -m32") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -m32") else() set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -m64") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -m64") endif() endif() SimGrid-3.14.159/tools/cmake/CTestConfig.cmake000644 001750 001750 00000003074 13031031273 022467 0ustar00mquinsonmquinson000000 000000 # Configure CTest. For details, see: # http://www.cmake.org/Wiki/CMake_Testing_With_CTest#Customizing_CTest if(APPLE) SET(BUILDNAME "APPLE" CACHE INTERNAL "Buildname" FORCE) else() SET(BUILDNAME "UNIX" CACHE INTERNAL "Buildname" FORCE) if(WIN32) SET(BUILDNAME "WINDOWS" CACHE INTERNAL "Buildname" FORCE) endif() endif() if(NOT enable_memcheck) set(DART_TESTING_TIMEOUT "300") #TIMEOUT FOR EACH TEST else() set(DART_TESTING_TIMEOUT "3000") #TIMEOUT FOR EACH TEST endif() if(enable_compile_warnings AND enable_compile_optimizations) SET(BUILDNAME "FULL_FLAGS" CACHE INTERNAL "Buildname" FORCE) endif() if(HAVE_MC) SET(BUILDNAME "MODEL-CHECKING" CACHE INTERNAL "Buildname" FORCE) endif() if(enable_memcheck) SET(BUILDNAME "MEMCHECK" CACHE INTERNAL "Buildname" FORCE) endif() set(osname ${CMAKE_SYSTEM_NAME}) set(cpu ${CMAKE_SYSTEM_PROCESSOR}) set(DISTRIB2 ${CMAKE_SYSTEM_VERSION}) SET(SITE "${osname}_${DISTRIB2}_${cpu}") SET(CTEST_SITE "${osname}_${DISTRIB2}_${cpu}") SET(CTEST_PROJECT_NAME "${PROJECT_NAME}") SET(CTEST_CUSTOM_MAXIMUM_FAILED_TEST_OUTPUT_SIZE "3000000") SET(CTEST_CUSTOM_MAXIMUM_PASSED_TEST_OUTPUT_SIZE "3000000") set(PATTERN_CTEST_IGNORED "") if(enable_coverage) set(PATTERN_CTEST_IGNORED "/tools/" "/buildtools/" "/include/" "/teshsuite/" "/src/bindings/" ) if(NOT release) set(PATTERN_CTEST_IGNORED ${PATTERN_CTEST_IGNORED} "/examples/" ) endif() endif() CONFIGURE_FILE(${CMAKE_HOME_DIRECTORY}/tools/cmake/CTestCustom.cmake ${CMAKE_BINARY_DIR}/CTestCustom.cmake @ONLY) SimGrid-3.14.159/tools/cmake/DefinePackages.cmake000644 001750 001750 00000102021 13031031273 023140 0ustar00mquinsonmquinson000000 000000 ### define source packages set(EXTRA_DIST src/bindings/java/MANIFEST.in src/include/instr/instr_interface.h src/include/mc/datatypes.h src/include/mc/mc.h src/mc/mc_mmu.h src/mc/PageStore.hpp src/mc/mc_record.h src/include/simgrid/sg_config.h src/include/smpi/smpi_interface.h src/include/smpi/smpi_utils.hpp src/include/surf/datatypes.h src/include/surf/maxmin.h src/include/surf/surf.h src/msg/msg_private.h src/simdag/dax.dtd src/simdag/dax_dtd.c src/simdag/dax_dtd.h src/simdag/simdag_private.hpp src/simix/simcalls.in src/simix/simcalls.py src/simix/popping_private.h src/simix/popping_bodies.cpp src/simix/popping_generated.cpp src/simix/popping_enum.h src/simix/popping_accessors.h src/simix/smx_host_private.h src/simix/smx_io_private.h src/simix/smx_network_private.h src/simix/smx_private.h src/simix/ActorImpl.hpp src/simix/smx_synchro_private.h src/kernel/activity/ActivityImpl.hpp src/kernel/activity/SynchroComm.hpp src/kernel/activity/SynchroExec.hpp src/kernel/activity/SynchroIo.hpp src/kernel/activity/SynchroSleep.hpp src/kernel/activity/SynchroRaw.hpp src/smpi/colls/coll_tuned_topo.h src/smpi/colls/colls.h src/smpi/colls/colls_private.h src/smpi/colls/smpi_mvapich2_selector_stampede.h src/smpi/private.h src/smpi/private.hpp src/smpi/smpi_mpi_dt_private.h src/surf/cpu_cas01.hpp src/surf/cpu_interface.hpp src/surf/cpu_ti.hpp src/surf/maxmin_private.hpp src/surf/network_cm02.hpp src/surf/network_constant.hpp src/surf/network_interface.hpp src/surf/network_ns3.hpp src/surf/network_smpi.hpp src/surf/network_ib.hpp src/surf/ns3/ns3_interface.h src/surf/ns3/ns3_simulator.h src/surf/plugins/energy.hpp src/surf/xml/simgrid.dtd src/surf/xml/simgrid_dtd.h src/surf/xml/simgrid_dtd.c src/surf/xml/surfxml_sax_cb.cpp src/surf/storage_interface.hpp src/surf/storage_n11.hpp src/surf/surf_interface.hpp src/surf/surf_private.h src/surf/PropertyHolder.hpp src/surf/host_clm03.hpp src/surf/HostImpl.hpp src/surf/ptask_L07.hpp src/xbt/automaton/automaton_lexer.yy.c src/xbt/automaton/parserPromela.lex src/xbt/automaton/parserPromela.tab.cacc src/xbt/automaton/parserPromela.tab.hacc src/xbt/automaton/parserPromela.yacc src/xbt/backtrace_dummy.cpp src/xbt/backtrace_linux.cpp src/xbt/dict_private.h src/xbt/ex_interface.h src/xbt/fifo_private.h src/xbt/graph_private.h src/xbt/heap_private.h src/xbt/log_private.h src/xbt/mallocator_private.h src/xbt/mmalloc/mfree.c src/xbt/mmalloc/mm.c src/xbt/mmalloc/mm_legacy.c src/xbt/mmalloc/mm_module.c src/xbt/mmalloc/mmalloc.c src/xbt/mmalloc/mmalloc.info src/xbt/mmalloc/mmalloc.texi src/xbt/mmalloc/mmorecore.c src/xbt/mmalloc/mmprivate.h src/xbt/mmalloc/mrealloc.c tools/tesh/generate_tesh tools/lualib.patch teshsuite/lua/lua_platforms.tesh examples/smpi/mc/only_send_deterministic.tesh examples/smpi/mc/non_deterministic.tesh examples/java/.classpath examples/java/.project ) set(SMPI_SRC src/smpi/colls/allgather-2dmesh.c src/smpi/colls/allgather-3dmesh.c src/smpi/colls/allgather-GB.c src/smpi/colls/allgather-NTSLR-NB.c src/smpi/colls/allgather-NTSLR.c src/smpi/colls/allgather-SMP-NTS.c src/smpi/colls/allgather-bruck.c src/smpi/colls/allgather-loosely-lr.c src/smpi/colls/allgather-ompi-neighborexchange.c src/smpi/colls/allgather-pair.c src/smpi/colls/allgather-mvapich-smp.c src/smpi/colls/allgather-rdb.c src/smpi/colls/allgather-rhv.c src/smpi/colls/allgather-ring.c src/smpi/colls/allgather-smp-simple.c src/smpi/colls/allgather-spreading-simple.c src/smpi/colls/allgatherv-GB.c src/smpi/colls/allgatherv-mpich-rdb.c src/smpi/colls/allgatherv-mpich-ring.c src/smpi/colls/allgatherv-ompi-bruck.c src/smpi/colls/allgatherv-ompi-neighborexchange.c src/smpi/colls/allgatherv-pair.c src/smpi/colls/allgatherv-ring.c src/smpi/colls/allreduce-lr.c src/smpi/colls/allreduce-ompi-ring-segmented.c src/smpi/colls/allreduce-rab-rdb.c src/smpi/colls/allreduce-rab1.c src/smpi/colls/allreduce-rab2.c src/smpi/colls/allreduce-rdb.c src/smpi/colls/allreduce-redbcast.c src/smpi/colls/allreduce-smp-binomial-pipeline.c src/smpi/colls/allreduce-smp-binomial.c src/smpi/colls/allreduce-smp-rdb.c src/smpi/colls/allreduce-smp-rsag-lr.c src/smpi/colls/allreduce-smp-rsag-rab.c src/smpi/colls/allreduce-smp-rsag.c src/smpi/colls/allreduce-mvapich-rs.c src/smpi/colls/allreduce-mvapich-two-level.c src/smpi/colls/alltoall-2dmesh.c src/smpi/colls/alltoall-3dmesh.c # src/smpi/colls/alltoall-bruck.c src/smpi/colls/alltoall-pair-light-barrier.c src/smpi/colls/alltoall-pair-mpi-barrier.c src/smpi/colls/alltoall-pair-one-barrier.c src/smpi/colls/alltoall-pair.c src/smpi/colls/alltoall-rdb.c src/smpi/colls/alltoall-ring-light-barrier.c src/smpi/colls/alltoall-ring-mpi-barrier.c src/smpi/colls/alltoall-ring-one-barrier.c src/smpi/colls/alltoall-ring.c src/smpi/colls/alltoall-mvapich-scatter-dest.c src/smpi/colls/alltoallv-bruck.c src/smpi/colls/alltoallv-ompi-basic-linear.c src/smpi/colls/alltoallv-pair-light-barrier.c src/smpi/colls/alltoallv-pair-mpi-barrier.c src/smpi/colls/alltoallv-pair-one-barrier.c src/smpi/colls/alltoallv-pair.c src/smpi/colls/alltoallv-ring-light-barrier.c src/smpi/colls/alltoallv-ring-mpi-barrier.c src/smpi/colls/alltoallv-ring-one-barrier.c src/smpi/colls/alltoallv-ring.c src/smpi/colls/barrier-ompi.c src/smpi/colls/barrier-mvapich2-pair.c src/smpi/colls/bcast-NTSB.c src/smpi/colls/bcast-NTSL-Isend.c src/smpi/colls/bcast-NTSL.c src/smpi/colls/bcast-SMP-binary.c src/smpi/colls/bcast-SMP-binomial.c src/smpi/colls/bcast-SMP-linear.c src/smpi/colls/bcast-arrival-pattern-aware-wait.c src/smpi/colls/bcast-arrival-pattern-aware.c src/smpi/colls/bcast-arrival-scatter.c src/smpi/colls/bcast-binomial-tree.c src/smpi/colls/bcast-flattree-pipeline.c src/smpi/colls/bcast-flattree.c src/smpi/colls/bcast-ompi-pipeline.c src/smpi/colls/bcast-ompi-split-bintree.c src/smpi/colls/bcast-mvapich-smp.c src/smpi/colls/bcast-scatter-LR-allgather.c src/smpi/colls/bcast-scatter-rdb-allgather.c src/smpi/colls/coll_tuned_topo.c src/smpi/colls/colls_global.c src/smpi/colls/gather-ompi.c src/smpi/colls/gather-mvapich.c src/smpi/colls/reduce-NTSL.c src/smpi/colls/reduce-arrival-pattern-aware.c src/smpi/colls/reduce-binomial.c src/smpi/colls/reduce-flat-tree.c src/smpi/colls/reduce-ompi.c src/smpi/colls/reduce-scatter-gather.c src/smpi/colls/reduce_scatter-mpich.c src/smpi/colls/reduce_scatter-ompi.c src/smpi/colls/reduce-mvapich-knomial.c src/smpi/colls/reduce-mvapich-two-level.c src/smpi/colls/reduce-rab.c src/smpi/colls/scatter-ompi.c src/smpi/colls/scatter-mvapich-two-level.c src/smpi/colls/smpi_automatic_selector.cpp src/smpi/colls/smpi_mpich_selector.c src/smpi/colls/smpi_intel_mpi_selector.c src/smpi/colls/smpi_openmpi_selector.c src/smpi/colls/smpi_mvapich2_selector.c src/smpi/instr_smpi.cpp src/smpi/smpi_base.cpp src/smpi/smpi_bench.cpp src/smpi/smpi_memory.cpp src/smpi/smpi_static_variables.cpp src/smpi/smpi_coll.cpp src/smpi/smpi_comm.cpp src/smpi/smpi_deployment.cpp src/smpi/smpi_dvfs.cpp src/smpi/smpi_global.cpp src/smpi/smpi_group.cpp src/smpi/smpi_mpi.cpp src/smpi/smpi_mpi_dt.cpp src/smpi/smpi_pmpi.cpp src/smpi/smpi_replay.cpp src/smpi/smpi_rma.cpp src/smpi/smpi_topo.cpp src/smpi/smpi_utils.cpp src/smpi/smpi_f77.cpp src/surf/network_smpi.cpp src/surf/network_ib.cpp ) set(XBT_SRC src/xbt/RngStream.c src/xbt/automaton/automaton.c src/xbt/automaton/automatonparse_promela.c src/xbt/backtrace.cpp src/xbt/config.cpp src/xbt/cunit.cpp src/xbt/dict.cpp src/xbt/dict_cursor.c src/xbt/dict_elm.c src/xbt/dynar.cpp src/xbt/ex.cpp src/xbt/exception.cpp src/xbt/fifo.c src/xbt/graph.c src/xbt/heap.c src/xbt/lib.c src/xbt/log.c src/xbt/mallocator.c src/xbt/memory_map.cpp src/xbt/memory_map.hpp src/xbt/parmap.cpp src/xbt/snprintf.c src/xbt/string.cpp src/xbt/swag.c src/xbt/xbt_log_appender_file.c src/xbt/xbt_log_layout_format.c src/xbt/xbt_log_layout_simple.c src/xbt/xbt_main.cpp src/xbt/xbt_matrix.c src/xbt/xbt_os_file.c src/xbt/xbt_os_synchro.c src/xbt/xbt_os_time.c src/xbt/xbt_replay.cpp src/xbt/xbt_str.cpp src/xbt/xbt_strbuff.c src/xbt/xbt_virtu.c src/xbt_modinter.h ) if(HAVE_MMALLOC) set(XBT_SRC ${XBT_SRC} src/xbt/mmalloc/mm.c ) endif() set(NS3_SRC src/surf/network_ns3.cpp src/surf/ns3/ns3_simulator.cc ) set(SURF_SRC src/kernel/routing/ClusterZone.cpp src/kernel/routing/ClusterZone.hpp src/kernel/routing/DijkstraZone.cpp src/kernel/routing/DijkstraZone.hpp src/kernel/routing/DragonflyZone.cpp src/kernel/routing/DragonflyZone.hpp src/kernel/routing/EmptyZone.cpp src/kernel/routing/EmptyZone.hpp src/kernel/routing/FatTreeZone.cpp src/kernel/routing/FatTreeZone.hpp src/kernel/routing/FloydZone.cpp src/kernel/routing/FloydZone.hpp src/kernel/routing/FullZone.cpp src/kernel/routing/FullZone.hpp src/kernel/routing/NetCard.cpp src/kernel/routing/NetCard.hpp src/kernel/routing/NetZoneImpl.cpp src/kernel/routing/NetZoneImpl.hpp src/kernel/routing/RoutedZone.cpp src/kernel/routing/RoutedZone.hpp src/kernel/routing/TorusZone.cpp src/kernel/routing/TorusZone.hpp src/kernel/routing/VivaldiZone.cpp src/kernel/routing/VivaldiZone.hpp src/kernel/EngineImpl.cpp src/kernel/EngineImpl.hpp src/surf/cpu_cas01.cpp src/surf/cpu_interface.cpp src/surf/cpu_ti.cpp src/surf/fair_bottleneck.cpp src/surf/instr_routing.cpp src/surf/instr_surf.cpp src/surf/lagrange.cpp src/surf/maxmin.cpp src/surf/network_cm02.cpp src/surf/network_constant.cpp src/surf/network_interface.cpp src/surf/plugins/energy.cpp src/surf/PropertyHolder.cpp src/surf/sg_platf.cpp src/surf/storage_interface.cpp src/surf/storage_n11.cpp src/surf/surf_c_bindings.cpp src/surf/surf_interface.cpp src/surf/surf_routing.cpp src/surf/xml/platf.hpp src/surf/xml/platf_private.hpp src/surf/xml/surfxml_sax_cb.cpp src/surf/xml/surfxml_parseplatf.cpp src/surf/trace_mgr.hpp src/surf/trace_mgr.cpp src/surf/host_clm03.cpp src/surf/HostImpl.cpp src/surf/ptask_L07.cpp ) set(PLUGINS_SRC src/plugins/vm/s4u_VirtualMachine.cpp src/plugins/vm/VirtualMachineImpl.hpp src/plugins/vm/VirtualMachineImpl.cpp src/plugins/vm/VmHostExt.hpp src/plugins/vm/VmHostExt.cpp ) set(SIMIX_GENERATED_SRC src/simix/popping_generated.cpp ) set(SIMIX_SRC src/kernel/future.cpp src/simix/libsmx.cpp src/simix/smx_context.cpp src/kernel/context/Context.cpp src/kernel/context/Context.hpp src/kernel/context/ContextRaw.cpp src/simix/smx_deployment.cpp src/simix/smx_environment.cpp src/simix/smx_global.cpp src/simix/smx_host.cpp src/simix/smx_io.cpp src/simix/smx_network.cpp src/simix/ActorImpl.cpp src/simix/smx_synchro.cpp src/simix/popping.cpp src/kernel/activity/ActivityImpl.cpp src/kernel/activity/SynchroComm.cpp src/kernel/activity/SynchroExec.cpp src/kernel/activity/SynchroSleep.cpp src/kernel/activity/SynchroRaw.cpp src/kernel/activity/SynchroIo.cpp ${SIMIX_GENERATED_SRC} ) # Boost context may not be available if (HAVE_BOOST_CONTEXTS) set(SIMIX_SRC ${SIMIX_SRC} src/kernel/context/ContextBoost.hpp src/kernel/context/ContextBoost.cpp) else() set(EXTRA_DIST ${EXTRA_DIST} src/kernel/context/ContextBoost.hpp src/kernel/context/ContextBoost.cpp) endif() set(S4U_SRC src/s4u/s4u_actor.cpp src/s4u/s4u_activity.cpp src/s4u/s4u_conditionVariable.cpp src/s4u/s4u_comm.cpp src/s4u/s4u_engine.cpp src/s4u/s4u_file.cpp src/s4u/s4u_host.cpp src/s4u/s4u_mailbox.cpp src/s4u/s4u_mutex.cpp src/s4u/s4u_netzone.cpp src/s4u/s4u_storage.cpp ) set(SIMGRID_SRC src/simgrid/sg_config.cpp src/simgrid/host.cpp src/simgrid/util.hpp ) set(MSG_SRC src/msg/instr_msg_process.cpp src/msg/instr_msg_task.cpp src/msg/msg_actions.cpp src/msg/msg_deployment.cpp src/msg/msg_environment.cpp src/msg/msg_global.cpp src/msg/msg_gos.cpp src/msg/msg_host.cpp src/msg/msg_io.cpp src/msg/msg_mailbox.cpp src/msg/msg_process.cpp src/msg/msg_synchro.cpp src/msg/msg_task.cpp src/msg/msg_vm.cpp ) set(SIMDAG_SRC src/simdag/sd_daxloader.cpp src/simdag/sd_dotloader.cpp src/simdag/sd_global.cpp src/simdag/sd_task.cpp src/simdag/sd_workstation.cpp ) set(BINDINGS_SRC src/bindings/lua/lua_private.h src/bindings/lua/lua_utils.h src/bindings/lua/simgrid_lua.h ) set(JMSG_C_SRC src/bindings/java/jmsg.cpp src/bindings/java/jmsg.h src/bindings/java/jmsg_as.cpp src/bindings/java/jmsg_as.h src/bindings/java/jmsg_comm.cpp src/bindings/java/jmsg_comm.h src/bindings/java/jmsg_file.cpp src/bindings/java/jmsg_file.h src/bindings/java/jmsg_host.cpp src/bindings/java/jmsg_host.h src/bindings/java/jmsg_process.cpp src/bindings/java/jmsg_process.h src/bindings/java/jmsg_rngstream.cpp src/bindings/java/jmsg_rngstream.h src/bindings/java/jmsg_synchro.cpp src/bindings/java/jmsg_synchro.h src/bindings/java/jmsg_task.cpp src/bindings/java/jmsg_task.h src/bindings/java/jmsg_vm.cpp src/bindings/java/jmsg_vm.h src/bindings/java/jxbt_utilities.cpp src/bindings/java/jxbt_utilities.h src/bindings/java/JavaContext.cpp src/bindings/java/JavaContext.hpp src/bindings/java/jmsg_storage.cpp src/bindings/java/jmsg_storage.h ) set(JMSG_JAVA_SRC src/bindings/java/org/simgrid/NativeLib.java src/bindings/java/org/simgrid/msg/As.java src/bindings/java/org/simgrid/msg/Comm.java src/bindings/java/org/simgrid/msg/File.java src/bindings/java/org/simgrid/msg/Host.java src/bindings/java/org/simgrid/msg/HostFailureException.java src/bindings/java/org/simgrid/msg/HostNotFoundException.java src/bindings/java/org/simgrid/msg/JniException.java src/bindings/java/org/simgrid/msg/Msg.java src/bindings/java/org/simgrid/msg/MsgException.java src/bindings/java/org/simgrid/msg/Mutex.java src/bindings/java/org/simgrid/msg/NativeException.java src/bindings/java/org/simgrid/msg/Process.java src/bindings/java/org/simgrid/msg/ProcessKilledError.java src/bindings/java/org/simgrid/msg/ProcessNotFoundException.java src/bindings/java/org/simgrid/msg/RngStream.java src/bindings/java/org/simgrid/msg/Semaphore.java src/bindings/java/org/simgrid/msg/Storage.java src/bindings/java/org/simgrid/msg/StorageNotFoundException.java src/bindings/java/org/simgrid/msg/Task.java src/bindings/java/org/simgrid/msg/TaskCancelledException.java src/bindings/java/org/simgrid/msg/TimeoutException.java src/bindings/java/org/simgrid/msg/TransferFailureException.java src/bindings/java/org/simgrid/msg/VM.java ) set(JTRACE_C_SRC src/bindings/java/jtrace.cpp src/bindings/java/jtrace.h ) set(JTRACE_JAVA_SRC src/bindings/java/org/simgrid/trace/Trace.java) list(APPEND JMSG_C_SRC ${JTRACE_C_SRC}) list(APPEND JMSG_JAVA_SRC ${JTRACE_JAVA_SRC}) set(LUA_SRC src/bindings/lua/lua_host.cpp src/bindings/lua/lua_platf.cpp src/bindings/lua/lua_debug.cpp src/bindings/lua/simgrid_lua.cpp ) set(TRACING_SRC src/instr/instr_TI_trace.cpp src/instr/instr_config.cpp src/instr/instr_interface.cpp src/instr/instr_paje_containers.cpp src/instr/instr_paje_header.cpp src/instr/instr_paje_trace.cpp src/instr/instr_paje_types.cpp src/instr/instr_paje_values.cpp src/instr/instr_private.h src/instr/instr_smpi.h src/instr/instr_resource_utilization.cpp src/instr/instr_trace.cpp ) set(JEDULE_SRC include/simgrid/jedule/jedule_events.hpp include/simgrid/jedule/jedule.hpp include/simgrid/jedule/jedule_platform.hpp include/simgrid/jedule/jedule_sd_binding.h src/instr/jedule/jedule_events.cpp src/instr/jedule/jedule.cpp src/instr/jedule/jedule_platform.cpp src/instr/jedule/jedule_sd_binding.cpp ) set(MC_SRC_BASE src/mc/mc_base.cpp src/mc/mc_base.h src/mc/mc_record.h src/mc/mc_replay.h src/mc/mc_record.cpp src/mc/mc_config.cpp src/mc/mc_global.cpp ) set(MC_SRC src/mc/RemotePtr.hpp src/mc/AddressSpace.hpp src/mc/AddressSpace.cpp src/mc/Channel.cpp src/mc/Channel.hpp src/mc/Checker.cpp src/mc/Checker.hpp src/mc/Client.cpp src/mc/Client.hpp src/mc/CommunicationDeterminismChecker.cpp src/mc/CommunicationDeterminismChecker.hpp src/mc/Frame.hpp src/mc/Frame.cpp src/mc/ModelChecker.hpp src/mc/ModelChecker.cpp src/mc/ObjectInformation.hpp src/mc/ObjectInformation.cpp src/mc/PageStore.hpp src/mc/PageStore.cpp src/mc/SafetyChecker.cpp src/mc/SafetyChecker.hpp src/mc/ChunkedData.hpp src/mc/ChunkedData.cpp src/mc/RegionSnapshot.cpp src/mc/RegionSnapshot.hpp src/mc/Type.hpp src/mc/Variable.cpp src/mc/Variable.hpp src/mc/mc_forward.hpp src/mc/Process.hpp src/mc/Process.cpp src/mc/Session.cpp src/mc/Session.hpp src/mc/mc_unw.h src/mc/mc_unw.cpp src/mc/mc_unw_vmread.cpp src/mc/mc_checkpoint.cpp src/mc/mc_snapshot.h src/mc/mc_snapshot.cpp src/mc/mc_page_snapshot.cpp src/mc/mc_comm_pattern.h src/mc/mc_comm_pattern.cpp src/mc/compare.cpp src/mc/mc_dwarf.hpp src/mc/mc_dwarf.cpp src/mc/mc_dwarf_attrnames.cpp src/mc/DwarfExpression.hpp src/mc/DwarfExpression.cpp src/mc/mc_dwarf_tagnames.cpp src/mc/mc_hash.hpp src/mc/mc_hash.cpp src/mc/mc_ignore.h src/mc/mc_mmalloc.h src/mc/LivenessChecker.hpp src/mc/LocationList.hpp src/mc/LocationList.cpp src/mc/LivenessChecker.cpp src/mc/mc_record.cpp src/mc/mc_member.cpp src/mc/mc_memory.cpp src/mc/mc_private.h src/mc/mc_request.h src/mc/mc_request.cpp src/mc/mc_safety.h src/mc/mc_state.h src/mc/mc_state.cpp src/mc/VisitedState.cpp src/mc/VisitedState.hpp src/mc/mc_client_api.cpp src/mc/mc_protocol.h src/mc/mc_protocol.cpp src/mc/mc_smx.h src/mc/mc_smx.cpp src/mc/mc_xbt.hpp src/mc/mc_xbt.cpp src/mc/mc_exit.h src/mc/Transition.hpp ) set(MC_SIMGRID_MC_SRC src/mc/simgrid_mc.cpp) set(headers_to_install include/msg/msg.h include/msg/datatypes.h include/simdag/simdag.h include/simdag/datatypes.h include/simgrid/chrono.hpp include/simgrid/plugins/energy.h include/simgrid/instr.h include/simgrid/msg.h include/simgrid/simdag.h include/simgrid/datatypes.h include/simgrid/modelchecker.h include/simgrid/forward.h include/simgrid/simix.h include/simgrid/simix.hpp include/simgrid/simix/blocking_simcall.hpp include/simgrid/kernel/future.hpp include/simgrid/host.h include/simgrid/link.h include/simgrid/s4u/forward.hpp include/simgrid/s4u/Activity.hpp include/simgrid/s4u/Actor.hpp include/simgrid/s4u/comm.hpp include/simgrid/s4u/conditionVariable.hpp include/simgrid/s4u/engine.hpp include/simgrid/s4u/file.hpp include/simgrid/s4u/host.hpp include/simgrid/s4u/Mailbox.hpp include/simgrid/s4u/Mutex.hpp include/simgrid/s4u/NetZone.hpp include/simgrid/s4u/storage.hpp include/simgrid/s4u/VirtualMachine.hpp include/simgrid/s4u.hpp include/simgrid/plugins/energy.h include/smpi/mpi.h include/smpi/smpi.h include/smpi/smpi_main.h include/smpi/smpi_extended_traces.h include/smpi/smpi_extended_traces_fortran.h include/surf/surf_routing.h include/xbt.h include/xbt/RngStream.h include/xbt/asserts.h include/xbt/automaton.h include/xbt/automaton.hpp include/xbt/base.h include/xbt/config.h include/xbt/config.hpp include/xbt/cunit.h include/xbt/dict.h include/xbt/string.hpp include/xbt/signal.hpp include/xbt/dynar.h include/xbt/dynar.hpp include/xbt/ex.h include/xbt/ex.hpp include/xbt/exception.hpp include/xbt/backtrace.h include/xbt/backtrace.hpp include/xbt/fifo.h include/xbt/file.h include/xbt/function_types.h include/xbt/functional.hpp include/xbt/future.hpp include/xbt/graph.h include/xbt/heap.h include/xbt/lib.h include/xbt/Extendable.hpp include/xbt/log.h include/xbt/log.hpp include/xbt/mallocator.h include/xbt/matrix.h include/xbt/memory.hpp include/xbt/misc.h include/xbt/mmalloc.h include/xbt/module.h include/xbt/parmap.h include/xbt/range.hpp include/xbt/replay.h include/xbt/str.h include/xbt/strbuff.h include/xbt/swag.h include/xbt/synchro.h include/xbt/sysdep.h include/xbt/system_error.hpp include/xbt/utility.hpp include/xbt/virtu.h include/xbt/xbt_os_thread.h include/xbt/xbt_os_time.h ) set(source_of_generated_headers include/simgrid_config.h.in include/smpi/mpif.h.in) ### depend of some variables setted upper # -->HAVE_THREAD_CONTEXTS HAVE_UCONTEXT_CONTEXTS if(${HAVE_THREAD_CONTEXTS}) #pthread set(SURF_SRC ${SURF_SRC} src/kernel/context/ContextThread.cpp src/kernel/context/ContextThread.hpp ) else() # NOT pthread set(EXTRA_DIST ${EXTRA_DIST} src/kernel/context/ContextThread.cpp src/kernel/context/ContextThread.hpp ) endif() if(${HAVE_THREAD_CONTEXTS}) #pthread set(SURF_SRC ${SURF_SRC} src/xbt/xbt_os_thread.c) else() # NOT pthread set(EXTRA_DIST ${EXTRA_DIST} src/xbt/xbt_os_thread.c ) endif() if(${HAVE_UCONTEXT_CONTEXTS}) #ucontext set(SURF_SRC ${SURF_SRC} src/kernel/context/ContextUnix.cpp) else() # NOT ucontext set(EXTRA_DIST ${EXTRA_DIST} src/kernel/context/ContextUnix.cpp) endif() ### Simgrid Lib sources set(simgrid_sources ${PLUGINS_SRC} ${BINDINGS_SRC} ${MC_SRC_BASE} ${MSG_SRC} ${S4U_SRC} ${SIMDAG_SRC} ${SIMGRID_SRC} ${SIMIX_SRC} ${SURF_SRC} ${TRACING_SRC} ${XBT_SRC} ) if(${HAVE_JEDULE}) set(simgrid_sources ${simgrid_sources} ${JEDULE_SRC}) else() set(EXTRA_DIST ${EXTRA_DIST} ${JEDULE_SRC}) endif() if(enable_smpi) set(simgrid_sources ${simgrid_sources} ${SMPI_SRC}) endif() if(HAVE_MC) set(simgrid_sources ${simgrid_sources} ${MC_SRC}) endif() if(HAVE_NS3) set(simgrid_sources ${simgrid_sources} ${NS3_SRC}) endif() # WINDOWS if(WIN32) set(simgrid_sources ${simgrid_sources} src/kernel/context/ContextThread.cpp src/kernel/context/ContextThread.hpp src/xbt/xbt_os_thread.c ) endif() if(HAVE_LUA) set(simgrid_sources ${simgrid_sources} ${LUA_SRC}) else() set(EXTRA_DIST ${EXTRA_DIST} ${LUA_SRC}) endif() set(DOC_SOURCES doc/Doxyfile.in doc/Layout.xml doc/sg_thread_model.fig doc/simix.fig doc/surf_nutshell.fig doc/surf++.png doc/surf++.pdf doc/surf++.graphml doc/surf++.uml doc/triva-graph_configuration.png doc/triva-graph_configuration.svg doc/triva-graph_visualization.png doc/triva-graph_visualization.svg doc/triva-time_interval.png doc/triva-time_interval.svg doc/doxygen/FAQ.doc doc/doxygen/application.doc doc/doxygen/community.doc doc/doxygen/deployment.doc doc/doxygen/examples.doc doc/doxygen/footer.html doc/doxygen/getting_started.doc doc/doxygen/header.html doc/doxygen/howtos.doc doc/doxygen/index.doc doc/doxygen/inside.doc doc/doxygen/inside_tests.doc doc/doxygen/inside_cmake.doc doc/doxygen/inside_doxygen.doc doc/doxygen/inside_extending.doc doc/doxygen/inside_release.doc doc/doxygen/install.doc doc/doxygen/install_yours.doc doc/doxygen/java.doc doc/doxygen/tutorial.doc doc/doxygen/models.doc doc/doxygen/module-msg.doc doc/doxygen/module-s4u.doc doc/doxygen/module-sd.doc doc/doxygen/module-simix.doc doc/doxygen/module-smpi.doc doc/doxygen/module-surf.doc doc/doxygen/module-trace.doc doc/doxygen/module-xbt.doc doc/doxygen/module-index.doc doc/doxygen/ns3.doc doc/doxygen/options.doc doc/doxygen/outcomes.doc doc/doxygen/outcomes_logs.doc doc/doxygen/outcomes_MC.doc doc/doxygen/outcomes_vizu.doc doc/doxygen/platform.doc doc/doxygen/platform_lua.doc doc/doxygen/scenario.doc doc/doxygen/stylesheet.css doc/doxygen/uhood.doc doc/doxygen/uhood_switch.doc doc/doxygen/uhood_arch.doc doc/manpage/smpicc.1 doc/manpage/smpicxx.1 doc/manpage/smpif90.1 doc/manpage/smpiff.1 doc/manpage/smpirun.1 doc/manpage/tesh.pod doc/msg-tuto-src/deployment0.xml doc/msg-tuto-src/deployment1.xml doc/msg-tuto-src/deployment2.xml doc/msg-tuto-src/deployment3.xml doc/msg-tuto-src/deployment_general.xml doc/msg-tuto-src/masterworker0.c doc/msg-tuto-src/masterworker1.c doc/msg-tuto-src/masterworker2.c doc/msg-tuto-src/masterworker3.c doc/msg-tuto-src/masterworker4.c CITATION.bib ) set(DOC_FIGS ${CMAKE_HOME_DIRECTORY}/doc/shared/fig/simgrid_modules.fig ${CMAKE_HOME_DIRECTORY}/doc/shared/fig/simgrid_modules2.fig ) set(DOC_TOOLS tools/doxygen/fig2dev_postprocessor.pl tools/doxygen/xbt_log_extract_hierarchy.pl tools/doxygen/list_routing_models_examples.sh ) # these files get copied automatically to the html documentation set(DOC_IMG ${CMAKE_HOME_DIRECTORY}/doc/sc3-description.png ${CMAKE_HOME_DIRECTORY}/doc/webcruft/AS_hierarchy.png ${CMAKE_HOME_DIRECTORY}/doc/webcruft/Paje_MSG_screenshot.jpg ${CMAKE_HOME_DIRECTORY}/doc/webcruft/Paje_MSG_screenshot_thn.jpg ${CMAKE_HOME_DIRECTORY}/doc/webcruft/SGicon.gif ${CMAKE_HOME_DIRECTORY}/doc/webcruft/SGicon.icns ${CMAKE_HOME_DIRECTORY}/doc/webcruft/SGicon.ico ${CMAKE_HOME_DIRECTORY}/doc/webcruft/awstats_logo3.png ${CMAKE_HOME_DIRECTORY}/doc/webcruft/output.goal.pdf ${CMAKE_HOME_DIRECTORY}/doc/webcruft/poster_thumbnail.png ${CMAKE_HOME_DIRECTORY}/doc/webcruft/storage_sample_scenario.png ${CMAKE_HOME_DIRECTORY}/doc/webcruft/simgrid_logo_2011.gif ${CMAKE_HOME_DIRECTORY}/doc/webcruft/simgrid_logo_2011.png ${CMAKE_HOME_DIRECTORY}/doc/webcruft/simgrid_logo_2011_small.png ${CMAKE_HOME_DIRECTORY}/doc/webcruft/simgrid_logo_win.bmp ${CMAKE_HOME_DIRECTORY}/doc/webcruft/simgrid_logo_win_2011.bmp ${CMAKE_HOME_DIRECTORY}/doc/webcruft/smpi_simgrid_alltoall_pair_16.png ${CMAKE_HOME_DIRECTORY}/doc/webcruft/smpi_simgrid_alltoall_ring_16.png ) set(bin_files ${bin_files} src/smpi/smpicc.in src/smpi/smpicxx.in src/smpi/smpiff.in src/smpi/smpif90.in src/smpi/smpirun.in src/smpi/smpitools.sh ) set(txt_files ${txt_files} AUTHORS COPYING README ChangeLog INSTALL LICENSE-LGPL-2.1 NEWS TODO ) # The list of cmake build directories is constructed from the following list. # Add your CMakeLists file here to see your subdir built. set(CMAKEFILES_TXT examples/java/CMakeLists.txt examples/msg/CMakeLists.txt examples/msg/mc/CMakeLists.txt examples/s4u/CMakeLists.txt examples/simdag/CMakeLists.txt examples/smpi/CMakeLists.txt examples/smpi/NAS/CMakeLists.txt examples/smpi/smpi_msg_masterslave/CMakeLists.txt examples/smpi/replay_multiple/CMakeLists.txt examples/smpi/energy/f77/CMakeLists.txt examples/smpi/energy/f90/CMakeLists.txt teshsuite/java/CMakeLists.txt teshsuite/mc/CMakeLists.txt teshsuite/msg/CMakeLists.txt teshsuite/simdag/CMakeLists.txt teshsuite/simix/CMakeLists.txt teshsuite/smpi/CMakeLists.txt teshsuite/surf/CMakeLists.txt teshsuite/xbt/CMakeLists.txt teshsuite/smpi/isp/umpire/CMakeLists.txt teshsuite/smpi/mpich3-test/CMakeLists.txt teshsuite/smpi/mpich3-test/attr/CMakeLists.txt teshsuite/smpi/mpich3-test/coll/CMakeLists.txt teshsuite/smpi/mpich3-test/comm/CMakeLists.txt teshsuite/smpi/mpich3-test/datatype/CMakeLists.txt # teshsuite/smpi/mpich3-test/f77/attr/CMakeLists.txt teshsuite/smpi/mpich3-test/f77/coll/CMakeLists.txt teshsuite/smpi/mpich3-test/f77/info/CMakeLists.txt teshsuite/smpi/mpich3-test/f77/comm/CMakeLists.txt teshsuite/smpi/mpich3-test/f77/datatype/CMakeLists.txt teshsuite/smpi/mpich3-test/f77/ext/CMakeLists.txt teshsuite/smpi/mpich3-test/f77/init/CMakeLists.txt teshsuite/smpi/mpich3-test/f77/pt2pt/CMakeLists.txt teshsuite/smpi/mpich3-test/f77/util/CMakeLists.txt teshsuite/smpi/mpich3-test/f77/topo/CMakeLists.txt teshsuite/smpi/mpich3-test/f77/rma/CMakeLists.txt teshsuite/smpi/mpich3-test/f90/coll/CMakeLists.txt teshsuite/smpi/mpich3-test/f90/datatype/CMakeLists.txt teshsuite/smpi/mpich3-test/f90/info/CMakeLists.txt teshsuite/smpi/mpich3-test/f90/init/CMakeLists.txt teshsuite/smpi/mpich3-test/f90/pt2pt/CMakeLists.txt teshsuite/smpi/mpich3-test/f90/util/CMakeLists.txt teshsuite/smpi/mpich3-test/f90/rma/CMakeLists.txt teshsuite/smpi/mpich3-test/group/CMakeLists.txt teshsuite/smpi/mpich3-test/info/CMakeLists.txt teshsuite/smpi/mpich3-test/init/CMakeLists.txt teshsuite/smpi/mpich3-test/pt2pt/CMakeLists.txt teshsuite/smpi/mpich3-test/topo/CMakeLists.txt teshsuite/smpi/mpich3-test/rma/CMakeLists.txt teshsuite/smpi/mpich3-test/perf/CMakeLists.txt tools/CMakeLists.txt tools/graphicator/CMakeLists.txt tools/tesh/CMakeLists.txt ) set(CMAKE_SOURCE_FILES CMakeLists.txt FindSimGrid.cmake tools/cmake/Tests.cmake tools/cmake/CTestConfig.cmake tools/cmake/CTestCustom.cmake tools/cmake/DefinePackages.cmake tools/cmake/Distrib.cmake tools/cmake/GCCFlags.cmake tools/cmake/Documentation.cmake tools/cmake/MaintainerMode.cmake tools/cmake/Java.cmake tools/cmake/MakeLib.cmake tools/cmake/MakeLibWin.cmake tools/cmake/Modules/FindGraphviz.cmake tools/cmake/Modules/FindLibdw.cmake tools/cmake/Modules/FindLibunwind.cmake tools/cmake/Modules/FindLibevent.cmake tools/cmake/Modules/FindLuaSimgrid.cmake tools/cmake/Modules/FindNS3.cmake tools/cmake/Modules/FindPAPI.cmake tools/cmake/Modules/FindRngStream.cmake tools/cmake/Modules/FindValgrind.cmake tools/cmake/Option.cmake tools/cmake/scripts/my_valgrind.pl tools/cmake/scripts/update_tesh.pl tools/cmake/UnitTesting.cmake tools/cmake/src/internal_config.h.in tools/cmake/test_prog/prog_gnu_dynlinker.c tools/cmake/test_prog/prog_makecontext.c tools/cmake/test_prog/prog_mutex_timedlock.c tools/cmake/test_prog/prog_sem_init.c tools/cmake/test_prog/prog_sem_open.c tools/cmake/test_prog/prog_sem_timedwait.c tools/cmake/test_prog/prog_snprintf.c tools/cmake/test_prog/prog_stackgrowth.c tools/cmake/test_prog/prog_stacksetup.c tools/cmake/test_prog/prog_thread_storage.c tools/cmake/test_prog/prog_vsnprintf.c tools/cmake/cross-mingw.cmake tools/smpi/generate_smpi_defines.pl tools/stack-cleaner/as tools/stack-cleaner/cc tools/stack-cleaner/c++ tools/stack-cleaner/fortran tools/stack-cleaner/clean-stack-filter tools/stack-cleaner/compiler-wrapper tools/stack-cleaner/README ) set(PLATFORMS_EXAMPLES examples/platforms/bypassASroute.xml examples/platforms/bypassRoute.xml examples/platforms/cloud.xml examples/platforms/cluster.xml examples/platforms/cluster_and_one_host.xml examples/platforms/cluster_prototype.lua examples/platforms/cluster_no_backbone.xml examples/platforms/cluster_torus.xml examples/platforms/cluster_fat_tree.xml examples/platforms/cluster_dragonfly.xml examples/platforms/crosstraffic.xml examples/platforms/optorsim/gridpp_grid_2004.conf examples/platforms/optorsim/lcg_sept2004_grid.conf examples/platforms/optorsim/transform_optorsim_platform.pl examples/platforms/config.xml examples/platforms/config_tracing.xml examples/platforms/trace/bourassa_state.trace examples/platforms/trace/fafard_state.trace examples/platforms/trace/faulty_host.trace examples/platforms/trace/ginette_state.trace examples/platforms/trace/jupiter_speed.trace examples/platforms/trace/jupiter_state.trace examples/platforms/trace/link1_avail.trace examples/platforms/trace/link3_state.trace examples/platforms/trace/link4_state.trace examples/platforms/trace/linkBandwidth7.bw examples/platforms/trace/trace_A_failure.txt examples/platforms/trace/trace_A.txt examples/platforms/trace/trace_B.txt examples/platforms/data_center.xml examples/platforms/dogbone.xml examples/platforms/energy_platform.xml examples/platforms/faulty_host.xml examples/platforms/g5k.xml examples/platforms/griffon.xml examples/platforms/meta_cluster.xml examples/platforms/multicore_machine.xml examples/platforms/onelink.xml examples/platforms/prop.xml examples/platforms/routing_cluster.xml examples/platforms/routing_cluster.lua examples/platforms/routing_none.xml examples/platforms/simulacrum_7_hosts.xml examples/platforms/storage/content/small_content.txt examples/platforms/storage/content/storage_content.txt examples/platforms/storage/content/win_storage_content.txt examples/platforms/storage/remote_io.xml examples/platforms/storage/storage.xml examples/platforms/small_platform.xml examples/platforms/small_platform.lua examples/platforms/small_platform_fatpipe.xml examples/platforms/small_platform_one_link_routes.xml examples/platforms/small_platform_with_failures.xml examples/platforms/small_platform_with_routers.xml examples/platforms/syscoord/generate_peer_platform.pl examples/platforms/syscoord/median_harvard.syscoord examples/platforms/syscoord/median_meridian.syscoord examples/platforms/syscoord/median_p2psim.syscoord examples/platforms/three_multicore_hosts.xml examples/platforms/two_clusters.xml examples/platforms/two_hosts.xml examples/platforms/two_hosts_platform_shared.xml examples/platforms/two_hosts_platform_with_availability.xml examples/platforms/two_hosts_platform_with_availability_included.xml examples/platforms/two_peers.xml examples/platforms/vivaldi.xml ) set(generated_src_files src/xbt/automaton/automaton_lexer.yy.c src/xbt/automaton/parserPromela.tab.cacc src/xbt/automaton/parserPromela.tab.hacc ) foreach(file ${generated_src_files}) set_source_files_properties(${file} PROPERTIES GENERATED true) endforeach(file ${generated_src_files}) SimGrid-3.14.159/tools/cmake/CTestCustom.cmake000644 001750 001750 00000000072 13031031273 022527 0ustar00mquinsonmquinson000000 000000 SET(CTEST_CUSTOM_COVERAGE_EXCLUDE @PATTERN_CTEST_IGNORED@)SimGrid-3.14.159/tools/cmake/src/000755 001750 001750 00000000000 13031031274 020101 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/tools/cmake/src/internal_config.h.in000644 001750 001750 00000006412 13031031274 024023 0ustar00mquinsonmquinson000000 000000 /* internal_config.h -- characteristics of the platform, as probed by Cmake */ /* Warning: The file internal_config.h is AUTOMATICALLY GENERATED by Cmake. * Edit the template instead: tools/cmake/src/internal_config.h.in */ /* Copyright (c) 2004-2016. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid_config.h" /* what was compiled in? */ /* Non-standard header files */ /* */ #cmakedefine01 HAVE_EXECINFO_H /* */ #cmakedefine01 HAVE_FUTEX_H /* */ #cmakedefine01 HAVE_SIGNAL_H /* */ #cmakedefine01 HAVE_UNISTD_H /* */ #cmakedefine01 HAVE_UCONTEXT_H /* */ #cmakedefine01 HAVE_VALGRIND_H /* Time portability */ /* Function gettimeofday */ #cmakedefine01 HAVE_GETTIMEOFDAY /* Function clock_gettime */ #cmakedefine01 HAVE_POSIX_GETTIME /* Function nanosleep */ #cmakedefine01 HAVE_NANOSLEEP /* The usable context factories */ #define HAVE_BOOST_CONTEXTS @HAVE_BOOST_CONTEXTS@ #cmakedefine01 HAVE_RAW_CONTEXTS #cmakedefine01 HAVE_THREAD_CONTEXTS #cmakedefine01 HAVE_UCONTEXT_CONTEXTS /* Variables for the thread contexts (and parallel mode of raw contexts) */ /* Define to 1 if threads are usable . */ #cmakedefine01 HAVE_PTHREAD /* Does not seems defined on Mac nor Windows */ #cmakedefine01 HAVE_PTHREAD_SETAFFINITY /* If __thread is available */ #cmakedefine01 HAVE_THREAD_LOCAL_STORAGE /* Variables for the raw contexts (to select the right assembly code) */ #cmakedefine01 SIMGRID_PROCESSOR_i686 #cmakedefine01 SIMGRID_PROCESSOR_x86_64 /* Variables for the SysV contexts */ @sg_makecontext_stack_addr@ @sg_makecontext_stack_size@ /* Variable for SysV, raw and Boost contexts */ /* Does the stack growth upward, or downward? */ #define PTH_STACKGROWTH @PTH_STACKGROWTH@ /* MC variables */ /* getdtablesize: get descriptor table size */ #cmakedefine01 HAVE_GETDTABLESIZE /* Did we compile mmalloc in? */ #cmakedefine01 HAVE_MMALLOC /* process_vm_readv: transfer data between process address spaces */ #cmakedefine01 HAVE_PROCESS_VM_READV /* Set to true if enable_model-checking is true and the dependencies available */ #cmakedefine01 HAVE_MC /* SMPI variables */ /* SMPI enabled */ #cmakedefine01 HAVE_SMPI /* Fortran language is available for SMPI */ #cmakedefine01 SMPI_FORTRAN /* We have mmap and objdump to handle privatization */ #cmakedefine01 HAVE_PRIVATIZATION /* We have PAPI to fine-grain trace execution time */ #cmakedefine01 HAVE_PAPI /* Other function checks */ /* Function backtrace */ #cmakedefine01 HAVE_BACKTRACE /* Function dlfunc */ #cmakedefine01 HAVE_DLFUNC /* Function mmap */ #cmakedefine01 HAVE_MMAP /* Function mremap */ #cmakedefine01 HAVE_MREMAP /* Function sem_init (part of XPG6 standard only) */ #cmakedefine01 HAVE_SEM_INIT /* Function popen */ #cmakedefine01 HAVE_POPEN /* Function sysconf */ #cmakedefine01 HAVE_SYSCONF /* Function vasprintf */ #cmakedefine01 HAVE_VASPRINTF /* Other checks */ /* Path to the addr2line tool */ #cmakedefine ADDR2LINE "@ADDR2LINE@" /* The graphviz library */ #cmakedefine01 HAVE_GRAPHVIZ /* The lib unwind library (for MC and backtrace display) */ #cmakedefine01 HAVE_LIBUNWIND SimGrid-3.14.159/tools/cmake/Modules/000755 001750 001750 00000000000 13031031274 020722 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/tools/cmake/Modules/FindNS3.cmake000644 001750 001750 00000005353 13031031274 023136 0ustar00mquinsonmquinson000000 000000 # Try to find the NS3 library. # # The user can hint a path using the NS3_HINT option. # # Once done, the following will be defined: # # NS3_INCLUDE_DIRS - the NS3 include directories # NS3_LIBRARY_PATH - path to the libs # NS3_LIBRARIES - link these to use NS3 (full path to libs) # # This could be improved in many ways (patches welcome): # - No way to specify a minimal version (v3.10 is hardcoded). # - No proper find_package() integration ## ADDING A NS3 VERSION. # - Add ns3.${version}-core to the NAME line of the find_library below # - Add include/ns3{version} to the PATH_SUFFIXES line of the find_path below set(HAVE_NS3 0) set(NS3_HINT ${ns3_path} CACHE PATH "Path to search for NS3 lib and include") find_library(NS3_LIBRARIES NAME ns3-core ns3.14-core ns3.15-core ns3.16-core ns3.17-core ns3.18-core ns3.19-core ns3.20-core ns3.21-core ns3.22-core ns3.25-core ns3.26-core ns3.27-core ns3.28-core PATH_SUFFIXES lib64 lib ns3/lib PATHS ${NS3_HINT} ) find_path(NS3_INCLUDE_DIR NAME ns3/core-module.h PATH_SUFFIXES include ns3/include include/ns3.14 include/ns3.15 include/ns3.16 include/ns3.17 include/ns3.18 include/ns3.19 include/ns3.20 include/ns3.21 include/ns3.22 include/ns3.25 include/ns3.26 include/ns3.27 include/ns3.28 PATHS ${NS3_HINT} ) if(NS3_INCLUDE_DIR) message(STATUS "Looking for ns3/core-module.h - found") else() message(STATUS "Looking for ns3/core-module.h - not found") endif() mark_as_advanced(NS3_INCLUDE_DIR) message(STATUS "Looking for lib ns3-core") if(NS3_LIBRARIES) message(STATUS "Looking for lib ns3-core - found") else() message(STATUS "Looking for lib ns3-core - not found") endif() mark_as_advanced(NS3_LIBRARIES) if(NS3_INCLUDE_DIR) if(NS3_LIBRARIES) set(HAVE_NS3 1) string(REGEX REPLACE ".*ns([.0-9]+)-core.*" "\\1" NS3_VERSION "${NS3_LIBRARIES}") get_filename_component(NS3_LIBRARY_PATH "${NS3_LIBRARIES}" PATH) # Compute NS3_PATH string(REGEX REPLACE "(.*)/lib" "\\1" NS3_PATH "${NS3_LIBRARY_PATH}") message(STATUS "NS-3 found (v3.${NS3_VERSION} in ${NS3_PATH}).") if (NOT NS3_LIBRARY_PATH STREQUAL "/usr/lib") string(REGEX MATCH "${NS3_LIBRARY_PATH}" MatchResult "$ENV{LD_LIBRARY_PATH}") if(NOT MatchResult) message(STATUS "Warning: NS3 not installed in system path, and not listed in LD_LIBRARY_PATH." " You want to: export LD_LIBRARY_PATH=${NS3_LIBRARY_PATH}\${LD_LIBRARY_PATH:+:\$LD_LIBRARY_PATH}") endif() endif() endif() endif() mark_as_advanced(NS3_LIBRARY_PATH) if(HAVE_NS3) link_directories(${NS3_LIBRARY_PATH}) include_directories(${NS3_INCLUDE_DIR}) else() message(STATUS "Warning: To use NS-3 Please install ns3 at least version 3.10 (http://www.nsnam.org/releases/)") endif() SimGrid-3.14.159/tools/cmake/Modules/FindLuaSimgrid.cmake000644 001750 001750 00000004702 13031031274 024570 0ustar00mquinsonmquinson000000 000000 # Search for the Lua 5.3 include files and libraries # # Input variable: # LUA_HINT: path to Lua installation -- only needed for non-standard installs # Output variable: # HAVE_LUA : if Lua was found # LUA_LIBRARY : the path to the dynamic library # LUA_INCLUDE_DIR : where to find lua.h # LUA_VERSION_MAJOR: First part of the version (often, 5) # LUA_VERSION_MINOR: Second part of the version (3 when we have 5.3) find_path(LUA_INCLUDE_DIR lua.h HINTS ENV LUA_HINT PATH_SUFFIXES include/lua53 include/lua5.3 include/lua-5.3 include/lua include PATHS ~/Library/Frameworks /Library/Frameworks /sw # Fink /opt/local # DarwinPorts /opt/csw # Blastwave /opt ) find_library(LUA_LIBRARY NAMES # I don't want a lua.a lua53.so lua5.3.so lua-5.3.so lua53.dynlib lua5.3.dynlib lua-5.3.dynlib lua53.dll lua5.3.dll lua-5.3.dll lua.so lua.dynlib lua.dll lua53 lua5.3 lua-5.3 lua HINTS ENV LUA_HINT PATH_SUFFIXES lib PATHS ~/Library/Frameworks /Library/Frameworks /sw /opt/local /opt/csw /opt ) if (NOT LUA_LIBRARY) message(FATAL_ERROR "Error: Lua library not found. Please install that package (and set LUA_HINT) or disable Lua.") endif() if (NOT LUA_INCLUDE_DIR OR NOT EXISTS "${LUA_INCLUDE_DIR}/lua.h") message(FATAL_ERROR "Error: Lua header file not found. Please install that package (and set LUA_HINT) or disable Lua.") endif() # Extract the version info out of the header file file(STRINGS "${LUA_INCLUDE_DIR}/lua.h" lua_version_str REGEX "^#define[ \t]+LUA_VERSION_MAJOR[ \t]+\"[456]+\"") string(REGEX REPLACE "^#define[ \t]+LUA_VERSION_MAJOR[ \t]+\"([^\"]+)\"" "\\1" LUA_VERSION_MAJOR "${lua_version_str}") file(STRINGS "${LUA_INCLUDE_DIR}/lua.h" lua_version_str REGEX "^#define[ \t]+LUA_VERSION_MINOR[ \t]+\"[0123456789]+\"") string(REGEX REPLACE "^#define[ \t]+LUA_VERSION_MINOR[ \t]+\"([^\"]+)\"" "\\1" LUA_VERSION_MINOR "${lua_version_str}") unset(lua_version_str) # Check that we have a sufficient version of Lua if(LUA_VERSION_MAJOR EQUAL 5 AND LUA_VERSION_MINOR EQUAL 3) set(HAVE_LUA 1) include_directories(${LUA_INCLUDE_DIR}) else() message(FATAL_ERROR "Error: Lua version 5.3 is required, but version ${LUA_VERSION_MAJOR}.${LUA_VERSION_MINOR} found instead.") endif() message(STATUS "Lua version: ${LUA_VERSION_MAJOR}.${LUA_VERSION_MINOR}") message(STATUS "Lua library: ${LUA_LIBRARY}") mark_as_advanced(LUA_INCLUDE_DIR) mark_as_advanced(LUA_LIBRARY) SimGrid-3.14.159/tools/cmake/Modules/FindGraphviz.cmake000644 001750 001750 00000002345 13031031273 024322 0ustar00mquinsonmquinson000000 000000 find_path(HAVE_CGRAPH_H cgraph.h HINTS $ENV{LD_LIBRARY_PATH} PATH_SUFFIXES include/graphviz include PATHS /opt;/opt/local;/opt/csw;/sw;/usr ) find_library(HAVE_CGRAPH_LIB NAME cgraph HINTS $ENV{LD_LIBRARY_PATH} PATH_SUFFIXES lib/graphviz lib PATHS /opt;/opt/local;/opt/csw;/sw;/usr) find_library(HAVE_CDT_LIB NAME cdt HINTS $ENV{LD_LIBRARY_PATH} PATH_SUFFIXES lib/graphviz lib PATHS /opt;/opt/local;/opt/csw;/sw;/usr) if(HAVE_CDT_LIB AND HAVE_CGRAPH_LIB AND HAVE_CGRAPH_H) string(REGEX REPLACE "/libcgraph.*" "" lib_graphviz ${HAVE_CGRAPH_LIB}) string(REPLACE "/graphviz/cgraph.h" "" file_graphviz_h ${HAVE_CGRAPH_H}) string(REPLACE "/graphviz" "" file_graphviz_h ${file_graphviz_h}) include_directories(${file_graphviz_h} ${file_graphviz_h}/graphviz) link_directories(${lib_graphviz}) set(HAVE_GRAPHVIZ "1") else() set(HAVE_GRAPHVIZ "0") endif() mark_as_advanced(HAVE_GRAPHVIZ) unset(HAVE_CGRAPH_H) unset(HAVE_CGRAPH_LIB) unset(HAVE_CDT_LIB) message(STATUS "Looking for graphviz") if(HAVE_GRAPHVIZ) message(STATUS "Looking for graphviz - found") else() message(STATUS "Looking for graphviz - not found (try installing libgraphviz-dev)") endif() SimGrid-3.14.159/tools/cmake/Modules/FindValgrind.cmake000644 001750 001750 00000002130 13031031274 024267 0ustar00mquinsonmquinson000000 000000 find_program(VALGRIND_EXE NAME valgrind PATH_SUFFIXES bin/ PATHS /opt /opt/local /opt/csw /sw /usr ) mark_as_advanced(VALGRIND_EXE) if(enable_memcheck) if (NOT VALGRIND_EXE MATCHES "NOTFOUND") execute_process(COMMAND ${VALGRIND_EXE} --version OUTPUT_VARIABLE "VALGRIND_VERSION") message(STATUS "Valgrind version: ${VALGRIND_VERSION}") set(TESH_WRAPPER ${CMAKE_HOME_DIRECTORY}/tools/cmake/scripts/my_valgrind.pl) set(TESH_WRAPPER ${TESH_WRAPPER}\ --trace-children=yes\ --trace-children-skip=/usr/bin/*,/bin/*\ --leak-check=full\ --show-reachable=yes\ --track-origins=no\ --read-var-info=no\ --num-callers=20\ --suppressions=${CMAKE_HOME_DIRECTORY}/tools/simgrid.supp\ ) if(enable_memcheck_xml) SET(TESH_WRAPPER ${TESH_WRAPPER}\ --xml=yes\ --xml-file=memcheck_test_%p.memcheck\ --child-silent-after-fork=yes\ ) endif() # message(STATUS "tesh wrapper: ${TESH_WRAPPER}") mark_as_advanced(TESH_WRAPPER) else() set(enable_memcheck false) message(STATUS "Error: Command valgrind not found --> enable_memcheck autoset to false.") endif() endif() SimGrid-3.14.159/tools/cmake/Modules/FindRngStream.cmake000644 001750 001750 00000002255 13031031274 024433 0ustar00mquinsonmquinson000000 000000 find_path(HAVE_RNGSTREAM_H NAME RngStream.h HINTS $ENV{HOME} PATH_SUFFIXES include PATHS /opt /opt/local /opt/csw /sw /usr ) find_library(HAVE_RNGSTREAM_LIB NAME rngstreams HINTS $ENV{HOME} PATH_SUFFIXES lib64 lib lib32 PATHS /opt /opt/local /opt/csw /sw /usr ) message(STATUS "Looking for RngStream.h") if(HAVE_RNGSTREAM_H) message(STATUS "Looking for RngStream.h - found") string(REGEX MATCH "-I${HAVE_RNGSTREAM_H} " operation "${CMAKE_C_FLAGS}") if(NOT operation) SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS}-I${HAVE_RNGSTREAM_H} ") endif() else() message(STATUS "Looking for RngStream.h - not found") endif() message(STATUS "Looking for lib rngstreams") if(HAVE_RNGSTREAM_LIB) message(STATUS "Looking for lib rngstreams - found") string(REGEX REPLACE "/librngstreams.*" "" HAVE_RNGSTREAM_LIB "${HAVE_RNGSTREAM_LIB}") string(REGEX MATCH "-L${HAVE_RNGSTREAM_LIB} " operation "${CMAKE_C_FLAGS}") if(NOT operation) SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS}-L${HAVE_RNGSTREAM_LIB} ") endif() else() message(STATUS "Looking for lib rngstreams - not found") endif() mark_as_advanced(HAVE_RNGSTREAM_LIB) mark_as_advanced(HAVE_RNGSTREAM_H)SimGrid-3.14.159/tools/cmake/Modules/FindLibunwind.cmake000644 001750 001750 00000003366 13031031274 024470 0ustar00mquinsonmquinson000000 000000 if(SIMGRID_PROCESSOR_x86_64) find_library(PATH_LIBUNWIND_LIB NAMES unwind-x86_64 HINTS $ENV{SIMGRID_LIBUNWIND_LIBRARY_PATH} $ENV{LD_LIBRARY_PATH} $ENV{LIBUNWIND_LIBRARY_PATH} PATH_SUFFIXES lib/ GnuWin32/lib lib/system PATHS /opt /opt/local /opt/csw /sw /usr) endif() if(NOT PATH_LIBUNWIND_LIB) find_library(PATH_LIBUNWIND_LIB NAMES unwind HINTS $ENV{SIMGRID_LIBUNWIND_LIBRARY_PATH} $ENV{LD_LIBRARY_PATH} $ENV{LIBUNWIND_LIBRARY_PATH} PATH_SUFFIXES lib/ GnuWin32/lib lib/system PATHS /opt /opt/local /opt/csw /sw /usr /usr/lib/) endif() find_path(PATH_LIBUNWIND_H "libunwind.h" HINTS $ENV{SIMGRID_LIBUNWIND_LIBRARY_PATH} $ENV{LD_LIBRARY_PATH} $ENV{LIBUNWIND_LIBRARY_PATH} PATH_SUFFIXES include/ GnuWin32/include PATHS /opt /opt/local /opt/csw /sw /usr) message(STATUS "Looking for libunwind.h") if(PATH_LIBUNWIND_H) message(STATUS "Looking for libunwind.h - found") else() message(STATUS "Looking for libunwind.h - not found") endif() message(STATUS "Looking for libunwind") if(PATH_LIBUNWIND_LIB) message(STATUS "Looking for libunwind - found") else() message(STATUS "Looking for libunwind - not found") endif() if(PATH_LIBUNWIND_LIB AND PATH_LIBUNWIND_H) string(REGEX REPLACE "/libunwind.*[.]${LIB_EXE}$" "" PATH_LIBUNWIND_LIB "${PATH_LIBUNWIND_LIB}") string(REGEX REPLACE "/libunwind.h" "" PATH_LIBUNWIND_H "${PATH_LIBUNWIND_H}") include_directories(${PATH_LIBUNWIND_H}) link_directories(${PATH_LIBUNWIND_LIB}) SET(HAVE_LIBUNWIND 1) else() SET(HAVE_LIBUNWIND 0) endif() mark_as_advanced(PATH_LIBDW_H) mark_as_advanced(PATH_LIBDW_LIB) mark_as_advanced(PATH_LIBUNWIND_LIB) mark_as_advanced(PATH_LIBUNWIND_H) SimGrid-3.14.159/tools/cmake/Modules/FindPAPI.cmake000644 001750 001750 00000003602 13031031274 023257 0ustar00mquinsonmquinson000000 000000 # Try to find PAPI headers and LIBRARY. # # Usage of this module as follows: # # find_package(PAPI) # # Variables used by this module, they can change the default behaviour and need # to be set before calling find_package: # # PAPI_PREFIX Set this variable to the root installation of # libpapi if the module has problems finding the # proper installation path. # # Variables defined by this module: # # PAPI_FOUND System has PAPI LIBRARY and headers # PAPI_LIBRARY The PAPI library # PAPI_INCLUDE_DIRS The location of PAPI headers set(HAVE_PAPI 0) set(PAPI_HINT ${papi_path} CACHE PATH "Path to search for PAPI headers and library") find_path(PAPI_PREFIX NAMES include/papi.h PATHS ${PAPI_HINT} ) message(STATUS "Looking for libpapi") find_library(PAPI_LIBRARY NAMES libpapi papi PATH_SUFFIXES lib64 lib # HINTS gets searched before PATHS HINTS ${PAPI_PREFIX}/lib ) if(PAPI_LIBRARY) message(STATUS "Looking for libpapi - found at ${PAPI_LIBRARY}") else() message(STATUS "Looking for libpapi - not found") endif() message(STATUS "Looking for papi.h") find_path(PAPI_INCLUDE_DIRS NAMES papi.h # HINTS gets searched before PATHS HINTS ${PAPI_PREFIX}/include ) if(PAPI_INCLUDE_DIRS) message(STATUS "Looking for papi.h - found at ${PAPI_INCLUDE_DIRS}") else() message(STATUS "Looking for papi.h - not found") endif() if (PAPI_LIBRARY) if(PAPI_INCLUDE_DIRS) set(HAVE_PAPI 1) mark_as_advanced(HAVE_PAPI) endif() endif() if(NOT HAVE_PAPI) message(FATAL_ERROR, "Could not find PAPI LIBRARY and/or papi.h. Make sure they are correctly installed!") endif() #include(FindPackageHandleStandardArgs) #find_package_handle_standard_args(PAPI DEFAULT_MSG # PAPI_LIBRARY # PAPI_INCLUDE_DIRS #) mark_as_advanced( PAPI_PREFIX_DIRS PAPI_LIBRARY PAPI_INCLUDE_DIRS ) SimGrid-3.14.159/tools/cmake/Modules/FindLibevent.cmake000644 001750 001750 00000000627 13031031274 024302 0ustar00mquinsonmquinson000000 000000 find_path(LIBEVENT_INCLUDE_DIR event2/event.h) find_library(LIBEVENT_LIBRARY NAMES event) find_library(LIBEVENT_THREADS_LIBRARY NAMES event_pthreads) set(LIBEVENT_LIBRARIES "${LIBEVENT_LIBRARY}") include(FindPackageHandleStandardArgs) find_package_handle_standard_args( Libevent DEFAULT_MSG LIBEVENT_LIBRARIES LIBEVENT_INCLUDE_DIR) mark_as_advanced(LIBEVENT_INCLUDE_DIR LIBEVENT_LIBRARIES) SimGrid-3.14.159/tools/cmake/Modules/FindLibdw.cmake000644 001750 001750 00000001253 13031031274 023567 0ustar00mquinsonmquinson000000 000000 find_path(LIBDW_INCLUDE_DIR "elfutils/libdw.h" HINTS $ENV{SIMGRID_LIBDW_LIBRARY_PATH} $ENV{LD_LIBRARY_PATH} $ENV{LIBDW_LIBRARY_PATH} PATH_SUFFIXES include/ GnuWin32/include PATHS /opt /opt/local /opt/csw /sw /usr) find_library(LIBDW_LIBRARY NAMES dw HINTS $ENV{SIMGRID_LIBDW_LIBRARY_PATH} $ENV{LD_LIBRARY_PATH} $ENV{LIBDW_LIBRARY_PATH} PATH_SUFFIXES lib/ GnuWin32/lib PATHS /opt /opt/local /opt/csw /sw /usr) set(LIBDW_LIBRARIES "${LIBDW_LIBRARY}") include(FindPackageHandleStandardArgs) find_package_handle_standard_args( Libdw DEFAULT_MSG LIBDW_LIBRARIES LIBDW_INCLUDE_DIR) mark_as_advanced(LIBDW_INCLUDE_DIR LIBDW_LIBRARIES) SimGrid-3.14.159/tools/cmake/Documentation.cmake000644 001750 001750 00000016523 13031031273 023133 0ustar00mquinsonmquinson000000 000000 ### ### Generate all parts of the documentation on non-Windows systems ### ### - HTML with doxygen (reference and manual) ### - Javadoc (reference) ### - manpages (reference of tools) ### ### This file is not loaded on windows #### Generate the html documentation if (enable_documentation) find_package(Doxygen REQUIRED) else() find_package(Doxygen) endif() find_path(FIG2DEV_PATH NAMES fig2dev PATHS NO_DEFAULT_PATHS) if(enable_documentation) ADD_CUSTOM_TARGET(documentation COMMENT "Generating the SimGrid documentation..." DEPENDS ${DOC_SOURCES} ${DOC_FIGS} ${source_doxygen} COMMAND ${CMAKE_COMMAND} -E remove_directory ${CMAKE_HOME_DIRECTORY}/doc/html COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_HOME_DIRECTORY}/doc/html WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY}/doc ) message(STATUS "Doxygen version: ${DOXYGEN_VERSION}") # This is a workaround for older cmake versions (such as 2.8.7 on Ubuntu 12.04). These cmake versions do not provide # the DOXYGEN_VERSION variable and hence, building the documentation will always fail. This code is the same as used # in the cmake library, version 3. if(DOXYGEN_EXECUTABLE) execute_process(COMMAND ${DOXYGEN_EXECUTABLE} "--version" OUTPUT_VARIABLE DOXYGEN_VERSION OUTPUT_STRIP_TRAILING_WHITESPACE) endif() if(DOXYGEN_VERSION VERSION_LESS "1.8") ADD_CUSTOM_TARGET(error_doxygen COMMAND ${CMAKE_COMMAND} -E echo "Doxygen must be at least version 1.8 to generate documentation. Version found: ${DOXYGEN_VERSION}" COMMAND false ) add_dependencies(documentation error_doxygen) endif() configure_file(${CMAKE_HOME_DIRECTORY}/doc/Doxyfile.in ${CMAKE_HOME_DIRECTORY}/doc/Doxyfile @ONLY) foreach(file ${DOC_FIGS}) string(REPLACE ".fig" ".png" tmp_file ${file}) string(REPLACE "${CMAKE_HOME_DIRECTORY}/doc/shared/fig/" "${CMAKE_HOME_DIRECTORY}/doc/html/" tmp_file ${tmp_file}) ADD_CUSTOM_COMMAND(TARGET documentation COMMAND ${FIG2DEV_PATH}/fig2dev -Lpng -S 4 ${file} ${tmp_file}) endforeach() foreach(file ${DOC_IMG}) ADD_CUSTOM_COMMAND(TARGET documentation COMMAND ${CMAKE_COMMAND} -E copy ${file} ${CMAKE_HOME_DIRECTORY}/doc/html/) endforeach() ADD_CUSTOM_COMMAND(TARGET documentation COMMAND ${FIG2DEV_PATH}/fig2dev -Lmap ${CMAKE_HOME_DIRECTORY}/doc/shared/fig/simgrid_modules.fig | perl -pe 's/imagemap/simgrid_modules/g'| perl -pe 's/ ${CMAKE_HOME_DIRECTORY}/doc/simgrid_modules.map COMMAND pwd COMMAND ${CMAKE_COMMAND} -E echo "XX Generate the index files" COMMAND ${CMAKE_COMMAND} -E remove ${CMAKE_HOME_DIRECTORY}/doc/doxygen/logcategories.doc COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/xbt_log_extract_hierarchy.pl > ${CMAKE_HOME_DIRECTORY}/doc/doxygen/logcategories.doc COMMAND ${CMAKE_COMMAND} -E echo "XX Generate list of files in examples/ for routing models" COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_HOME_DIRECTORY}/doc/example_lists/ COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/list_routing_models_examples.sh Floyd > ${CMAKE_HOME_DIRECTORY}/doc/example_lists/example_filelist_routing_floyd COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/list_routing_models_examples.sh Dijkstra > ${CMAKE_HOME_DIRECTORY}/doc/example_lists/example_filelist_routing_dijkstra COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/list_routing_models_examples.sh DijkstraCache > ${CMAKE_HOME_DIRECTORY}/doc/example_lists/example_filelist_routing_dijkstra_cache COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/list_routing_models_examples.sh 'routing="None"' > ${CMAKE_HOME_DIRECTORY}/doc/example_lists/example_filelist_routing_none COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/list_routing_models_examples.sh 'routing="Cluster"' > ${CMAKE_HOME_DIRECTORY}/doc/example_lists/example_filelist_routing_cluster COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/list_routing_models_examples.sh 'routing="Vivaldi"' > ${CMAKE_HOME_DIRECTORY}/doc/example_lists/example_filelist_routing_vivaldi COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/list_routing_models_examples.sh 'routing="Full"' > ${CMAKE_HOME_DIRECTORY}/doc/example_lists/example_filelist_routing_full COMMAND ${CMAKE_COMMAND} -E echo "XX Generate list of files in examples/ for XML tags" COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/list_routing_models_examples.sh ' ${CMAKE_HOME_DIRECTORY}/doc/example_lists/example_filelist_xmltag_mount COMMAND ${CMAKE_HOME_DIRECTORY}/tools/doxygen/list_routing_models_examples.sh ' ${CMAKE_HOME_DIRECTORY}/doc/example_lists/example_filelist_xmltag_linkctn COMMAND ${CMAKE_COMMAND} -E echo "XX Run doxygen" COMMAND ${DOXYGEN_EXECUTABLE} Doxyfile COMMAND ${CMAKE_COMMAND} -E remove ${CMAKE_HOME_DIRECTORY}/doc/simgrid_modules.map WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY}/doc ) ### Fill in the "make gforge-gforge" target ### set(RSYNC_CMD rsync --verbose --cvs-exclude --compress --delete --delete-excluded --rsh=ssh --ignore-times --recursive --links --times --omit-dir-times --perms --chmod=a+rX,ug+w,o-w,Dg+s) add_custom_target(gforge-sync COMMAND ssh scm.gforge.inria.fr mkdir -p -m 2775 /home/groups/simgrid/htdocs/simgrid/${release_version}/ || true COMMAND ${RSYNC_CMD} doc/html/ scm.gforge.inria.fr:/home/groups/simgrid/htdocs/simgrid/${release_version}/doc/ || true COMMAND ${RSYNC_CMD} doc/html/simgrid_modules2.png doc/html/simgrid_modules.png doc/webcruft/simgrid_logo_2011.png doc/webcruft/simgrid_logo_2011_small.png scm.gforge.inria.fr:/home/groups/simgrid/htdocs/simgrid/${release_version}/ COMMAND ${RSYNC_CMD} src/surf/xml/simgrid.dtd scm.gforge.inria.fr:/home/groups/simgrid/htdocs/simgrid/ COMMAND ${RSYNC_CMD} src/surf/xml/simgrid.dtd scm.gforge.inria.fr:/home/groups/simgrid/htdocs/simgrid/${release_version}/simgrid.dtd WORKING_DIRECTORY "${CMAKE_HOME_DIRECTORY}" ) add_dependencies(gforge-sync documentation) else(enable_documentation) ADD_CUSTOM_TARGET(documentation COMMENT "The generation of the SimGrid documentation was disabled in cmake" ) endif(enable_documentation) if (Java_FOUND) find_path(JAVADOC_PATH NAMES javadoc PATHS NO_DEFAULT_PATHS) mark_as_advanced(JAVADOC_PATH) ADD_CUSTOM_COMMAND(TARGET documentation COMMAND ${CMAKE_COMMAND} -E echo "XX Javadoc pass" COMMAND ${JAVADOC_PATH}/javadoc -quiet -d ${CMAKE_HOME_DIRECTORY}/doc/html/javadoc/ ${CMAKE_HOME_DIRECTORY}/src/bindings/java/org/simgrid/*.java ${CMAKE_HOME_DIRECTORY}/src/bindings/java/org/simgrid/*/*.java WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY}/doc ) endif() #### Generate the manpages if( NOT MANPAGE_DIR) set( MANPAGE_DIR ${CMAKE_BINARY_DIR}/manpages ) endif() add_custom_target(manpages ALL COMMAND ${CMAKE_COMMAND} -E make_directory ${MANPAGE_DIR} COMMAND pod2man ${CMAKE_HOME_DIRECTORY}/tools/simgrid_update_xml.pl > ${MANPAGE_DIR}/simgrid_update_xml.1 COMMAND pod2man ${CMAKE_HOME_DIRECTORY}/doc/manpage/tesh.pod > ${MANPAGE_DIR}/tesh.1 COMMENT "Generating manpages" ) install(FILES ${MANPAGE_DIR}/simgrid_update_xml.1 ${MANPAGE_DIR}/tesh.1 ${CMAKE_HOME_DIRECTORY}/doc/manpage/smpicc.1 ${CMAKE_HOME_DIRECTORY}/doc/manpage/smpicxx.1 ${CMAKE_HOME_DIRECTORY}/doc/manpage/smpif90.1 ${CMAKE_HOME_DIRECTORY}/doc/manpage/smpiff.1 ${CMAKE_HOME_DIRECTORY}/doc/manpage/smpirun.1 DESTINATION $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}/share/man/man1) SimGrid-3.14.159/tools/cmake/Option.cmake000644 001750 001750 00000010061 13031031274 021562 0ustar00mquinsonmquinson000000 000000 ### ARGs use -D[var]=[ON/OFF] or [1/0] or [true/false](see below) ### ex: cmake -Denable_java=ON -Denable_ns3=ON ./ set(BIBTEX2HTML ${BIBTEX2HTML} CACHE PATH "Path to bibtex2html") if(NOT CMAKE_INSTALL_PREFIX) set(CMAKE_INSTALL_PREFIX "/usr/local/simgrid/" CACHE PATH "Path where this project should be installed") else() set(CMAKE_INSTALL_PREFIX ${CMAKE_INSTALL_PREFIX} CACHE PATH "Path where this project should be installed") endif() option(release "Whether Release Mode is activated (disable tests on experimental parts)" on) # How to build ### option(enable_compile_optimizations "Whether to produce efficient code for the SimGrid library" on) option(enable_compile_warnings "Whether compilation warnings should be turned into errors." off) option(enable_lto "Whether we should try to activate the LTO (link time optimisation)" on) option(enable_mallocators "Enable mallocators (disable only for debugging purpose)." on) option(enable_maintainer_mode "Whether flex and flexml files should be rebuilt." off) option(enable_debug "Turn this off to remove all debug messages at compile time (faster, but no debug activatable)" on) # Optional modules ### option(enable_documentation "Whether to produce documentation" on) option(enable_ns3 "Whether ns3 model is activated." off) option(enable_java "Whether the Java bindings are activated." off) option(enable_lib_in_jar "Whether the native libraries are bundled in a Java jar file" on) option(enable_lua "Whether the Lua bindings are activated." off) option(enable_model-checking "Turn this on to experiment with our prototype of model-checker (hinders the simulation's performance even if turned off at runtime)" off) option(enable_jedule "Jedule output of SimDAG." off) if(WIN32) option(enable_smpi "Whether SMPI in included in library." off) option(enable_smpi_MPICH3_testsuite "Whether the test suite form MPICH 3 should be built" off) else() option(enable_smpi "Whether SMPI in included in library." on) # PAPI does not support windows (they did in 3.7, but not anymore in 5.x) # See http://icl.cs.utk.edu/papi/custom/index.html?lid=62&slid=96 option(enable_smpi_papi "Whether SMPI supports PAPI bindings." off) option(enable_smpi_MPICH3_testsuite "Whether the test suite form MPICH 3 should be built" off) endif() option(enable_smpi_ISP_testsuite "Whether the test suite from ISP should be built." off) # Internal targets used by jenkins ### option(enable_fortran "Whether fortran is used with SMPI. Turned on by default if gfortran is found." on) option(enable_coverage "Whether coverage should be enabled." off) mark_as_advanced(enable_coverage) option(enable_memcheck "Enable memcheck." off) mark_as_advanced(enable_memcheck) option(enable_memcheck_xml "Enable memcheck with xml output." off) mark_as_advanced(enable_memcheck_xml) option(enable_address_sanitizer "Whether address sanitizer is turned on." off) mark_as_advanced(enable_address_sanitizer) option(enable_thread_sanitizer "Whether thread sanitizer is turned on." off) mark_as_advanced(enable_thread_sanitizer) option(enable_undefined_sanitizer "Whether undefined sanitizer is turned on." off) mark_as_advanced(enable_undefined_sanitizer) # Cmake, Y U NO hide your garbage?? ### mark_as_advanced(HAVE_SSH) mark_as_advanced(HAVE_RSYNC) mark_as_advanced(BIBTEX2HTML_PATH) mark_as_advanced(BUILDNAME) mark_as_advanced(ADDR2LINE) mark_as_advanced(BIBTOOL_PATH) mark_as_advanced(BUILD_TESTING) mark_as_advanced(CMAKE_BUILD_TYPE) mark_as_advanced(DART_ROOT) mark_as_advanced(DOXYGEN_PATH) mark_as_advanced(FIG2DEV_PATH) mark_as_advanced(FLEXML_EXE) mark_as_advanced(FLEX_EXE) mark_as_advanced(GCOV_PATH) mark_as_advanced(ICONV_PATH) mark_as_advanced(MAKE_PATH) mark_as_advanced(SVN) mark_as_advanced(GIT) mark_as_advanced(CMAKE_OSX_ARCHITECTURES) mark_as_advanced(CMAKE_OSX_DEPLOYMENT_TARGET) mark_as_advanced(CMAKE_OSX_SYSROOT) mark_as_advanced(SED_EXE) mark_as_advanced(BIBTEX2HTML) mark_as_advanced(CMAKE_C_LINK_FLAGS) mark_as_advanced(CMAKE_CXX_FLAGS) mark_as_advanced(CMAKE_Fortran_LINK_FLAGS) SimGrid-3.14.159/tools/cmake/UnitTesting.cmake000644 001750 001750 00000003531 13031031274 022573 0ustar00mquinsonmquinson000000 000000 # This file is in charge of the unit testing in SimGrid. # See http://simgrid.gforge.inria.fr/simgrid/3.13/doc/inside_tests.html#inside_tests_add_units # To register a file containing unit tests, simply add it to # FILES_CONTAINING_UNITTESTS and have a pleasant day. set(FILES_CONTAINING_UNITTESTS src/xbt/cunit.cpp src/xbt/ex.cpp src/xbt/dynar.cpp src/xbt/dict.cpp src/xbt/swag.c src/xbt/xbt_str.cpp src/xbt/xbt_strbuff.c src/xbt/config.cpp ) if(HAVE_MC) set(FILES_CONTAINING_UNITTESTS ${FILES_CONTAINING_UNITTESTS} src/mc/PageStore.cpp src/mc/mc_snapshot.cpp ) endif() #### Nothing to change below this line to add a new tested file ################################################################ foreach(file ${FILES_CONTAINING_UNITTESTS}) get_filename_component(basename ${file} NAME_WE) get_filename_component(ext ${file} EXT) set(EXTRACTED_TEST_SOURCE_FILES ${EXTRACTED_TEST_SOURCE_FILES} ${CMAKE_CURRENT_BINARY_DIR}/src/${basename}_unit${ext}) endforeach() set(EXTRACTED_TEST_SOURCE_FILES ${EXTRACTED_TEST_SOURCE_FILES} ${CMAKE_CURRENT_BINARY_DIR}/src/simgrid_units_main.c) set_source_files_properties(${EXTRACTED_TEST_SOURCE_FILES} PROPERTIES GENERATED true) ADD_CUSTOM_COMMAND( OUTPUT ${EXTRACTED_TEST_SOURCE_FILES} DEPENDS ${CMAKE_HOME_DIRECTORY}/tools/sg_unit_extractor.pl ${FILES_CONTAINING_UNITTESTS} COMMENT "Generating *_units files for testall..." WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY} COMMAND ${CMAKE_COMMAND} -E remove -f ${EXTRACTED_TEST_SOURCE_FILES} COMMAND chmod +x ${CMAKE_HOME_DIRECTORY}/tools/sg_unit_extractor.pl COMMAND ${CMAKE_HOME_DIRECTORY}/tools/sg_unit_extractor.pl --root=src/ --outdir=${CMAKE_CURRENT_BINARY_DIR}/src/ ${FILES_CONTAINING_UNITTESTS} ) add_executable (testall ${EXTRACTED_TEST_SOURCE_FILES}) target_link_libraries(testall simgrid) SimGrid-3.14.159/tools/cmake/Java.cmake000644 001750 001750 00000013404 13031031273 021176 0ustar00mquinsonmquinson000000 000000 ## ## The Cmake definitions for the use of Java (and Scala) ## This file is loaded only if the Java option is activated ## find_package(Java 1.7 COMPONENTS Runtime Development) if (NOT ${Java_FOUND}) message(FATAL_ERROR "Java not found (need at least Java7). Please install the JDK or disable that option") endif() set(Java_FOUND 1) include(UseJava) find_package(JNI REQUIRED) message("-- [Java] JNI found: ${JNI_FOUND}") message("-- [Java] JNI include dirs: ${JNI_INCLUDE_DIRS}") if(WIN32) execute_process(COMMAND java -d64 -version OUTPUT_VARIABLE JVM_IS_64_BITS) if("${JVM_IS_64_BITS}" MATCHES "Error") message(fatal_error "SimGrid can only use Java 64 bits") endif() endif() # find_package(SWIG) # if(${SWIG_FOUND}) # include(UseSWIG) # message("-- [Java] Swig found: version ${SWIG_VERSION}") # else() # message("-- [Java] Swig NOT FOUND. That's fine unless you work on this part yourself.") # endif() #mark_as_advanced(SWIG_EXECUTABLE) # Rules to build libsimgrid-java ################################ add_library(simgrid-java SHARED ${JMSG_C_SRC}) set_target_properties(simgrid-java PROPERTIES VERSION ${libsimgrid-java_version}) set_target_properties(simgrid-java PROPERTIES SKIP_BUILD_RPATH ON) target_link_libraries(simgrid-java simgrid) get_target_property(COMMON_INCLUDES simgrid-java INCLUDE_DIRECTORIES) if (COMMON_INCLUDES) set_target_properties(simgrid-java PROPERTIES INCLUDE_DIRECTORIES "${COMMON_INCLUDES};${JNI_INCLUDE_DIRS}") else() set_target_properties(simgrid-java PROPERTIES INCLUDE_DIRECTORIES "${JNI_INCLUDE_DIRS}") endif() get_target_property(CHECK_INCLUDES simgrid-java INCLUDE_DIRECTORIES) message("-- [Java] simgrid-java includes: ${CHECK_INCLUDES}") # Rules to build simgrid.jar ############################ ## Files to include in simgrid.jar ## set(SIMGRID_JAR "${CMAKE_BINARY_DIR}/simgrid.jar") set(MANIFEST_IN_FILE "${CMAKE_HOME_DIRECTORY}/src/bindings/java/MANIFEST.in") set(MANIFEST_FILE "${CMAKE_BINARY_DIR}/src/bindings/java/MANIFEST.MF") set(LIBSIMGRID_SO libsimgrid${CMAKE_SHARED_LIBRARY_SUFFIX}) set(LIBSIMGRID_JAVA_SO ${CMAKE_SHARED_LIBRARY_PREFIX}simgrid-java${CMAKE_SHARED_LIBRARY_SUFFIX}) ## Here is how to build simgrid.jar ## if(CMAKE_VERSION VERSION_LESS "2.8.12") set(CMAKE_JAVA_TARGET_OUTPUT_NAME simgrid) add_jar(simgrid-java_jar ${JMSG_JAVA_SRC}) else() add_jar(simgrid-java_jar ${JMSG_JAVA_SRC} OUTPUT_NAME simgrid) endif() add_dependencies(simgrid-java_jar simgrid-java) add_dependencies(simgrid-java_jar simgrid) if (enable_documentation) add_custom_command( TARGET simgrid-java_jar POST_BUILD COMMENT "Add the documentation into simgrid.jar..." DEPENDS ${MANIFEST_IN_FILE} ${CMAKE_HOME_DIRECTORY}/COPYING ${CMAKE_HOME_DIRECTORY}/ChangeLog ${CMAKE_HOME_DIRECTORY}/NEWS ${CMAKE_HOME_DIRECTORY}/LICENSE-LGPL-2.1 COMMAND ${CMAKE_COMMAND} -E copy ${MANIFEST_IN_FILE} ${MANIFEST_FILE} COMMAND ${CMAKE_COMMAND} -E echo "Specification-Version: \\\"${SIMGRID_VERSION_MAJOR}.${SIMGRID_VERSION_MINOR}.${SIMGRID_VERSION_PATCH}\\\"" >> ${MANIFEST_FILE} COMMAND ${CMAKE_COMMAND} -E echo "Implementation-Version: \\\"${GIT_VERSION}\\\"" >> ${MANIFEST_FILE} COMMAND ${Java_JAVADOC_EXECUTABLE} -quiet -d doc/javadoc -sourcepath ${CMAKE_HOME_DIRECTORY}/src/bindings/java/ org.simgrid.msg org.simgrid.trace COMMAND ${JAVA_ARCHIVE} -uvmf ${MANIFEST_FILE} ${SIMGRID_JAR} doc/javadoc COMMAND ${JAVA_ARCHIVE} -uvf ${SIMGRID_JAR} -C ${CMAKE_HOME_DIRECTORY} COPYING -C ${CMAKE_HOME_DIRECTORY} ChangeLog -C ${CMAKE_HOME_DIRECTORY} LICENSE-LGPL-2.1 -C ${CMAKE_HOME_DIRECTORY} NEWS ) endif() ### ### Pack the java libraries into the jarfile if asked to do so ### if(enable_lib_in_jar) set(SG_SYSTEM_NAME ${CMAKE_SYSTEM_NAME}) if(${SG_SYSTEM_NAME} MATCHES "kFreeBSD") set(SG_SYSTEM_NAME GNU/kFreeBSD) endif() set(JAVA_NATIVE_PATH NATIVE/${SG_SYSTEM_NAME}/${CMAKE_SYSTEM_PROCESSOR}) if(${CMAKE_SYSTEM_PROCESSOR} MATCHES "^i[3-6]86$") set(JAVA_NATIVE_PATH NATIVE/${SG_SYSTEM_NAME}/x86) endif() if( (${CMAKE_SYSTEM_PROCESSOR} MATCHES "x86_64") OR (${CMAKE_SYSTEM_PROCESSOR} MATCHES "AMD64") ) set(JAVA_NATIVE_PATH NATIVE/${SG_SYSTEM_NAME}/amd64) endif() if(${CMAKE_SYSTEM_PROCESSOR} MATCHES "armv7l") set(JAVA_NATIVE_PATH NATIVE/${SG_SYSTEM_NAME}/arm) # Default arm (soft-float ABI) endif() add_custom_command( TARGET simgrid-java_jar POST_BUILD COMMENT "Add the native libs into simgrid.jar..." DEPENDS simgrid simgrid-java ${JAVALIBS} COMMAND ${CMAKE_COMMAND} -E make_directory ${JAVA_NATIVE_PATH} COMMAND ${CMAKE_COMMAND} -E copy_if_different ${CMAKE_BINARY_DIR}/lib/${LIBSIMGRID_SO} ${JAVA_NATIVE_PATH}/${LIBSIMGRID_SO} COMMAND ${CMAKE_COMMAND} -E copy_if_different ${CMAKE_BINARY_DIR}/lib/${LIBSIMGRID_JAVA_SO} ${JAVA_NATIVE_PATH}/${LIBSIMGRID_JAVA_SO} # There is no way to disable the dependency of mingw-64 on that lib, unfortunately nor to script cmake -E properly # So let's be brutal and copy it in any case (even on non-windows builds) from the location where chocolatey installs it. # The copy is only expected to work on the appveyor builder, but that's all we need right now # since our users are directed to download that file as nightly build. COMMAND ${CMAKE_COMMAND} -E copy_if_different C:/tools/mingw64/bin/libwinpthread-1.dll ${JAVA_NATIVE_PATH}/libwinpthread-1.dll || true COMMAND ${JAVA_ARCHIVE} -uvf ${SIMGRID_JAR} ${JAVA_NATIVE_PATH} COMMAND ${CMAKE_COMMAND} -E echo "-- Cmake put the native code in ${JAVA_NATIVE_PATH}" COMMAND "${Java_JAVA_EXECUTABLE}" -classpath "${SIMGRID_JAR}" org.simgrid.NativeLib ) endif(enable_lib_in_jar) include_directories(${JNI_INCLUDE_DIRS} ${JAVA_INCLUDE_PATH} ${JAVA_INCLUDE_PATH2}) SimGrid-3.14.159/tools/cmake/test_prog/000755 001750 001750 00000000000 13031031274 021320 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/tools/cmake/test_prog/prog_makecontext.c000644 001750 001750 00000002110 13031031274 025027 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010, 2014, 2016. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifdef __APPLE__ #define _XOPEN_SOURCE 700 #endif #include #include #include ucontext_t uc_child; ucontext_t uc_main; static void child(void) { /* switch back to the main context */ if (swapcontext(&uc_child, &uc_main) != 0) exit(2); } int main(int argc, char *argv[]) { void *stack = malloc(64 * 1024); /* configure a child user-space context */ if (stack == NULL) exit(3); if (getcontext(&uc_child) != 0) exit(4); uc_child.uc_link = NULL; uc_child.uc_stack.ss_sp = (char *) stack + (32 * 1024); uc_child.uc_stack.ss_size = 32 * 1024; uc_child.uc_stack.ss_flags = 0; makecontext(&uc_child, child, 0); /* switch into the user context */ if (swapcontext(&uc_main, &uc_child) != 0) exit(5); /* Fine, child came home */ printf("yes\n"); /* die successfully */ return 0; } SimGrid-3.14.159/tools/cmake/test_prog/prog_sem_init.c000644 001750 001750 00000000712 13031031274 024322 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include int main(void) { sem_t s; if (sem_init(&s, 0, 0) != 0) return 1; return 0; } SimGrid-3.14.159/tools/cmake/test_prog/prog_stacksetup.c000644 001750 001750 00000003543 13031031274 024706 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010, 2014-2016. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifdef __APPLE__ #define _XOPEN_SOURCE 700 #endif #include #include #include #include union alltypes { long l; double d; void *vp; void (*fp) (void); char *cp; }; static char *handler_addr = (char *) 0xDEAD; static ucontext_t uc_handler; static ucontext_t uc_main; void handler(void) { char garbage[1024]; auto int dummy; for (int i = 0; i < 1024; i++) garbage[i] = 'X'; handler_addr = (char *) &dummy; swapcontext(&uc_handler, &uc_main); return; } int main(int argc, char *argv[]) { int sksize = 32768; char *skbuf = (char *) malloc(sksize * 2 + 2 * sizeof(union alltypes)); if (skbuf == NULL) exit(1); for (int i = 0; i < sksize * 2 + 2 * sizeof(union alltypes); i++) skbuf[i] = 'A'; char *skaddr = skbuf + sizeof(union alltypes); if (getcontext(&uc_handler) != 0) exit(1); uc_handler.uc_link = NULL; uc_handler.uc_stack.ss_sp = (void *) (skaddr + sksize); uc_handler.uc_stack.ss_size = sksize; uc_handler.uc_stack.ss_flags = 0; makecontext(&uc_handler, handler, 0); swapcontext(&uc_main, &uc_handler); if (handler_addr == (char *) 0xDEAD) exit(1); if (handler_addr < skaddr + sksize) { /* stack was placed into lower area */ if (*(skaddr + sksize) != 'A') printf("(skaddr)+(sksize)-%d;(sksize)-%d", sizeof(union alltypes), sizeof(union alltypes)); else printf("(skaddr)+(sksize);(sksize)"); } else { /* stack was placed into higher area */ if (*(skaddr + sksize * 2) != 'A') printf("(skaddr);(sksize)-%d", sizeof(union alltypes)); else printf("(skaddr);(sksize)"); } return 0; } SimGrid-3.14.159/tools/cmake/test_prog/prog_sem_timedwait.c000644 001750 001750 00000000600 13031031274 025342 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010, 2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include int main() { sem_t *s; const struct timespec *t; sem_timedwait(s, t); } SimGrid-3.14.159/tools/cmake/test_prog/prog_gnu_dynlinker.c000644 001750 001750 00000001762 13031031274 025371 0ustar00mquinsonmquinson000000 000000 /* prog_gnu_dynlinker.c -- check that RTLD_NEXT is defined as in GNU linker */ /* Copyright (c) 2012-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #define _GNU_SOURCE 1 #include #include #include #include static void * (*real_malloc) (size_t); int main(void) { char *error; dlerror(); // clear any previous error real_malloc = (void * (*) (size_t)) dlsym(RTLD_NEXT, "malloc"); error = dlerror(); if (!error && real_malloc) { char *A = real_malloc(20); strcpy(A,"epic success"); free(A); return 0; // SUCCESS } else { if (error) printf("Error while checking for dlsym: %s\n",error); else printf("dlsym did not return any error, but failed to find malloc()\n"); return 1; // FAILED } } SimGrid-3.14.159/tools/cmake/test_prog/prog_vsnprintf.c000644 001750 001750 00000001501 13031031274 024541 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010, 2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include int my_vsnprintf(char *buf, const char *tmpl, ...) { int i; va_list args; va_start(args, tmpl); i = vsnprintf(buf, 2, tmpl, args); va_end(args); return i; } int main(void) { char bufs[5] = { 'x', 'x', 'x', '\0', '\0' }; char bufd[5] = { 'x', 'x', 'x', '\0', '\0' }; int i; i = my_vsnprintf(bufs, "%s", "111"); if (strcmp(bufs, "1")) exit(1); if (i != 3) exit(1); i = my_vsnprintf(bufd, "%d", 111); if (strcmp(bufd, "1")) exit(1); if (i != 3) exit(1); exit(0); } SimGrid-3.14.159/tools/cmake/test_prog/prog_mutex_timedlock.c000644 001750 001750 00000000642 13031031274 025712 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010-2011, 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include int main(void) { pthread_mutex_t s; const struct timespec t; sem_timedlock(&s, &t); return 0; } SimGrid-3.14.159/tools/cmake/test_prog/prog_thread_storage.c000644 001750 001750 00000000715 13031031274 025511 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010-2011, 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include __thread int thread_specific_variable = 0; int main(void) { thread_specific_variable++; printf("%d\n", thread_specific_variable); return 0; } SimGrid-3.14.159/tools/cmake/test_prog/prog_snprintf.c000644 001750 001750 00000001201 13031031274 024350 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010, 2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include int main(void) { char bufs[5] = { 'x', 'x', 'x', '\0', '\0' }; char bufd[5] = { 'x', 'x', 'x', '\0', '\0' }; int i; i = snprintf(bufs, 2, "%s", "111"); if (strcmp(bufs, "1")) exit(1); if (i != 3) exit(1); i = snprintf(bufd, 2, "%d", 111); if (strcmp(bufd, "1")) exit(1); if (i != 3) exit(1); exit(0); } SimGrid-3.14.159/tools/cmake/test_prog/prog_stackgrowth.c000644 001750 001750 00000001751 13031031274 025057 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010, 2014-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include static int iterate = 10; static int growsdown(int *x) { int y = (x > &y); if (--iterate > 0) y = growsdown(&y); /* The stack sometimes changes at the 0th level. * Original version did fail in this case, but I changed this around SimGrid 3.13 because of https://bugs.debian.org/814272 * Every arch failed on that day :( */ if (iterate != 0 && y != (x > &y)) { fprintf(stderr, "The stack changed its direction! (Iteration: %d. It was growing %s; &y=%p; &prevY=%p)\n", (10-iterate), y?"down":"up", &y, x); exit(1); } return y; } int main(int argc, char *argv[]) { int x; printf("%s", growsdown(&x) ? "down" : "up"); return 0; } SimGrid-3.14.159/tools/cmake/test_prog/prog_sem_open.c000644 001750 001750 00000001252 13031031274 024320 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #ifndef SEM_FAILED #define SEM_FAILED (-1) #endif int main(void) { #ifdef WIN32 int s; #else sem_t * s; #endif s = sem_open("/0", O_CREAT, 0644, 10); if (s == SEM_FAILED){ // printf("sem_open failed\n"); return 1; } // printf("sem_open succeeded\n"); return 0; } SimGrid-3.14.159/tools/cmake/cross-mingw.cmake000644 001750 001750 00000001047 13031031274 022566 0ustar00mquinsonmquinson000000 000000 # cmake -DCMAKE_TOOLCHAIN_FILE=tools/cmake/cross-mingw.cmake .. set (CMAKE_SYSTEM_NAME Windows) set (CMAKE_SYSTEM_PROCESSOR x86_64) set (CMAKE_C_COMPILER /usr/bin/x86_64-w64-mingw32-gcc-win32) set (CMAKE_CXX_COMPILER /usr/bin/x86_64-w64-mingw32-g++-win32) set (CMAKE_Fortran_COMPILER /usr/bin/x86_64-w64-mingw32-gfortran-win32) set (CMAKE_FIND_ROOT_PATH /usr/lib/gcc/x86_64-w64-mingw32/ /usr/i686-w64-mingw32/) set (CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER) set (CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY) set (CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) SimGrid-3.14.159/tools/cmake/MakeLib.cmake000644 001750 001750 00000007661 13031031273 021631 0ustar00mquinsonmquinson000000 000000 ### Make Libs ############################### # Declare the library content # ############################### # Actually declare our libraries add_library(simgrid SHARED ${simgrid_sources}) set_target_properties(simgrid PROPERTIES VERSION ${libsimgrid_version}) add_dependencies(simgrid maintainer_files) if(enable_model-checking) add_executable(simgrid-mc ${MC_SIMGRID_MC_SRC}) target_link_libraries(simgrid-mc simgrid) set_target_properties(simgrid-mc PROPERTIES RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin") endif() # Compute the dependencies of SimGrid ##################################### if (HAVE_BOOST_CONTEXTS) set(SIMGRID_DEP "${SIMGRID_DEP} ${Boost_CONTEXT_LIBRARY}") endif() if(HAVE_PTHREAD AND ${HAVE_THREAD_CONTEXTS} AND NOT APPLE) # Clang on recent Mac OS X is not happy about -pthread. SET(SIMGRID_DEP "${SIMGRID_DEP} -pthread") endif() if(HAVE_LUA) ADD_CUSTOM_TARGET(link_simgrid_lua ALL DEPENDS simgrid ${CMAKE_BINARY_DIR}/examples/lua/simgrid.${LIB_EXE} ${CMAKE_BINARY_DIR}/examples/msg/masterslave/simgrid.${LIB_EXE} ${CMAKE_BINARY_DIR}/examples/simdag/simgrid.${LIB_EXE} ) add_custom_command( OUTPUT ${CMAKE_BINARY_DIR}/examples/lua/simgrid.${LIB_EXE} ${CMAKE_BINARY_DIR}/examples/msg/masterslave/simgrid.${LIB_EXE} ${CMAKE_BINARY_DIR}/examples/simdag/simgrid.${LIB_EXE} COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_BINARY_DIR}/examples/lua/simgrid.${LIB_EXE} # if it exists, creating the link fails. So cleanup before hand COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_BINARY_DIR}/examples/lua/ COMMAND ${CMAKE_COMMAND} -E create_symlink ${CMAKE_BINARY_DIR}/lib/libsimgrid.${LIB_EXE} ${CMAKE_BINARY_DIR}/examples/lua/simgrid.${LIB_EXE} #for test COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_BINARY_DIR}/examples/msg/masterslave/simgrid.${LIB_EXE} # if it exists, creating the link fails. So cleanup before hand COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_BINARY_DIR}/examples/msg/masterslave/ COMMAND ${CMAKE_COMMAND} -E create_symlink ${CMAKE_BINARY_DIR}/lib/libsimgrid.${LIB_EXE} ${CMAKE_BINARY_DIR}/examples/msg/masterslave/simgrid.${LIB_EXE} #for test COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_BINARY_DIR}/examples/simdag/simgrid.${LIB_EXE} # if it exists, creating the link fails. So cleanup before hand COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_BINARY_DIR}/examples/simdag/ COMMAND ${CMAKE_COMMAND} -E create_symlink ${CMAKE_BINARY_DIR}/lib/libsimgrid.${LIB_EXE} ${CMAKE_BINARY_DIR}/examples/simdag/simgrid.${LIB_EXE} #for test ) SET(SIMGRID_DEP "${SIMGRID_DEP} ${LUA_LIBRARY} ${DL_LIBRARY}") endif() if(HAVE_PAPI) SET(SIMGRID_DEP "${SIMGRID_DEP} -lpapi") endif() if(HAVE_GRAPHVIZ) if(HAVE_CGRAPH_LIB) SET(SIMGRID_DEP "${SIMGRID_DEP} -lcgraph") else() if(HAVE_AGRAPH_LIB) SET(SIMGRID_DEP "${SIMGRID_DEP} -lagraph -lcdt") endif() endif() endif() if(HAVE_MC AND HAVE_GNU_LD) SET(SIMGRID_DEP "${SIMGRID_DEP} ${DL_LIBRARY}") endif() if(HAVE_NS3) SET(SIMGRID_DEP "${SIMGRID_DEP} -lns${NS3_VERSION}-core -lns${NS3_VERSION}-csma -lns${NS3_VERSION}-point-to-point -lns${NS3_VERSION}-internet -lns${NS3_VERSION}-applications") endif() if(HAVE_POSIX_GETTIME) SET(SIMGRID_DEP "${SIMGRID_DEP} -lrt") endif() if("${CMAKE_SYSTEM}" MATCHES "FreeBSD") set(SIMGRID_DEP "${SIMGRID_DEP} -lprocstat") endif() # Compute the dependencies of SMPI ################################## if(enable_smpi AND APPLE) set(SIMGRID_DEP "${SIMGRID_DEP} -Wl,-U -Wl,_smpi_simulated_main") endif() target_link_libraries(simgrid ${SIMGRID_DEP}) # Dependencies from maintainer mode ################################### if(enable_maintainer_mode) add_dependencies(simgrid smpi_generated_headers_call_location_tracing) endif() if(enable_maintainer_mode AND PYTHON_EXE) add_dependencies(simgrid simcalls_generated_src) endif() if(enable_maintainer_mode AND BISON_EXE AND LEX_EXE) add_dependencies(simgrid automaton_generated_src) endif() SimGrid-3.14.159/tools/cmake/MaintainerMode.cmake000644 001750 001750 00000031756 13031031273 023223 0ustar00mquinsonmquinson000000 000000 # Change the following when we need a recent enough version of flexml to get the maintainer mode working set(FLEXML_MIN_MAJOR 1) set(FLEXML_MIN_MINOR 9) set(FLEXML_MIN_PATCH 6) # Change the following when we need a recent enough version of flex to get the maintainer mode working. # Maintainer mode does NOT work with version 2.5.35! set(FLEX_MIN_MAJOR 2) set(FLEX_MIN_MINOR 5) set(FLEX_MIN_PATCH 39) # the rest should only be changed if you understand what you're doing if(enable_maintainer_mode AND NOT WIN32) if (PYTHON_EXECUTABLE) add_custom_command( OUTPUT ${CMAKE_HOME_DIRECTORY}/src/simix/popping_generated.cpp ${CMAKE_HOME_DIRECTORY}/src/simix/popping_bodies.cpp ${CMAKE_HOME_DIRECTORY}/src/simix/popping_enum.h ${CMAKE_HOME_DIRECTORY}/src/simix/popping_accessors.h DEPENDS ${CMAKE_HOME_DIRECTORY}/src/simix/simcalls.py ${CMAKE_HOME_DIRECTORY}/src/simix/simcalls.in COMMENT "Generating simcalls source files" COMMAND ${PYTHON_EXECUTABLE} simcalls.py WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY}/src/simix/ ) add_custom_target(simcalls_generated_src DEPENDS ${CMAKE_HOME_DIRECTORY}/src/simix/popping_generated.cpp ${CMAKE_HOME_DIRECTORY}/src/simix/popping_bodies.cpp ${CMAKE_HOME_DIRECTORY}/src/simix/popping_enum.h ${CMAKE_HOME_DIRECTORY}/src/simix/popping_accessors.h ) SET_DIRECTORY_PROPERTIES(PROPERTIES ADDITIONAL_MAKE_CLEAN_FILES "${CMAKE_HOME_DIRECTORY}/src/simix/popping_enum.h;${CMAKE_HOME_DIRECTORY}/src/simix/popping_generated.cpp;${CMAKE_HOME_DIRECTORY}/src/simix/popping_bodies.cpp;${CMAKE_HOME_DIRECTORY}/src/simix/popping_accessors.h" ) endif() endif() # Let's generate header files required by SMPI when the call location tracing # has been activated. if(enable_maintainer_mode AND NOT WIN32) add_custom_command(OUTPUT ${CMAKE_HOME_DIRECTORY}/include/smpi/smpi_extended_traces.h ${CMAKE_HOME_DIRECTORY}/include/smpi/smpi_extended_traces_fortran.h COMMENT "Generating header files for call-location tracing with SMPI" # Make sure there is no space after the redirection operator (>). I received # error messages in that case on my Debian system. COMMAND "${CMAKE_HOME_DIRECTORY}/tools/smpi/generate_smpi_defines.pl" "${CMAKE_HOME_DIRECTORY}/include/smpi/smpi.h >${CMAKE_HOME_DIRECTORY}/include/smpi/smpi_extended_traces.h" COMMAND "${CMAKE_HOME_DIRECTORY}/tools/smpi/generate_smpi_defines.pl" "-f" "${CMAKE_HOME_DIRECTORY}/include/smpi/smpi.h >${CMAKE_HOME_DIRECTORY}/include/smpi/smpi_extended_traces_fortran.h" ) add_custom_target(smpi_generated_headers_call_location_tracing DEPENDS ${CMAKE_HOME_DIRECTORY}/include/smpi/smpi_extended_traces.h ${CMAKE_HOME_DIRECTORY}/include/smpi/smpi_extended_traces_fortran.h ) SET_DIRECTORY_PROPERTIES(PROPERTIES ADDITIONAL_MAKE_CLEAN_FILES "${CMAKE_HOME_DIRECTORY}/include/smpi/smpi_extended_traces.h;${CMAKE_HOME_DIRECTORY}/include/smpi/smpi_extended_traces_fortran.h" ) endif() if(enable_maintainer_mode AND NOT WIN32) find_program(FLEX_EXE NAMES flex) find_program(FLEXML_EXE NAMES flexml) find_program(SED_EXE NAMES sed) find_program(BISON_EXE NAMES bison) find_program(LEX_EXE NAMES lex) mark_as_advanced(BISON_EXE) mark_as_advanced(LEX_EXE) if(BISON_EXE AND LEX_EXE) add_custom_command( OUTPUT ${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/automaton_lexer.yy.c ${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/parserPromela.tab.cacc ${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/parserPromela.tab.hacc DEPENDS ${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/parserPromela.lex ${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/parserPromela.yacc COMMENT "Generating automaton source files" COMMAND ${BISON_EXE} --name-prefix=xbt_automaton_parser_ -d parserPromela.yacc COMMAND ${LEX_EXE} --prefix=xbt_automaton_parser_ --outfile=automaton_lexer.yy.c parserPromela.lex WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/ ) add_custom_target(automaton_generated_src DEPENDS ${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/automaton_lexer.yy.c ${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/parserPromela.tab.cacc ${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/parserPromela.tab.hacc ) SET_DIRECTORY_PROPERTIES(PROPERTIES ADDITIONAL_MAKE_CLEAN_FILES "${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/parserPromela.tab.cacc;${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/parserPromela.tab.hacc;${CMAKE_HOME_DIRECTORY}/src/xbt/automaton/automaton_parse.yy.c" ) endif() message(STATUS "Found flex: ${FLEX_EXE}") IF(FLEX_EXE) set(HAVE_FLEX 1) execute_process(COMMAND ${FLEX_EXE} --version OUTPUT_VARIABLE FLEX_VERSION) string(REGEX MATCH "[0-9]+[.]+[0-9]+[.]+[0-9]+" FLEX_VERSION "${FLEX_VERSION}") string(REGEX MATCH "^[0-9]+" FLEX_MAJOR_VERSION "${FLEX_VERSION}") string(REGEX MATCH "[0-9]+[.]+[0-9]+$" FLEX_VERSION "${FLEX_VERSION}") string(REGEX MATCH "^[0-9]+" FLEX_MINOR_VERSION "${FLEX_VERSION}") string(REGEX MATCH "[0-9]+$" FLEX_PATCH_VERSION "${FLEX_VERSION}") ENDIF() message(STATUS "Found flexml: ${FLEXML_EXE}") IF(FLEXML_EXE) set(HAVE_FLEXML 1) execute_process(COMMAND ${FLEXML_EXE} --version OUTPUT_VARIABLE FLEXML_VERSION) if (FLEXML_VERSION MATCHES "version Id:") message(FATAL_ERROR "You have an ancient flexml version (${FLEXML_VERSION}). You need at least v${FLEXML_MIN_MAJOR}.${FLEXML_MIN_MINOR}.${FLEXML_MIN_PATCH} to compile in maintainer mode. Upgrade your flexml, or disable the Maintainer mode option in cmake.") endif() string(REGEX MATCH "[0-9]+[.]+[0-9]+[.]+[0-9]+" FLEXML_VERSION "${FLEXML_VERSION}") string(REGEX MATCH "^[0-9]*" FLEXML_MAJOR_VERSION "${FLEXML_VERSION}") string(REGEX MATCH "[0-9]+[.]+[0-9]+$" FLEXML_VERSION "${FLEXML_VERSION}") string(REGEX MATCH "^[0-9]+" FLEXML_MINOR_VERSION "${FLEXML_VERSION}") string(REGEX MATCH "[0-9]+$" FLEXML_PATCH_VERSION "${FLEXML_VERSION}") ENDIF() message(STATUS "Found sed: ${SED_EXE}") if(HAVE_FLEXML AND HAVE_FLEX AND SED_EXE) message(STATUS "Flex version: ${FLEX_MAJOR_VERSION}.${FLEX_MINOR_VERSION}.${FLEX_PATCH_VERSION} (need at least version ${FLEX_MIN_MAJOR}.${FLEX_MIN_MINOR}.${FLEX_MIN_PATCH})") message(STATUS "Flexml version: ${FLEXML_MAJOR_VERSION}.${FLEXML_MINOR_VERSION}.${FLEXML_PATCH_VERSION} (need at least version ${FLEXML_MIN_MAJOR}.${FLEXML_MIN_MINOR}.${FLEXML_MIN_PATCH})") IF( (${FLEXML_MAJOR_VERSION} LESS ${FLEXML_MIN_MAJOR}) OR ((${FLEXML_MAJOR_VERSION} EQUAL ${FLEXML_MIN_MAJOR}) AND (${FLEXML_MINOR_VERSION} LESS ${FLEXML_MIN_MINOR}) ) OR ( (${FLEXML_MAJOR_VERSION} EQUAL ${FLEXML_MIN_MAJOR}) AND (${FLEXML_MINOR_VERSION} EQUAL ${FLEXML_MIN_MINOR}) AND (${FLEXML_PATCH_VERSION} LESS ${FLEXML_MIN_PATCH}) )) message(FATAL_ERROR "Your flexml version is too old to compile in maintainer mode (need at least v${FLEXML_MIN_MAJOR}.${FLEXML_MIN_MINOR}.${FLEXML_MIN_PATCH}). Upgrade your flexml, or disable the Maintainer mode option in cmake.") ENDIF() IF( (${FLEX_MAJOR_VERSION} LESS ${FLEX_MIN_MAJOR}) OR ((${FLEX_MAJOR_VERSION} EQUAL ${FLEX_MIN_MAJOR}) AND (${FLEX_MINOR_VERSION} LESS ${FLEX_MIN_MINOR}) ) OR ( (${FLEX_MAJOR_VERSION} EQUAL ${FLEX_MIN_MAJOR}) AND (${FLEX_MINOR_VERSION} EQUAL ${FLEX_MIN_MINOR}) AND (${FLEX_PATCH_VERSION} LESS ${FLEX_MIN_PATCH}) )) message(FATAL_ERROR "Your flex version is too old to compile in maintainer mode (need at least v${FLEX_MIN_MAJOR}.${FLEX_MIN_MINOR}.${FLEX_MIN_PATCH}). Upgrade your flex, or disable the Maintainer mode option in cmake (run 'ccmake').") ENDIF() set(string1 "'s/extern *\\([^(]*\\)\\( \\|\\( \\*\\)\\)/XBT_PUBLIC_DATA(\\1\\3) /'") set(string2 "'s/XBT_PUBLIC_DATA(\\([^)]*\\)) *\\([^(]*\\)(/XBT_PUBLIC(\\1) \\2(/'") set(string5 "'s/SET(DOCTYPE)/SET(ROOT_dax__adag)/'") set(string9 "'s/#include /#if defined(_WIN32)\\n# ifndef __STRICT_ANSI__\\n# include \\n# include \\n# endif\\n#else\\n# include \\n#endif/g'") set(string14 "'\\!^ \\* Generated [0-9/]\\{10\\} [0-9:]\\{8\\}\\.$$!d'") set(string15 "'s/FAIL(\"Premature EOF/if(!ETag_surfxml_include_state()) FAIL(\"Premature EOF/'") ADD_CUSTOM_COMMAND( OUTPUT ${CMAKE_HOME_DIRECTORY}/src/surf/xml/simgrid_dtd.h ${CMAKE_HOME_DIRECTORY}/src/simdag/dax_dtd.h ${CMAKE_HOME_DIRECTORY}/src/surf/xml/simgrid_dtd.c ${CMAKE_HOME_DIRECTORY}/src/simdag/dax_dtd.c DEPENDS ${CMAKE_HOME_DIRECTORY}/src/surf/xml/simgrid.dtd ${CMAKE_HOME_DIRECTORY}/src/simdag/dax.dtd #${CMAKE_HOME_DIRECTORY}/src/surf/xml/simgrid_dtd.l: ${CMAKE_HOME_DIRECTORY}/src/surf/xml/simgrid.dtd COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_HOME_DIRECTORY}/src/surf/xml COMMAND ${FLEXML_EXE} --root-tags platform -b 1000000 -P surfxml --sysid=http://simgrid.gforge.inria.fr/simgrid/simgrid.dtd -S src/surf/xml/simgrid_dtd.l -L src/surf/xml/simgrid.dtd COMMAND ${SED_EXE} -i ${string14} src/surf/xml/simgrid_dtd.l COMMAND ${SED_EXE} -i "'s/FAIL(\"Bad declaration %s.\",yytext)/FAIL(\"Bad declaration %s.\\\\nIf your are using a XML v3 file (check the version attribute in ), please update it with tools\\/simgrid_update_xml.pl\",yytext)/'" src/surf/xml/simgrid_dtd.l COMMAND ${CMAKE_COMMAND} -E echo " Generated src/surf/xml/simgrid_dtd.l" #${CMAKE_HOME_DIRECTORY}/src/simdag/dax_dtd.l: ${CMAKE_HOME_DIRECTORY}/src/simdag/dax.dtd COMMAND ${FLEXML_EXE} -b 1000000 --root-tags adag -P dax_ --sysid=dax.dtd -S src/simdag/dax_dtd.l -L src/simdag/dax.dtd COMMAND ${SED_EXE} -i ${string5} src/simdag/dax_dtd.l COMMAND ${SED_EXE} -i ${string14} src/simdag/dax_dtd.l COMMAND ${CMAKE_COMMAND} -E echo " Generated src/simdag/dax_dtd.l" #${CMAKE_HOME_DIRECTORY}/src/surf/xml/simgrid_dtd.h: ${CMAKE_HOME_DIRECTORY}/src/surf/xml/simgrid.dtd COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_HOME_DIRECTORY}/include/surf/simgrid.h COMMAND ${FLEXML_EXE} --root-tags platform -P surfxml --sysid=http://simgrid.gforge.inria.fr/simgrid/simgrid.dtd -H src/surf/xml/simgrid_dtd.h -L src/surf/xml/simgrid.dtd COMMAND ${SED_EXE} -i ${string1} src/surf/xml/simgrid_dtd.h COMMAND ${SED_EXE} -i ${string2} src/surf/xml/simgrid_dtd.h COMMAND ${SED_EXE} -i ${string14} src/surf/xml/simgrid_dtd.h COMMAND ${CMAKE_COMMAND} -E echo " Generated src/surf/xml/simgrid_dtd.h" #${CMAKE_HOME_DIRECTORY}/src/simdag/dax_dtd.h: ${CMAKE_HOME_DIRECTORY}/src/simdag/dax.dtd COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_HOME_DIRECTORY}/src/simdag/dax_dtd.h COMMAND ${FLEXML_EXE} --root-tags adag -P dax_ --sysid=dax.dtd -H src/simdag/dax_dtd.h -L src/simdag/dax.dtd COMMAND ${SED_EXE} -i ${string1} src/simdag/dax_dtd.h COMMAND ${SED_EXE} -i ${string2} src/simdag/dax_dtd.h COMMAND ${SED_EXE} -i ${string14} src/simdag/dax_dtd.h COMMAND ${CMAKE_COMMAND} -E echo " Generated src/simdag/dax_dtd.h" #surf/xml/simgrid_dtd.c: surf/xml/simgrid_dtd.l COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_HOME_DIRECTORY}/src/surf/xml/simgrid_dtd.c COMMAND ${FLEX_EXE} -o src/surf/xml/simgrid_dtd.c -Psurf_parse_ --noline src/surf/xml/simgrid_dtd.l COMMAND ${SED_EXE} -i ${string9} src/surf/xml/simgrid_dtd.c COMMAND ${SED_EXE} -i ${string15} src/surf/xml/simgrid_dtd.c COMMAND ${SED_EXE} -i 's/int yyl\;/unsigned int yyl\;/' src/surf/xml/simgrid_dtd.c COMMAND ${SED_EXE} -i "s/register //" src/surf/xml/simgrid_dtd.c COMMAND ${CMAKE_COMMAND} -E echo " Generated surf/xml/simgrid_dtd.c" #simdag/dax_dtd.c: simdag/dax_dtd.l COMMAND ${CMAKE_COMMAND} -E remove -f ${CMAKE_HOME_DIRECTORY}/src/simdag/dax_dtd.c COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_HOME_DIRECTORY}/src/simdag COMMAND ${FLEX_EXE} -o src/simdag/dax_dtd.c -Pdax_ --noline src/simdag/dax_dtd.l COMMAND ${SED_EXE} -i ${string9} src/simdag/dax_dtd.c COMMAND ${SED_EXE} -i 's/int yyl\;/unsigned int yyl\;/' src/simdag/dax_dtd.c COMMAND ${CMAKE_COMMAND} -E echo " Generated src/simdag/dax_dtd.c" WORKING_DIRECTORY ${CMAKE_HOME_DIRECTORY} COMMENT "Generating files in maintainer mode..." ) else() if(NOT HAVE_FLEXML) message(STATUS "Error : Install flexml to use the maintainer mode.") endif() if(NOT HAVE_FLEX) message(STATUS "Error : Install flex to use the maintainer mode.") endif() if(NOT SED_EXE) message(STATUS "Error : Install sed before use maintainer mode.") endif() message(FATAL_ERROR STATUS "Error : Need to install all tools for maintainer mode !!!") endif() endif() add_custom_target(maintainer_files DEPENDS ${CMAKE_HOME_DIRECTORY}/src/surf/xml/simgrid_dtd.h ${CMAKE_HOME_DIRECTORY}/src/surf/xml/simgrid_dtd.c ${CMAKE_HOME_DIRECTORY}/src/simdag/dax_dtd.h ${CMAKE_HOME_DIRECTORY}/src/simdag/dax_dtd.c ) SimGrid-3.14.159/tools/cmake/scripts/000755 001750 001750 00000000000 13031031274 021001 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/tools/cmake/scripts/update_tesh.pl000755 001750 001750 00000002154 13031031274 023650 0ustar00mquinsonmquinson000000 000000 #!/usr/bin/env perl # Copyright (c) 2012, 2014. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. use strict; use warnings; if($#ARGV!=1) { die "Usage: perl update_tesh.pl \n"; } my($directory)=$ARGV[0]; my($old)=$ARGV[1]; chdir("$directory"); open SH_LIGNE, $old or die "Unable to open $old. $!\n"; my($line); my($line_exec); my($l); my($tmp); while(defined($line=)) { if($line =~ /^\$(.*)$/) { $line_exec = $line; $line =~ s/\$\{srcdir\:\=\.\}/./g; $line =~ s/\(/\\(/g; $line =~ s/\)/\\)/g; $line =~ s/\$SG_TEST_EXENV//g; $line =~ s/\$EXEEXT//g; $line =~ s/^\$\ */.\//g; $line =~ s/^.\/lua/lua/g; $line =~ s/^.\/ruby/ruby/g; $line =~ s/--log=([^ ]*)/--log="$1"/g; print "$line_exec"; chomp $line; open (FILE, "$line 2>&1|"); while(defined($l=)) { chomp $l; print "\> $l\n"; } close(FILE); } else { if($line =~ /^\>(.*)$/) { } else { print "$line"; } } } close(SH_LIGNE); SimGrid-3.14.159/tools/cmake/scripts/my_valgrind.pl000755 001750 001750 00000001310 13031031274 023647 0ustar00mquinsonmquinson000000 000000 #!/usr/bin/env perl # Copyright (c) 2012-2016. The SimGrid Team. All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. use strict; use warnings; my @argv = ("valgrind"); my $count = 0; while (defined(my $arg = shift)) { # print "arg($count)$arg\n"; if ($arg =~ m!/smpirun$!) { @argv = ( $arg, "-wrapper", "@argv" ); } elsif ($arg eq "--cd") { $arg = shift; print "cd $arg\n"; chdir ($arg); $count++; } else { push @argv, $arg; } $count++; } # print "COMMAND:"; # map { print "$_ " } @argv; # print "\n"; system @argv; SimGrid-3.14.159/tools/simgrid_update_xml.pl000755 001750 001750 00000014610 13031031276 022456 0ustar00mquinsonmquinson000000 000000 #! /usr/bin/env perl eval 'exec perl -S $0 ${1+"$@"}' if $running_under_some_shell; # This script updates the simgrid XML file passed as argument (modification in place) # It is built to do the conversion incrementally. # Copyright (c) 2006-2016. The SimGrid Team. # All rights reserved. # # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. =encoding UTF-8 =head1 NAME simgrid_update_xml - updates simgrid XML files to latest version =head1 SYNOPSIS B I =head1 DESCRIPTION simgrid_update_xml updates the simgrid XML file passed as argument. The file is modified in place, without any kind of backup. You may want to save a copy before running the script. In SimGrid XML files, the standard version is indicated in the version attribute of the platform tag. Current version is 4. Here is a list of major changes in each version. =over 4 =item B Used before SimGrid 3.3 =item B Introduced in SimGrid 3.3 =over 4 =item The version attribute of platform were added to allow file versioning. =item The link bandwidth changed from Mb/s to b/s; and the CPU power were changed from MFlop/s to Flop/s =back =item B Introduced in SimGrid 3.4 =over =item Several tags were renamed: CPU -> HOST NETWORK_LINK -> LINK ROUTE_ELEMENT -> LINK_CTN PLATFORM_DESCRIPTION -> PLATFORM =back =item B Introduced in SimGrid 3.5 =over 4 =item The AS tag were introduced. Every platform should now contain an englobing AS tag. =item Routes are now symmetric by default. =item Several tags were renamed (for sake of XML sanity): LINK:CTN -> LINK_CTN TRACE:CONNECT -> TRACE_CONNECT =back =item B Introduced in SimGrid 3.13 (this is the current version) =over 4 =item Rename the attributes describing the amount of flop that a host / peer / cluster / cabinet can deliver per second. -> =item In , attribute kind="POWER" is now kind="SPEED". =item The DOCTYPE points to the right URL: http://simgrid.gforge.inria.fr/simgrid/simgrid.dtd =item Units are now mandatory in attributes. USE THE SCRIPT sg_xml_unit_converter.py TO CONVERT THIS - speed. Old default: 'f' or 'flops'. Also defined: 'Yf', 'Zf', 'Ef', 'Pf', 'Tf', 'Gf', 'Mf', 'kf' 'yottaflops', 'zettaflops', 'exaflops', 'petaflops', 'teraflops', 'gigaflops', 'megaflops', 'kiloflops' - bandwidth. Old default: 'Bps' bytes per second (or 'bps' but 1 Bps = 8 bps) Also defined in bytes: 'TiBps', 'GiBps', 'MiBps', 'KiBps', 'TBps', 'GBps', 'MBps', 'kBps', 'Bps' And the same in bits: 'Tibps', 'Gibps', 'Mibps', 'Kibps', 'Tbps', 'Gbps', 'Mbps', 'kbps', 'bps' - latency. Old default: 's' second. Also defined: 'w' week, 'd' day, 'h' hour, 'm' minute, 'ms' millisecond, 'us' microsecond, 'ns' nanosecond, 'ps' picosecond =back =back =head1 AUTHORS The SimGrid team (simgrid-devel@lists.gforge.inria.fr) =head1 COPYRIGHT AND LICENSE Copyright (c) 2006-2016. The SimGrid Team. All rights reserved. This program is free software; you may redistribute it and/or modify it under the terms of GNU LGPL (v2.1) license. =cut use strict; my $fromversion=-1; my $toversion=4; my $filename = $ARGV[0] or die "Usage: simgrid_update_xml.pl file_to_convert.xml\nPlease provide an XML to convert as a parameter.\n"; open INPUT, "$filename" or die "Cannot open input file $filename: $!\n"; my $output_string = "\n". "\n". "\n"; my($AS_opened)=0; my $line; while (defined($line = )) { chomp $line; # eat the header, whatever form it has next if ($line =~ s/<\?xml[^>]*>// && ! $line =~ /\S/); # just in case several tags are on the same line next if ($line =~ s/]*>// && ! $line =~ /\S/); if ($line =~ s///) { $fromversion = 0; print "$filename was using version 0\n"; next if !$line =~ /\S/; } elsif ($line =~ s///) { $fromversion = $1; if ($fromversion == $toversion) { die "Input platform file $filename is already conformant to version $fromversion. This should be a no-op.\n"; } if ($fromversion > $toversion) { die "Input platform file $filename is more recent than this script (file version: $fromversion; script version: $toversion)\n"; } next if !$line =~ /\S/; print "$filename was using version $fromversion\n"; } if ($fromversion == 0) { while ($line =~ m|^(.*?)/) || ($line=~ /\n"; $AS_opened=1; } if($line=~/ $filename"; print OUTPUT $output_string; close OUTPUT; SimGrid-3.14.159/CITATION.bib000644 001750 001750 00000000745 13031031275 016771 0ustar00mquinsonmquinson000000 000000 @Article{simgrid, hal_id = {hal-01017319}, url = {http://hal.inria.fr/hal-01017319}, title = {Versatile, Scalable, and Accurate Simulation of Distributed Applications and Platforms}, author = {Casanova, Henri and Giersch, Arnaud and Legrand, Arnaud and Quinson, Martin and Suter, Fr{\'e}d{\'e}ric}, publisher = {Elsevier}, pages = {2899-2917}, journal = {Journal of Parallel and Distributed Computing}, volume = {74}, number = {10}, year = {2014}, month = Jun, } SimGrid-3.14.159/TODO000644 001750 001750 00000001732 13031031306 015561 0ustar00mquinsonmquinson000000 000000 ************************************************ *** This file is a TODO. It is thus kinda *** *** outdated. You know the story, right? *** ************************************************ ### ### Urgent stuff: ### * Have a proper todo file ### ### Ongoing stuff ### * Clean up CMake files Non exhaustive list of subgoals: - Use genuine cmake mechanisms and variables when available, instead of reinventing the wheel. - Correctly determine system and architecture (e.g. x32). - Correctly determine compiler type and version (e.g. clang). - Use git to generate the dist archive. Either use git-archive to generate the tarball, or keep using cmake -E tar, but use git-ls-files to get the list of files to pack. * Replace XBT with the C++ standard library * Some XBT datacontainers seem to overlap. Kill some/all of them. - replace fifo with dynars - replace set with SWAG SimGrid-3.14.159/src/000755 001750 001750 00000000000 13031031273 015660 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/src/mc/000755 001750 001750 00000000000 13031031273 016257 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/src/mc/LivenessChecker.cpp000644 001750 001750 00000041420 13031031270 022036 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2011-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include #include #include #include #include #include "src/mc/mc_request.h" #include "src/mc/LivenessChecker.hpp" #include "src/mc/mc_private.h" #include "src/mc/mc_record.h" #include "src/mc/mc_smx.h" #include "src/mc/Client.hpp" #include "src/mc/mc_private.h" #include "src/mc/mc_replay.h" #include "src/mc/mc_safety.h" #include "src/mc/mc_exit.h" #include "src/mc/Transition.hpp" #include "src/mc/Session.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_liveness, mc, "Logging specific to algorithms for liveness properties verification"); /********* Static functions *********/ namespace simgrid { namespace mc { VisitedPair::VisitedPair( int pair_num, xbt_automaton_state_t automaton_state, std::shared_ptr> atomic_propositions, std::shared_ptr graph_state) { simgrid::mc::Process* process = &(mc_model_checker->process()); this->graph_state = std::move(graph_state); if(this->graph_state->system_state == nullptr) this->graph_state->system_state = simgrid::mc::take_snapshot(pair_num); this->heap_bytes_used = mmalloc_get_bytes_used_remote( process->get_heap()->heaplimit, process->get_malloc_info()); this->nb_processes = mc_model_checker->process().simix_processes().size(); this->automaton_state = automaton_state; this->num = pair_num; this->other_num = -1; this->atomic_propositions = std::move(atomic_propositions); } VisitedPair::~VisitedPair() { } static bool evaluate_label( xbt_automaton_exp_label_t l, std::vector const& values) { switch (l->type) { case xbt_automaton_exp_label::AUT_OR: return evaluate_label(l->u.or_and.left_exp, values) || evaluate_label(l->u.or_and.right_exp, values); case xbt_automaton_exp_label::AUT_AND: return evaluate_label(l->u.or_and.left_exp, values) && evaluate_label(l->u.or_and.right_exp, values); case xbt_automaton_exp_label::AUT_NOT: return !evaluate_label(l->u.exp_not, values); case xbt_automaton_exp_label::AUT_PREDICAT:{ unsigned int cursor = 0; xbt_automaton_propositional_symbol_t p = nullptr; xbt_dynar_foreach(simgrid::mc::property_automaton->propositional_symbols, cursor, p) { if (std::strcmp(xbt_automaton_propositional_symbol_get_name(p), l->u.predicat) == 0) return values[cursor] != 0; } xbt_die("Missing predicate"); } case xbt_automaton_exp_label::AUT_ONE: return true; default: xbt_die("Unexpected vaue for automaton"); } } Pair::Pair(unsigned long expanded_pairs) : num(expanded_pairs) {} Pair::~Pair() {} std::shared_ptr> LivenessChecker::getPropositionValues() { std::vector values; unsigned int cursor = 0; xbt_automaton_propositional_symbol_t ps = nullptr; xbt_dynar_foreach(simgrid::mc::property_automaton->propositional_symbols, cursor, ps) values.push_back(xbt_automaton_propositional_symbol_evaluate(ps)); return std::make_shared>(std::move(values)); } int LivenessChecker::compare(simgrid::mc::VisitedPair* state1, simgrid::mc::VisitedPair* state2) { simgrid::mc::Snapshot* s1 = state1->graph_state->system_state.get(); simgrid::mc::Snapshot* s2 = state2->graph_state->system_state.get(); int num1 = state1->num; int num2 = state2->num; return simgrid::mc::snapshot_compare(num1, s1, num2, s2); } std::shared_ptr LivenessChecker::insertAcceptancePair(simgrid::mc::Pair* pair) { std::shared_ptr new_pair = std::make_shared( pair->num, pair->automaton_state, pair->atomic_propositions, pair->graph_state); auto res = boost::range::equal_range(acceptancePairs_, new_pair.get(), simgrid::mc::DerefAndCompareByNbProcessesAndUsedHeap()); if (pair->search_cycle) for (auto i = res.first; i != res.second; ++i) { std::shared_ptr const& pair_test = *i; if (xbt_automaton_state_compare( pair_test->automaton_state, new_pair->automaton_state) != 0 || *(pair_test->atomic_propositions) != *(new_pair->atomic_propositions) || this->compare(pair_test.get(), new_pair.get()) != 0) continue; XBT_INFO("Pair %d already reached (equal to pair %d) !", new_pair->num, pair_test->num); explorationStack_.pop_back(); if (dot_output != nullptr) fprintf(dot_output, "\"%d\" -> \"%d\" [%s];\n", this->previousPair_, pair_test->num, this->previousRequest_.c_str()); return nullptr; } acceptancePairs_.insert(res.first, new_pair); return new_pair; } void LivenessChecker::removeAcceptancePair(int pair_num) { for (auto i = acceptancePairs_.begin(); i != acceptancePairs_.end(); ++i) if ((*i)->num == pair_num) { acceptancePairs_.erase(i); break; } } void LivenessChecker::prepare(void) { this->previousPair_ = 0; std::shared_ptr> propos = this->getPropositionValues(); // For each initial state of the property automaton, push a // (application_state, automaton_state) pair to the exploration stack: unsigned int cursor = 0; xbt_automaton_state_t automaton_state; xbt_dynar_foreach(simgrid::mc::property_automaton->states, cursor, automaton_state) if (automaton_state->type == -1) explorationStack_.push_back(this->newPair(nullptr, automaton_state, propos)); } void LivenessChecker::replay() { XBT_DEBUG("**** Begin Replay ****"); /* Intermediate backtracking */ if(_sg_mc_checkpoint > 0) { simgrid::mc::Pair* pair = explorationStack_.back().get(); if(pair->graph_state->system_state){ simgrid::mc::restore_snapshot(pair->graph_state->system_state); return; } } /* Restore the initial state */ simgrid::mc::session->restoreInitialState(); /* Traverse the stack from the initial state and re-execute the transitions */ int depth = 1; for (std::shared_ptr const& pair : explorationStack_) { if (pair == explorationStack_.back()) break; std::shared_ptr state = pair->graph_state; if (pair->exploration_started) { int req_num = state->transition.argument; smx_simcall_t saved_req = &state->executed_req; smx_simcall_t req = nullptr; if (saved_req != nullptr) { /* because we got a copy of the executed request, we have to fetch the real one, pointed by the request field of the issuer process */ const smx_actor_t issuer = MC_smx_simcall_get_issuer(saved_req); req = &issuer->simcall; /* Debug information */ XBT_DEBUG("Replay (depth = %d) : %s (%p)", depth, simgrid::mc::request_to_string( req, req_num, simgrid::mc::RequestType::simix).c_str(), state.get()); } this->getSession().execute(state->transition); } /* Update statistics */ visitedPairsCount_++; mc_model_checker->executed_transitions++; depth++; } XBT_DEBUG("**** End Replay ****"); } /** * \brief Checks whether a given pair has already been visited by the algorithm. */ int LivenessChecker::insertVisitedPair(std::shared_ptr visited_pair, simgrid::mc::Pair* pair) { if (_sg_mc_visited == 0) return -1; if (visited_pair == nullptr) visited_pair = std::make_shared( pair->num, pair->automaton_state, pair->atomic_propositions, pair->graph_state); auto range = boost::range::equal_range(visitedPairs_, visited_pair.get(), simgrid::mc::DerefAndCompareByNbProcessesAndUsedHeap()); for (auto i = range.first; i != range.second; ++i) { VisitedPair* pair_test = i->get(); if (xbt_automaton_state_compare( pair_test->automaton_state, visited_pair->automaton_state) != 0 || *(pair_test->atomic_propositions) != *(visited_pair->atomic_propositions) || this->compare(pair_test, visited_pair.get()) != 0) continue; if (pair_test->other_num == -1) visited_pair->other_num = pair_test->num; else visited_pair->other_num = pair_test->other_num; if (dot_output == nullptr) XBT_DEBUG("Pair %d already visited ! (equal to pair %d)", visited_pair->num, pair_test->num); else XBT_DEBUG("Pair %d already visited ! (equal to pair %d (pair %d in dot_output))", visited_pair->num, pair_test->num, visited_pair->other_num); (*i) = std::move(visited_pair); return (*i)->other_num; } visitedPairs_.insert(range.first, std::move(visited_pair)); this->purgeVisitedPairs(); return -1; } void LivenessChecker::purgeVisitedPairs() { if (_sg_mc_visited != 0 && visitedPairs_.size() > (std::size_t) _sg_mc_visited) { // Remove the oldest entry with a linear search: visitedPairs_.erase(boost::min_element(visitedPairs_, [](std::shared_ptr const a, std::shared_ptr const& b) { return a->num < b->num; } )); } } LivenessChecker::LivenessChecker(Session& session) : Checker(session) { } LivenessChecker::~LivenessChecker() { } RecordTrace LivenessChecker::getRecordTrace() // override { RecordTrace res; for (std::shared_ptr const& pair : explorationStack_) res.push_back(pair->graph_state->getTransition()); return res; } void LivenessChecker::logState() // override { Checker::logState(); XBT_INFO("Expanded pairs = %lu", expandedPairsCount_); XBT_INFO("Visited pairs = %lu", visitedPairsCount_); XBT_INFO("Executed transitions = %lu", mc_model_checker->executed_transitions); } void LivenessChecker::showAcceptanceCycle(std::size_t depth) { XBT_INFO("*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*"); XBT_INFO("| ACCEPTANCE CYCLE |"); XBT_INFO("*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*"); XBT_INFO("Counter-example that violates formula :"); simgrid::mc::dumpRecordPath(); for (auto& s : this->getTextualTrace()) XBT_INFO("%s", s.c_str()); simgrid::mc::session->logState(); XBT_INFO("Counter-example depth : %zd", depth); } std::vector LivenessChecker::getTextualTrace() // override { std::vector trace; for (std::shared_ptr const& pair : explorationStack_) { int req_num = pair->graph_state->transition.argument; smx_simcall_t req = &pair->graph_state->executed_req; if (req && req->call != SIMCALL_NONE) trace.push_back(simgrid::mc::request_to_string( req, req_num, simgrid::mc::RequestType::executed)); } return trace; } int LivenessChecker::main(void) { while (!explorationStack_.empty()){ std::shared_ptr current_pair = explorationStack_.back(); /* Update current state in buchi automaton */ simgrid::mc::property_automaton->current_state = current_pair->automaton_state; XBT_DEBUG("********************* ( Depth = %d, search_cycle = %d, interleave size = %zd, pair_num = %d, requests = %d)", current_pair->depth, current_pair->search_cycle, current_pair->graph_state->interleaveSize(), current_pair->num, current_pair->requests); if (current_pair->requests == 0) { this->backtrack(); continue; } std::shared_ptr reached_pair; if (current_pair->automaton_state->type == 1 && !current_pair->exploration_started && (reached_pair = this->insertAcceptancePair(current_pair.get())) == nullptr) { this->showAcceptanceCycle(current_pair->depth); return SIMGRID_MC_EXIT_LIVENESS; } /* Pair already visited ? stop the exploration on the current path */ int visited_num = -1; if ((!current_pair->exploration_started) && (visited_num = this->insertVisitedPair( reached_pair, current_pair.get())) != -1) { if (dot_output != nullptr){ fprintf(dot_output, "\"%d\" -> \"%d\" [%s];\n", this->previousPair_, visited_num, this->previousRequest_.c_str()); fflush(dot_output); } XBT_DEBUG("Pair already visited (equal to pair %d), exploration on the current path stopped.", visited_num); current_pair->requests = 0; this->backtrack(); continue; } smx_simcall_t req = MC_state_get_request(current_pair->graph_state.get()); int req_num = current_pair->graph_state->transition.argument; if (dot_output != nullptr) { if (this->previousPair_ != 0 && this->previousPair_ != current_pair->num) { fprintf(dot_output, "\"%d\" -> \"%d\" [%s];\n", this->previousPair_, current_pair->num, this->previousRequest_.c_str()); this->previousRequest_.clear(); } this->previousPair_ = current_pair->num; this->previousRequest_ = simgrid::mc::request_get_dot_output(req, req_num); if (current_pair->search_cycle) fprintf(dot_output, "%d [shape=doublecircle];\n", current_pair->num); fflush(dot_output); } XBT_DEBUG("Execute: %s", simgrid::mc::request_to_string( req, req_num, simgrid::mc::RequestType::simix).c_str()); /* Update stats */ mc_model_checker->executed_transitions++; if (!current_pair->exploration_started) visitedPairsCount_++; /* Answer the request */ mc_model_checker->handle_simcall(current_pair->graph_state->transition); /* Wait for requests (schedules processes) */ mc_model_checker->wait_for_requests(); current_pair->requests--; current_pair->exploration_started = true; /* Get values of atomic propositions (variables used in the property formula) */ std::shared_ptr> prop_values = this->getPropositionValues(); // For each enabled transition in the property automaton, push a // (application_state, automaton_state) pair to the exploration stack: int cursor = xbt_dynar_length(current_pair->automaton_state->out) - 1; while (cursor >= 0) { xbt_automaton_transition_t transition_succ = (xbt_automaton_transition_t)xbt_dynar_get_as(current_pair->automaton_state->out, cursor, xbt_automaton_transition_t); if (evaluate_label(transition_succ->label, *prop_values)) explorationStack_.push_back(this->newPair( current_pair.get(), transition_succ->dst, prop_values)); cursor--; } } XBT_INFO("No property violation found."); simgrid::mc::session->logState(); return SIMGRID_MC_EXIT_SUCCESS; } std::shared_ptr LivenessChecker::newPair(Pair* current_pair, xbt_automaton_state_t state, std::shared_ptr> propositions) { std::shared_ptr next_pair = std::make_shared(++expandedPairsCount_); next_pair->automaton_state = state; next_pair->graph_state = std::shared_ptr(MC_state_new(++expandedStatesCount_)); next_pair->atomic_propositions = std::move(propositions); if (current_pair) next_pair->depth = current_pair->depth + 1; else next_pair->depth = 1; /* Get enabled processes and insert them in the interleave set of the next graph_state */ for (auto& p : mc_model_checker->process().simix_processes()) if (simgrid::mc::process_is_enabled(p.copy.getBuffer())) next_pair->graph_state->interleave(p.copy.getBuffer()); next_pair->requests = next_pair->graph_state->interleaveSize(); /* FIXME : get search_cycle value for each acceptant state */ if (next_pair->automaton_state->type == 1 || (current_pair && current_pair->search_cycle)) next_pair->search_cycle = true; else next_pair->search_cycle = false; return next_pair; } void LivenessChecker::backtrack() { /* Traverse the stack backwards until a pair with a non empty interleave set is found, deleting all the pairs that have it empty in the way. */ while (!explorationStack_.empty()) { std::shared_ptr current_pair = explorationStack_.back(); explorationStack_.pop_back(); if (current_pair->requests > 0) { /* We found a backtracking point */ XBT_DEBUG("Backtracking to depth %d", current_pair->depth); explorationStack_.push_back(std::move(current_pair)); this->replay(); XBT_DEBUG("Backtracking done"); break; } else { XBT_DEBUG("Delete pair %d at depth %d", current_pair->num, current_pair->depth); if (current_pair->automaton_state->type == 1) this->removeAcceptancePair(current_pair->num); } } } int LivenessChecker::run() { XBT_INFO("Check the liveness property %s", _sg_mc_property_file); MC_automaton_load(_sg_mc_property_file); XBT_DEBUG("Starting the liveness algorithm"); simgrid::mc::session->initialize(); this->prepare(); int res = this->main(); return res; } Checker* createLivenessChecker(Session& session) { return new LivenessChecker(session); } } } SimGrid-3.14.159/src/mc/Frame.cpp000644 001750 001750 00000002201 13031031270 020005 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include "src/mc/Frame.hpp" namespace simgrid { namespace mc { void* Frame::frame_base(unw_cursor_t& unw_cursor) const { simgrid::dwarf::Location location = simgrid::dwarf::resolve( frame_base_location, object_info, &unw_cursor, nullptr, nullptr, -1); if (location.in_memory()) return location.address(); else if (location.in_register()) { // This is a special case. // The register is not the location of the frame base // (a frame base cannot be located in a register). // Instead, DWARF defines this to mean that the register // contains the address of the frame base. unw_word_t word; unw_get_reg(&unw_cursor, location.register_id(), &word); return (void*) word; } else xbt_die("Unexpected location type"); } } }SimGrid-3.14.159/src/mc/Channel.hpp000644 001750 001750 00000003545 13031031270 020344 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2015-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_CHANNEL_HPP #define SIMGRID_MC_CHANNEL_HPP #include #include #include "src/mc/mc_protocol.h" namespace simgrid { namespace mc { /** A channel for exchanging messages between model-checker and model-checked * * This abstracts away the way the messages are transferred. Currently, they * are sent over a (connected) `SOCK_DGRAM` socket. */ class Channel { int socket_ = -1; template static constexpr bool messageType() { return std::is_class::value && std::is_trivial::value; } public: Channel() {} explicit Channel(int sock) : socket_(sock) {} ~Channel(); // No copy: Channel(Channel const&) = delete; Channel& operator=(Channel const&) = delete; // Move: Channel(Channel&& that) : socket_(that.socket_) { that.socket_ = -1; } Channel& operator=(Channel&& that) { this->socket_ = that.socket_; that.socket_ = -1; return *this; } // Send int send(const void* message, size_t size) const; int send(e_mc_message_type type) const { s_mc_message message = { type }; return this->send(&message, sizeof(message)); } template typename std::enable_if< messageType(), int >::type send(M const& m) const { return this->send(&m, sizeof(M)); } // Receive ssize_t receive(void* message, size_t size, bool block = true) const; template typename std::enable_if< messageType(), ssize_t >::type receive(M& m) const { return this->receive(&m, sizeof(M)); } int getSocket() const { return socket_; } }; } } #endif SimGrid-3.14.159/src/mc/mc_snapshot.h000644 001750 001750 00000017326 13031031270 020754 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_SNAPSHOT_H #define SIMGRID_MC_SNAPSHOT_H #include #include #include #include #include #include #include // off_t #include #include "src/xbt/mmalloc/mmprivate.h" #include #include #include "src/mc/mc_forward.hpp" #include "src/mc/ModelChecker.hpp" #include "src/mc/PageStore.hpp" #include "src/mc/AddressSpace.hpp" #include "src/mc/mc_unw.h" #include "src/mc/RegionSnapshot.hpp" SG_BEGIN_DECL() // ***** Snapshot region XBT_PRIVATE void mc_region_restore_sparse(simgrid::mc::Process* process, mc_mem_region_t reg); static inline __attribute__((always_inline)) void* mc_translate_address_region_chunked(uintptr_t addr, mc_mem_region_t region) { auto split = simgrid::mc::mmu::split(addr - region->start().address()); auto pageno = split.first; auto offset = split.second; const void* snapshot_page = region->page_data().page(pageno); return (char*) snapshot_page + offset; } static inline __attribute__((always_inline)) void* mc_translate_address_region(uintptr_t addr, mc_mem_region_t region, int process_index) { switch (region->storage_type()) { case simgrid::mc::StorageType::NoData: default: xbt_die("Storage type not supported"); case simgrid::mc::StorageType::Flat: { uintptr_t offset = (uintptr_t) addr - (uintptr_t) region->start().address(); return (void *) ((uintptr_t) region->flat_data().get() + offset); } case simgrid::mc::StorageType::Chunked: return mc_translate_address_region_chunked(addr, region); case simgrid::mc::StorageType::Privatized: { xbt_assert(process_index >=0, "Missing process index for privatized region"); xbt_assert((size_t) process_index < region->privatized_data().size(), "Out of range process index"); simgrid::mc::RegionSnapshot& subregion= region->privatized_data()[process_index]; return mc_translate_address_region(addr, &subregion, process_index); } } } XBT_PRIVATE mc_mem_region_t mc_get_snapshot_region( const void* addr, const simgrid::mc::Snapshot *snapshot, int process_index); } // ***** MC Snapshot /** Ignored data * * Some parts of the snapshot are ignored by zeroing them out: the real * values is stored here. * */ typedef struct s_mc_snapshot_ignored_data { void* start; std::vector data; } s_mc_snapshot_ignored_data_t, *mc_snapshot_ignored_data_t; typedef struct s_fd_infos{ std::string filename; int number; off_t current_position; int flags; }s_fd_infos_t, *fd_infos_t; /** Information about a given stack frame * */ typedef struct s_mc_stack_frame { /** Instruction pointer */ unw_word_t ip; /** Stack pointer */ unw_word_t sp; unw_word_t frame_base; simgrid::mc::Frame* frame; std::string frame_name; unw_cursor_t unw_cursor; } s_mc_stack_frame_t, *mc_stack_frame_t; typedef struct s_local_variable{ simgrid::mc::Frame* subprogram; unsigned long ip; std::string name; simgrid::mc::Type* type; void *address; int region; } s_local_variable_t, *local_variable_t; typedef struct XBT_PRIVATE s_mc_snapshot_stack { std::vector local_variables; simgrid::mc::UnwindContext context; std::vector stack_frames; int process_index; } s_mc_snapshot_stack_t, *mc_snapshot_stack_t; namespace simgrid { namespace mc { class XBT_PRIVATE Snapshot final : public AddressSpace { public: Snapshot(Process* process); ~Snapshot(); const void* read_bytes(void* buffer, std::size_t size, RemotePtr address, int process_index = ProcessIndexAny, ReadOptions options = ReadOptions::none()) const override; public: // To be private int num_state; std::size_t heap_bytes_used; std::vector> snapshot_regions; std::set enabled_processes; int privatization_index; std::vector stack_sizes; std::vector stacks; std::vector to_ignore; std::uint64_t hash; std::vector ignored_data; std::vector current_fds; }; } } extern "C" { static inline __attribute__ ((always_inline)) mc_mem_region_t mc_get_region_hinted(void* addr, simgrid::mc::Snapshot* snapshot, int process_index, mc_mem_region_t region) { if (region->contain(simgrid::mc::remote(addr))) return region; else return mc_get_snapshot_region(addr, snapshot, process_index); } static const void* mc_snapshot_get_heap_end(simgrid::mc::Snapshot* snapshot); } #ifdef __cplusplus namespace simgrid { namespace mc { XBT_PRIVATE std::shared_ptr take_snapshot(int num_state); XBT_PRIVATE void restore_snapshot(std::shared_ptr snapshot); } } #endif extern "C" { XBT_PRIVATE void mc_restore_page_snapshot_region( simgrid::mc::Process* process, void* start_addr, simgrid::mc::ChunkedData const& pagenos); const void* MC_region_read_fragmented( mc_mem_region_t region, void* target, const void* addr, std::size_t size); int MC_snapshot_region_memcmp( const void* addr1, mc_mem_region_t region1, const void* addr2, mc_mem_region_t region2, std::size_t size); XBT_PRIVATE int MC_snapshot_memcmp( const void* addr1, simgrid::mc::Snapshot* snapshot1, const void* addr2, simgrid::mc::Snapshot* snapshot2, int process_index, std::size_t size); static inline __attribute__ ((always_inline)) const void* mc_snapshot_get_heap_end(simgrid::mc::Snapshot* snapshot) { if(snapshot==nullptr) xbt_die("snapshot is nullptr"); return mc_model_checker->process().get_heap()->breakval; } /** @brief Read memory from a snapshot region * * @param addr Process (non-snapshot) address of the data * @param region Snapshot memory region where the data is located * @param target Buffer to store the value * @param size Size of the data to read in bytes * @return Pointer where the data is located (target buffer of original location) */ static inline __attribute__((always_inline)) const void* MC_region_read( mc_mem_region_t region, void* target, const void* addr, std::size_t size) { xbt_assert(region); std::uintptr_t offset = (std::uintptr_t) addr - (std::uintptr_t) region->start().address(); xbt_assert(region->contain(simgrid::mc::remote(addr)), "Trying to read out of the region boundary."); switch (region->storage_type()) { case simgrid::mc::StorageType::NoData: default: xbt_die("Storage type not supported"); case simgrid::mc::StorageType::Flat: return (char*) region->flat_data().get() + offset; case simgrid::mc::StorageType::Chunked: { // Last byte of the region: void* end = (char*) addr + size - 1; if (simgrid::mc::mmu::sameChunk((std::uintptr_t) addr, (std::uintptr_t) end) ) { // The memory is contained in a single page: return mc_translate_address_region_chunked((uintptr_t) addr, region); } else { // The memory spans several pages: return MC_region_read_fragmented(region, target, addr, size); } } // We currently do not pass the process_index to this function so we assume // that the privatized region has been resolved in the callers: case simgrid::mc::StorageType::Privatized: xbt_die("Storage type not supported"); } } static inline __attribute__ ((always_inline)) void* MC_region_read_pointer(mc_mem_region_t region, const void* addr) { void* res; return *(void**) MC_region_read(region, &res, addr, sizeof(void*)); } SG_END_DECL() #endif SimGrid-3.14.159/src/mc/LocationList.hpp000644 001750 001750 00000005151 13031031270 021373 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2004-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_OBJECT_LOCATION_H #define SIMGRID_MC_OBJECT_LOCATION_H #include #include #include #include #include #include #include #include "src/mc/mc_base.h" #include "src/mc/mc_forward.hpp" #include "src/mc/DwarfExpression.hpp" namespace simgrid { namespace dwarf { /** A DWARF expression with optional validity constraints */ class LocationListEntry { public: typedef simgrid::xbt::Range range_type; private: DwarfExpression expression_; // By default, the expression is always valid: range_type range_ = {0, UINT64_MAX}; public: LocationListEntry() {} LocationListEntry(DwarfExpression expression, range_type range) : expression_(std::move(expression)), range_(range) {} LocationListEntry(DwarfExpression expression) : expression_(std::move(expression)), range_({0, UINT64_MAX}) {} DwarfExpression& expression() { return expression_; } DwarfExpression const& expression() const { return expression_; } bool valid_for_ip(unw_word_t ip) const { return range_.contain(ip); } }; typedef std::vector LocationList; /** Location of some variable in memory * * The variable is located either in memory of a register. */ class Location { private: void* memory_; int register_id_; public: Location(void* x) :memory_(x) {} Location(int register_id) : memory_(nullptr), register_id_(register_id) {} // Type of location: bool in_register() const { return memory_ == nullptr; } bool in_memory() const { return memory_ != nullptr; } // Get the location: void* address() const { return memory_; } int register_id() const { return register_id_; } }; XBT_PRIVATE Location resolve( simgrid::dwarf::DwarfExpression const& expression, simgrid::mc::ObjectInformation* object_info, unw_cursor_t* c, void* frame_pointer_address, simgrid::mc::AddressSpace* address_space, int process_index); Location resolve( simgrid::dwarf::LocationList const& locations, simgrid::mc::ObjectInformation* object_info, unw_cursor_t * c, void *frame_pointer_address, simgrid::mc::AddressSpace* address_space, int process_index); XBT_PRIVATE simgrid::dwarf::LocationList location_list( simgrid::mc::ObjectInformation& info, Dwarf_Attribute& attr); } } #endif SimGrid-3.14.159/src/mc/mc_memory.cpp000644 001750 001750 00000001556 13031031270 020756 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include "xbt/log.h" #include "mc/mc.h" #include "src/mc/mc_private.h" #include "src/xbt/mmalloc/mmprivate.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_memory, mc, "Logging specific to MC (memory)"); extern "C" { /* Initialize the model-checker memory subsystem */ /* It creates the two heap regions: std_heap and mc_heap */ void MC_memory_init() { if (!malloc_use_mmalloc()) xbt_die("Model-checking support is not enabled: run with simgrid-mc."); } /* Finalize the memory subsystem */ void MC_memory_exit(void) { } } SimGrid-3.14.159/src/mc/mc_state.cpp000644 001750 001750 00000017645 13031031270 020574 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include "src/simix/smx_private.h" #include "src/mc/mc_state.h" #include "src/mc/mc_request.h" #include "src/mc/mc_private.h" #include "src/mc/mc_comm_pattern.h" #include "src/mc/mc_smx.h" #include "src/mc/mc_xbt.hpp" #include "src/mc/Transition.hpp" using simgrid::mc::remote; XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_state, mc, "Logging specific to MC (state)"); /** * \brief Creates a state data structure used by the exploration algorithm */ simgrid::mc::State* MC_state_new(unsigned long state_number) { simgrid::mc::State* state = new simgrid::mc::State(); state->processStates.resize(MC_smx_get_maxpid()); state->num = state_number; /* Stateful model checking */ if((_sg_mc_checkpoint > 0 && (state_number % _sg_mc_checkpoint == 0)) || _sg_mc_termination){ state->system_state = simgrid::mc::take_snapshot(state->num); if(_sg_mc_comms_determinism || _sg_mc_send_determinism){ MC_state_copy_incomplete_communications_pattern(state); MC_state_copy_index_communications_pattern(state); } } return state; } namespace simgrid { namespace mc { State::State() { this->internal_comm.clear(); std::memset(&this->internal_req, 0, sizeof(this->internal_req)); std::memset(&this->executed_req, 0, sizeof(this->executed_req)); } std::size_t State::interleaveSize() const { return boost::range::count_if(this->processStates, [](simgrid::mc::ProcessState const& p) { return p.isToInterleave(); }); } Transition State::getTransition() const { return this->transition; } } } static inline smx_simcall_t MC_state_get_request_for_process( simgrid::mc::State* state, smx_actor_t process) { simgrid::mc::ProcessState* procstate = &state->processStates[process->pid]; state->transition.pid = -1; state->transition.argument = -1; if (!procstate->isToInterleave()) return nullptr; if (!simgrid::mc::process_is_enabled(process)) return nullptr; smx_simcall_t req = nullptr; switch (process->simcall.call) { case SIMCALL_COMM_WAITANY: state->transition.argument = -1; while (procstate->interleave_count < read_length(mc_model_checker->process(), remote(simcall_comm_waitany__get__comms(&process->simcall)))) { if (simgrid::mc::request_is_enabled_by_idx(&process->simcall, procstate->interleave_count++)) { state->transition.argument = procstate->interleave_count - 1; break; } } if (procstate->interleave_count >= simgrid::mc::read_length(mc_model_checker->process(), simgrid::mc::remote(simcall_comm_waitany__get__comms(&process->simcall)))) procstate->setDone(); if (state->transition.argument != -1) req = &process->simcall; break; case SIMCALL_COMM_TESTANY: { unsigned start_count = procstate->interleave_count; state->transition.argument = -1; while (procstate->interleave_count < simcall_comm_testany__get__count(&process->simcall)) if (simgrid::mc::request_is_enabled_by_idx(&process->simcall, procstate->interleave_count++)) { state->transition.argument = procstate->interleave_count - 1; break; } if (procstate->interleave_count >= simcall_comm_testany__get__count(&process->simcall)) procstate->setDone(); if (state->transition.argument != -1 || start_count == 0) req = &process->simcall; break; } case SIMCALL_COMM_WAIT: { simgrid::mc::RemotePtr remote_act = remote( static_cast(simcall_comm_wait__get__comm(&process->simcall))); simgrid::mc::Remote temp_act; mc_model_checker->process().read(temp_act, remote_act); simgrid::kernel::activity::Comm* act = temp_act.getBuffer(); if (act->src_proc && act->dst_proc) state->transition.argument = 0; else if (act->src_proc == nullptr && act->type == SIMIX_COMM_READY && act->detached == 1) state->transition.argument = 0; else state->transition.argument = -1; procstate->setDone(); req = &process->simcall; break; } case SIMCALL_MC_RANDOM: { int min_value = simcall_mc_random__get__min(&process->simcall); state->transition.argument = procstate->interleave_count + min_value; procstate->interleave_count++; if (state->transition.argument == simcall_mc_random__get__max(&process->simcall)) procstate->setDone(); req = &process->simcall; break; } default: procstate->setDone(); state->transition.argument = 0; req = &process->simcall; break; } if (!req) return nullptr; // Fetch the data of the request and translate it: state->transition.pid = process->pid; state->executed_req = *req; /* The waitany and testany request are transformed into a wait or test request * over the corresponding communication action so it can be treated later by * the dependence function. */ switch (req->call) { case SIMCALL_COMM_WAITANY: { state->internal_req.call = SIMCALL_COMM_WAIT; state->internal_req.issuer = req->issuer; smx_activity_t remote_comm; read_element(mc_model_checker->process(), &remote_comm, remote(simcall_comm_waitany__get__comms(req)), state->transition.argument, sizeof(remote_comm)); mc_model_checker->process().read(state->internal_comm, remote( static_cast(remote_comm))); simcall_comm_wait__set__comm(&state->internal_req, state->internal_comm.getBuffer()); simcall_comm_wait__set__timeout(&state->internal_req, 0); break; } case SIMCALL_COMM_TESTANY: state->internal_req.call = SIMCALL_COMM_TEST; state->internal_req.issuer = req->issuer; if (state->transition.argument > 0) { smx_activity_t remote_comm = mc_model_checker->process().read( remote(simcall_comm_testany__get__comms(req) + state->transition.argument)); mc_model_checker->process().read(state->internal_comm, remote( static_cast(remote_comm))); } simcall_comm_test__set__comm(&state->internal_req, state->internal_comm.getBuffer()); simcall_comm_test__set__result(&state->internal_req, state->transition.argument); break; case SIMCALL_COMM_WAIT: state->internal_req = *req; mc_model_checker->process().read_bytes(&state->internal_comm , sizeof(state->internal_comm), remote(simcall_comm_wait__get__comm(req))); simcall_comm_wait__set__comm(&state->executed_req, state->internal_comm.getBuffer()); simcall_comm_wait__set__comm(&state->internal_req, state->internal_comm.getBuffer()); break; case SIMCALL_COMM_TEST: state->internal_req = *req; mc_model_checker->process().read_bytes(&state->internal_comm, sizeof(state->internal_comm), remote(simcall_comm_test__get__comm(req))); simcall_comm_test__set__comm(&state->executed_req, state->internal_comm.getBuffer()); simcall_comm_test__set__comm(&state->internal_req, state->internal_comm.getBuffer()); break; default: state->internal_req = *req; break; } return req; } smx_simcall_t MC_state_get_request(simgrid::mc::State* state) { for (auto& p : mc_model_checker->process().simix_processes()) { smx_simcall_t res = MC_state_get_request_for_process(state, p.copy.getBuffer()); if (res) return res; } return nullptr; } SimGrid-3.14.159/src/mc/mc_dwarf_tagnames.cpp000644 001750 001750 00000007721 13031031270 022430 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* Warning: autogenerated, do not edit! */ #include #include #include "src/mc/mc_dwarf.hpp" #include namespace simgrid { namespace dwarf { /** \brief Get the name of a dwarf tag (DW_TAG_*) from its code * * \param tag tag code (see the DWARF specification) * \return name of the tag */ XBT_PRIVATE const char *tagname(int tag) { switch (tag) { case 0x01: return "DW_TAG_array_type"; case 0x02: return "DW_TAG_class_type"; case 0x03: return "DW_TAG_entry_point"; case 0x04: return "DW_TAG_enumeration_type"; case 0x05: return "DW_TAG_formal_parameter"; case 0x08: return "DW_TAG_imported_declaration"; case 0x0a: return "DW_TAG_label"; case 0x0b: return "DW_TAG_lexical_block"; case 0x0d: return "DW_TAG_member"; case 0x0f: return "DW_TAG_pointer_type"; case 0x10: return "DW_TAG_reference_type"; case 0x11: return "DW_TAG_compile_unit"; case 0x12: return "DW_TAG_string_type"; case 0x13: return "DW_TAG_structure_type"; case 0x15: return "DW_TAG_subroutine_type"; case 0x16: return "DW_TAG_typedef"; case 0x17: return "DW_TAG_union_type"; case 0x18: return "DW_TAG_unspecified_parameters"; case 0x19: return "DW_TAG_variant"; case 0x1a: return "DW_TAG_common_block"; case 0x1b: return "DW_TAG_common_inclusion"; case 0x1c: return "DW_TAG_inheritance"; case 0x1d: return "DW_TAG_inlined_subroutine"; case 0x1e: return "DW_TAG_module"; case 0x1f: return "DW_TAG_ptr_to_member_type"; case 0x20: return "DW_TAG_set_type"; case 0x21: return "DW_TAG_subrange_type"; case 0x22: return "DW_TAG_with_stmt"; case 0x23: return "DW_TAG_access_declaration"; case 0x24: return "DW_TAG_base_type"; case 0x25: return "DW_TAG_catch_block"; case 0x26: return "DW_TAG_const_type"; case 0x27: return "DW_TAG_constant"; case 0x28: return "DW_TAG_enumerator"; case 0x29: return "DW_TAG_file_type"; case 0x2a: return "DW_TAG_friend"; case 0x2b: return "DW_TAG_namelist"; case 0x2c: return "DW_TAG_namelist_item"; case 0x2d: return "DW_TAG_packed_type"; case 0x2e: return "DW_TAG_subprogram"; case 0x2f: return "DW_TAG_template_type_parameter"; case 0x30: return "DW_TAG_template_value_parameter"; case 0x31: return "DW_TAG_thrown_type"; case 0x32: return "DW_TAG_try_block"; case 0x33: return "DW_TAG_variant_part"; case 0x34: return "DW_TAG_variable"; case 0x35: return "DW_TAG_volatile_type"; case 0x36: return "DW_TAG_dwarf_procedure"; case 0x37: return "DW_TAG_restrict_type"; case 0x38: return "DW_TAG_interface_type"; case 0x39: return "DW_TAG_namespace"; case 0x3a: return "DW_TAG_imported_module"; case 0x3b: return "DW_TAG_unspecified_type"; case 0x3c: return "DW_TAG_partial_unit"; case 0x3d: return "DW_TAG_imported_unit"; case 0x3e: return "DW_TAG_mutable_type"; case 0x3f: return "DW_TAG_condition"; case 0x40: return "DW_TAG_shared_type"; case 0x41: return "DW_TAG_type_unit"; case 0x42: return "DW_TAG_rvalue_reference_type"; case 0x43: return "DW_TAG_template_alias"; case 0x4080: return "DW_TAG_lo_user"; case 0x4081: return "DW_TAG_MIPS_loop"; case 0x4101: return "DW_TAG_format_label"; case 0x4102: return "DW_TAG_function_template"; case 0x4103: return "DW_TAG_class_template"; case 0x4104: return "DW_TAG_GNU_BINCL"; case 0x4105: return "DW_TAG_GNU_EINCL"; case 0x4106: return "DW_TAG_GNU_template_template_param"; case 0x4107: return "DW_TAG_GNU_template_parameter_pack"; case 0x4108: return "DW_TAG_GNU_formal_parameter_pack"; case 0x4109: return "DW_TAG_GNU_call_site"; case 0x410a: return "DW_TAG_GNU_call_site_parameter"; case 0xffff: return "DW_TAG_hi_user"; case DW_TAG_invalid: return "DW_TAG_invalid"; default: return "DW_TAG_unknown"; } } } } SimGrid-3.14.159/src/mc/ModelChecker.hpp000644 001750 001750 00000004755 13031031270 021325 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_MODEL_CHECKER_HPP #define SIMGRID_MC_MODEL_CHECKER_HPP #include #include #include #include #include #include #include #include #include "src/mc/mc_forward.hpp" #include "src/mc/Process.hpp" #include "src/mc/PageStore.hpp" #include "src/mc/mc_protocol.h" #include "src/mc/Transition.hpp" namespace simgrid { namespace mc { /** State of the model-checker (global variables for the model checker) */ class ModelChecker { struct event_base *base_; struct event *socket_event_, *signal_event_; /** String pool for host names */ // TODO, use std::set with heterogeneous comparison lookup (C++14)? std::set hostnames_; // This is the parent snapshot of the current state: PageStore page_store_; std::unique_ptr process_; Checker* checker_ = nullptr; public: std::shared_ptr parent_snapshot_; public: ModelChecker(ModelChecker const&) = delete; ModelChecker& operator=(ModelChecker const&) = delete; ModelChecker(std::unique_ptr process); ~ModelChecker(); Process& process() { return *process_; } PageStore& page_store() { return page_store_; } std::string const& get_host_name(const char* hostname) { return *this->hostnames_.insert(hostname).first; } std::string const& get_host_name(std::string const& hostname) { return *this->hostnames_.insert(hostname).first; } void start(); void shutdown(); void resume(simgrid::mc::Process& process); void loop(); void handle_events(int fd, short events); void wait_client(simgrid::mc::Process& process); void handle_simcall(Transition const& transition); void wait_for_requests() { mc_model_checker->wait_client(mc_model_checker->process()); } void exit(int status); bool checkDeadlock(); Checker* getChecker() const { return checker_; } void setChecker(Checker* checker) { checker_ = checker; } private: void setup_ignore(); bool handle_message(char* buffer, ssize_t size); void handle_waitpid(); void on_signal(int signo); public: unsigned long visited_states = 0; unsigned long executed_transitions = 0; }; } } #endif SimGrid-3.14.159/src/mc/mc_forward.hpp000644 001750 001750 00000001471 13031031270 021113 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /** @file mc_forward.hpp * * Forward definitions for MC types */ #ifndef SIMGRID_MC_FORWARD_HPP #define SIMGRID_MC_FORWARD_HPP namespace simgrid { namespace mc { class PageStore; class ChunkedData; class ModelChecker; class AddressSpace; class Process; class Snapshot; class ObjectInformation; class Member; class Type; class Variable; class Frame; class SimixProcessInformation; class Session; class Checker; } } // TODO, try to get rid of the global ModelChecker variable extern simgrid::mc::ModelChecker* mc_model_checker; #endif SimGrid-3.14.159/src/mc/mc_protocol.h000644 001750 001750 00000007070 13031031270 020751 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_PROTOCOL_H #define SIMGRID_MC_PROTOCOL_H #include #include #include "mc/datatypes.h" SG_BEGIN_DECL() // ***** Environment variables for passing context to the model-checked process /** Environment variable name set by `simgrid-mc` to enable MC support in the * children MC processes */ #define MC_ENV_VARIABLE "SIMGRID_MC" /** Environment variable name used to pass the communication socket */ #define MC_ENV_SOCKET_FD "SIMGRID_MC_SOCKET_FD" // ***** Messages typedef enum { MC_MESSAGE_NONE, MC_MESSAGE_CONTINUE, MC_MESSAGE_IGNORE_HEAP, MC_MESSAGE_UNIGNORE_HEAP, MC_MESSAGE_IGNORE_MEMORY, MC_MESSAGE_STACK_REGION, MC_MESSAGE_REGISTER_SYMBOL, MC_MESSAGE_DEADLOCK_CHECK, MC_MESSAGE_DEADLOCK_CHECK_REPLY, MC_MESSAGE_WAITING, MC_MESSAGE_SIMCALL_HANDLE, MC_MESSAGE_ASSERTION_FAILED, // MCer request to finish the restoration: MC_MESSAGE_RESTORE, } e_mc_message_type; #define MC_MESSAGE_LENGTH 512 /** Basic structure for a MC message * * The current version of the client/server protocol sends C structures over `AF_LOCAL` * `SOCK_DGRAM` sockets. This means that the protocol is ABI/architecture specific: * we currently can't model-check a x86 process from a x86_64 process. * * Moreover the protocol is not stable. The same version of the library should be used * for the client and the server. * * This is the basic structure shared by all messages: all message start with a message * type. */ struct s_mc_message { e_mc_message_type type; }; typedef struct s_mc_message s_mc_message_t; typedef struct s_mc_message* mc_message_t; struct s_mc_int_message { e_mc_message_type type; uint64_t value; }; typedef struct s_mc_int_message s_mc_int_message_t; typedef struct s_mc_int_message* mc_int_message_t; struct s_mc_ignore_heap_message { e_mc_message_type type; int block; int fragment; void *address; size_t size; }; typedef struct s_mc_ignore_heap_message s_mc_ignore_heap_message_t; typedef struct s_mc_ignore_heap_message* mc_ignore_heap_message_t; struct s_mc_ignore_memory_message { e_mc_message_type type; uint64_t addr; size_t size; }; typedef struct s_mc_ignore_memory_message s_mc_ignore_memory_message_t; typedef struct s_mc_ignore_memory_message* mc_ignore_memory_message_t; struct s_mc_stack_region_message { e_mc_message_type type; s_stack_region_t stack_region; }; typedef struct s_mc_stack_region_message s_mc_stack_region_message_t; typedef struct s_mc_stack_region_message* mc_stack_region_message_t; struct s_mc_simcall_handle_message { e_mc_message_type type; unsigned long pid; int value; }; typedef struct s_mc_simcall_handle_message s_mc_simcall_handle_message_t; typedef struct s_mc_simcall_handle_message* mc_simcall_handle_message; struct s_mc_register_symbol_message { e_mc_message_type type; char name[128]; int (*callback)(void*); void* data; }; typedef struct s_mc_register_symbol_message s_mc_register_symbol_message_t; typedef struct s_mc_register_symbol_message* mc_register_symbol_message_t; struct s_mc_restore_message { e_mc_message_type type; int index; }; typedef struct s_mc_restore_message s_mc_restore_message_t; typedef struct s_mc_restore_message* mc_restore_message_t; XBT_PRIVATE const char* MC_message_type_name(e_mc_message_type type); SG_END_DECL() #endif SimGrid-3.14.159/src/mc/ChunkedData.cpp000644 001750 001750 00000003427 13031031270 021141 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include "src/mc/AddressSpace.hpp" #include "src/mc/ChunkedData.hpp" #include "src/mc/PageStore.hpp" namespace simgrid { namespace mc { /** Take a per-page snapshot of a region * * @param data The start of the region (must be at the beginning of a page) * @param page_count Number of pages of the region * @return Snapshot page numbers of this new snapshot */ ChunkedData::ChunkedData(PageStore& store, AddressSpace& as, RemotePtr addr, std::size_t page_count) { store_ = &store; this->pagenos_.resize(page_count); std::vector buffer(xbt_pagesize); for (size_t i = 0; i != page_count; ++i) { RemotePtr page = remote((void*) simgrid::mc::mmu::join(i, addr.address())); xbt_assert(simgrid::mc::mmu::split(page.address()).second == 0, "Not at the beginning of a page"); /* Adding another copy (and a syscall) will probably slow things a lot. TODO, optimize this somehow (at least by grouping the syscalls) if needed. Either: - reduce the number of syscalls; - let the application snapshot itself; - move the segments in shared memory (this will break `fork` however). */ as.read_bytes( buffer.data(), xbt_pagesize, page, simgrid::mc::ProcessIndexDisabled); pagenos_[i] = store_->store_page(buffer.data()); } } } } SimGrid-3.14.159/src/mc/mc_mmalloc.h000644 001750 001750 00000000000 13031031270 020516 0ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/src/mc/Frame.hpp000644 001750 001750 00000003334 13031031270 020022 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_FRAME_HPP #define SIMGRID_MC_FRAME_HPP #include #include #include #include #include "src/mc/mc_forward.hpp" #include "src/mc/LocationList.hpp" #include "src/mc/Variable.hpp" namespace simgrid { namespace mc { /** Debug information about a given function or scope within a function */ class Frame { public: Frame(); /** Kind of scope (DW_TAG_subprogram, DW_TAG_inlined_subroutine, etc.) */ int tag = DW_TAG_invalid; /** Name of the function (if it is a function) */ std::string name; /** Range of instruction addresses for which this scope is valid */ simgrid::xbt::Range range; simgrid::dwarf::LocationList frame_base_location; /** List of the variables (sorted by name) */ std::vector variables; /* Unique identifier for this scope (in the object_info) * * This is the global DWARF offset of the DIE. */ unsigned long int id = 0; std::vector scopes; /** Value of `DW_AT_abstract_origin` * * For inlined subprograms, this is the ID of the * parent function. */ unsigned long int abstract_origin_id = 0; simgrid::mc::ObjectInformation* object_info = nullptr; void* frame_base(unw_cursor_t& unw_cursor) const; void remove_variable(char* name); }; inline Frame::Frame() { this->tag = 0; this->range = {0, 0}; this->id = 0; this->abstract_origin_id = 0; this->object_info = nullptr; } } } #endif SimGrid-3.14.159/src/mc/mc_unw.cpp000644 001750 001750 00000020645 13031031270 020257 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /** \file * Libunwind support for mc_address_space objects. */ // We need this for the register indices: // #define _GNU_SOURCE #include // On x86_64, libunwind unw_context_t has the same layout as ucontext_t: #include #include #ifdef __FreeBSD__ typedef register_t greg_t; #endif #include #include "src/mc/Process.hpp" #include "src/mc/mc_unw.h" #include "src/mc/Frame.hpp" using simgrid::mc::remote; namespace simgrid { namespace mc { // ***** Implementation /** Get frame unwind information (libunwind method) * * Delegates to the local/ptrace implementation. */ int UnwindContext::find_proc_info(unw_addr_space_t as, unw_word_t ip, unw_proc_info_t *pip, int need_unwind_info, void* arg) noexcept { simgrid::mc::UnwindContext* context = (simgrid::mc::UnwindContext*) arg; return unw_get_accessors(context->process_->unw_underlying_addr_space)->find_proc_info( context->process_->unw_underlying_addr_space, ip, pip, need_unwind_info, context->process_->unw_underlying_context ); } /** Release frame unwind information (libunwind method) * * Delegates to the local/ptrace implementation. */ void UnwindContext::put_unwind_info(unw_addr_space_t as, unw_proc_info_t *pip, void* arg) noexcept { simgrid::mc::UnwindContext* context = (simgrid::mc::UnwindContext*) arg; return unw_get_accessors(context->process_->unw_underlying_addr_space)->put_unwind_info( context->process_->unw_underlying_addr_space, pip, context->process_->unw_underlying_context ); } /** (libunwind method) * * Not implemented. */ int UnwindContext::get_dyn_info_list_addr(unw_addr_space_t as, unw_word_t *dilap, void* arg) noexcept { simgrid::mc::UnwindContext* context = (simgrid::mc::UnwindContext*) arg; return unw_get_accessors(context->process_->unw_underlying_addr_space)->get_dyn_info_list_addr( context->process_->unw_underlying_addr_space, dilap, context->process_->unw_underlying_context ); } /** Read from the target address space memory (libunwind method) * * Delegates to the `simgrid::mc::Process*`. */ int UnwindContext::access_mem(unw_addr_space_t as, unw_word_t addr, unw_word_t *valp, int write, void* arg) noexcept { simgrid::mc::UnwindContext* context = (simgrid::mc::UnwindContext*) arg; if (write) return - UNW_EREADONLYREG; context->addressSpace_->read_bytes(valp, sizeof(unw_word_t), remote(addr)); return 0; } void* UnwindContext::get_reg(unw_context_t* context, unw_regnum_t regnum) noexcept { #ifdef __x86_64 mcontext_t* mcontext = &context->uc_mcontext; switch (regnum) { # ifdef __linux__ case UNW_X86_64_RAX: return &mcontext->gregs[REG_RAX]; case UNW_X86_64_RDX: return &mcontext->gregs[REG_RDX]; case UNW_X86_64_RCX: return &mcontext->gregs[REG_RCX]; case UNW_X86_64_RBX: return &mcontext->gregs[REG_RBX]; case UNW_X86_64_RSI: return &mcontext->gregs[REG_RSI]; case UNW_X86_64_RDI: return &mcontext->gregs[REG_RDI]; case UNW_X86_64_RBP: return &mcontext->gregs[REG_RBP]; case UNW_X86_64_RSP: return &mcontext->gregs[REG_RSP]; case UNW_X86_64_R8: return &mcontext->gregs[REG_R8]; case UNW_X86_64_R9: return &mcontext->gregs[REG_R9]; case UNW_X86_64_R10: return &mcontext->gregs[REG_R10]; case UNW_X86_64_R11: return &mcontext->gregs[REG_R11]; case UNW_X86_64_R12: return &mcontext->gregs[REG_R12]; case UNW_X86_64_R13: return &mcontext->gregs[REG_R13]; case UNW_X86_64_R14: return &mcontext->gregs[REG_R14]; case UNW_X86_64_R15: return &mcontext->gregs[REG_R15]; case UNW_X86_64_RIP: return &mcontext->gregs[REG_RIP]; # elif defined __FreeBSD__ case UNW_X86_64_RAX: return &mcontext->mc_rax; case UNW_X86_64_RDX: return &mcontext->mc_rdx; case UNW_X86_64_RCX: return &mcontext->mc_rcx; case UNW_X86_64_RBX: return &mcontext->mc_rbx; case UNW_X86_64_RSI: return &mcontext->mc_rsi; case UNW_X86_64_RDI: return &mcontext->mc_rdi; case UNW_X86_64_RBP: return &mcontext->mc_rbp; case UNW_X86_64_RSP: return &mcontext->mc_rsp; case UNW_X86_64_R8: return &mcontext->mc_r8; case UNW_X86_64_R9: return &mcontext->mc_r9; case UNW_X86_64_R10: return &mcontext->mc_r10; case UNW_X86_64_R11: return &mcontext->mc_r11; case UNW_X86_64_R12: return &mcontext->mc_r12; case UNW_X86_64_R13: return &mcontext->mc_r13; case UNW_X86_64_R14: return &mcontext->mc_r14; case UNW_X86_64_R15: return &mcontext->mc_r15; case UNW_X86_64_RIP: return &mcontext->mc_rip; # else # error "Unable to get register from ucontext, please add your case" # endif default: return nullptr; } #else return nullptr; #endif } /** Read a standard register (libunwind method) */ int UnwindContext::access_reg(unw_addr_space_t as, unw_regnum_t regnum, unw_word_t *valp, int write, void* arg) noexcept { simgrid::mc::UnwindContext* as_context = (simgrid::mc::UnwindContext*) arg; unw_context_t* context = &as_context->unwindContext_; if (write) return -UNW_EREADONLYREG; greg_t* preg = (greg_t*) get_reg(context, regnum); if (!preg) return -UNW_EBADREG; *valp = *preg; return 0; } /** Read a floating-point register (libunwind method) * * FP registers are caller-saved. The values saved by functions such as * `getcontext()` is not relevant for the caller. It is not really necessary * to save and handle them. */ int UnwindContext::access_fpreg(unw_addr_space_t as, unw_regnum_t regnum, unw_fpreg_t *fpvalp, int write, void* arg) noexcept { return -UNW_EBADREG; } /** Resume the execution of the context (libunwind method) * * We don't use this. */ int UnwindContext::resume(unw_addr_space_t as, unw_cursor_t *cp, void* arg) noexcept { return -UNW_EUNSPEC; } /** Find informations about a function (libunwind method) */ int UnwindContext::get_proc_name(unw_addr_space_t as, unw_word_t addr, char *bufp, size_t buf_len, unw_word_t *offp, void* arg) noexcept { simgrid::mc::UnwindContext* context = (simgrid::mc::UnwindContext*) arg; simgrid::mc::Frame* frame = context->process_->find_function(remote(addr)); if (!frame) return - UNW_ENOINFO; *offp = (unw_word_t) frame->range.begin() - addr; strncpy(bufp, frame->name.c_str(), buf_len); if (bufp[buf_len - 1]) { bufp[buf_len - 1] = 0; return -UNW_ENOMEM; } return 0; } // ***** Init /** Virtual table for our `libunwind` implementation * * Stack unwinding on a `simgrid::mc::Process*` (for memory, unwinding information) * and `ucontext_t` (for processor registers). * * It works with the `simgrid::mc::UnwindContext` context. */ unw_accessors_t UnwindContext::accessors = { &find_proc_info, &put_unwind_info, &get_dyn_info_list_addr, &access_mem, &access_reg, &access_fpreg, &resume, &get_proc_name }; unw_addr_space_t UnwindContext::createUnwindAddressSpace() { return unw_create_addr_space(&UnwindContext::accessors, BYTE_ORDER); } void UnwindContext::clear() { addressSpace_ = nullptr; process_ = nullptr; } void UnwindContext::initialize(simgrid::mc::Process* process, unw_context_t* c) { clear(); this->addressSpace_ = process; this->process_ = process; // Take a copy of the context for our own purpose: this->unwindContext_ = *c; #if SIMGRID_PROCESSOR_x86_64 || SIMGRID_PROCESSOR_i686 # ifdef __linux__ // On x86_64, ucontext_t contains a pointer to itself for FP registers. // We don't really need support for FR registers as they are caller saved // and probably never use those fields as libunwind-x86_64 does not read // FP registers from the unw_context_t // but we fix the pointer in order to avoid dangling pointers: // context->context_.uc_mcontext.fpregs = &(context->context.__fpregs_mem); // Let's ignore this and see what happens: this->unwindContext_.uc_mcontext.fpregs = nullptr; # endif #else // Do we need to do any fixup like this? #error Target CPU type is not handled. #endif } unw_cursor_t UnwindContext::cursor() { unw_cursor_t cursor; if (process_ == nullptr || addressSpace_ == nullptr || unw_init_remote(&cursor, process_->unw_addr_space, this) != 0) xbt_die("UnwindContext not initialized"); return cursor; } } } SimGrid-3.14.159/src/mc/RegionSnapshot.cpp000644 001750 001750 00000007746 13031031270 021741 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #ifdef __FreeBSD__ # define MAP_POPULATE MAP_PREFAULT_READ #endif #include "mc/mc.h" #include "src/mc/mc_snapshot.h" #include "src/mc/ChunkedData.hpp" #include "src/mc/RegionSnapshot.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_RegionSnaphot, mc, "Logging specific to region snapshots"); namespace simgrid { namespace mc { static inline const char* to_cstr(RegionType region) { switch (region) { case RegionType::Unknown: return "unknown"; case RegionType::Heap: return "Heap"; case RegionType::Data: return "Data"; default: return "?"; } } Buffer::Buffer(std::size_t size, Type type) : size_(size), type_(type) { switch(type_) { case Type::Malloc: data_ = ::malloc(size_); break; case Type::Mmap: data_ = ::mmap(nullptr, size, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_POPULATE, -1, 0); if (data_ == MAP_FAILED) { data_ = nullptr; size_ = 0; type_ = Type::Malloc; throw std::bad_alloc(); } break; default: abort(); } } void Buffer::clear() noexcept { switch(type_) { case Type::Malloc: std::free(data_); break; case Type::Mmap: if (munmap(data_, size_) != 0) abort(); break; default: abort(); } data_ = nullptr; size_ = 0; type_ = Type::Malloc; } RegionSnapshot dense_region( RegionType region_type, void *start_addr, void* permanent_addr, size_t size) { // When KSM support is enables, we allocate memory using mmap: // * we don't want to advise bits of the heap as mergable; // * mmap gives data aligned on page boundaries which is merge friendly. simgrid::mc::Buffer data; if (_sg_mc_ksm) data = Buffer::mmap(size); else data = Buffer::malloc(size); mc_model_checker->process().read_bytes(data.get(), size, remote(permanent_addr), simgrid::mc::ProcessIndexDisabled); #ifdef __linux__ if (_sg_mc_ksm) // Mark the region as mergeable *after* we have written into it. // Trying to merge them before is useless/counterproductive. madvise(data.get(), size, MADV_MERGEABLE); #endif simgrid::mc::RegionSnapshot region( region_type, start_addr, permanent_addr, size); region.flat_data(std::move(data)); XBT_DEBUG("New region : type : %s, data : %p (real addr %p), size : %zu", to_cstr(region_type), region.flat_data().get(), permanent_addr, size); return region; } /** @brief Take a snapshot of a given region * * @param type * @param start_addr Address of the region in the simulated process * @param permanent_addr Permanent address of this data (for privatized variables, this is the virtual address of the privatized mapping) * @param size Size of the data* */ RegionSnapshot region( RegionType type, void *start_addr, void* permanent_addr, size_t size) { if (_sg_mc_sparse_checkpoint) return sparse_region(type, start_addr, permanent_addr, size); else return dense_region(type, start_addr, permanent_addr, size); } RegionSnapshot sparse_region(RegionType region_type, void *start_addr, void* permanent_addr, size_t size) { simgrid::mc::Process* process = &mc_model_checker->process(); assert(process != nullptr); xbt_assert((((uintptr_t)start_addr) & (xbt_pagesize-1)) == 0, "Not at the beginning of a page"); xbt_assert((((uintptr_t)permanent_addr) & (xbt_pagesize-1)) == 0, "Not at the beginning of a page"); size_t page_count = simgrid::mc::mmu::chunkCount(size); simgrid::mc::ChunkedData page_data( mc_model_checker->page_store(), *process, permanent_addr, page_count); simgrid::mc::RegionSnapshot region( region_type, start_addr, permanent_addr, size); region.page_data(std::move(page_data)); return region; } } } SimGrid-3.14.159/src/mc/mc_unw.h000644 001750 001750 00000006177 13031031270 017730 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_UNW_H #define SIMGRID_MC_UNW_H /** @file * Libunwind implementation for the model-checker * * Libunwind provides an pluggable stack unwinding API: the way the current * registers and memory is accessed, the way unwinding informations is found * is pluggable. * * This component implements the libunwind API for he model-checker: * * * reading memory from a simgrid::mc::AddressSpace*; * * * reading stack registers from a saved snapshot (context). * * Parts of the libunwind information fetching is currently handled by the * standard `libunwind` implementations (either the local one or the ptrace one) * because parsing `.eh_frame` section is not fun and `libdw` does not help * much here. */ #include #include #include #include "src/mc/mc_forward.hpp" namespace simgrid { namespace unw { XBT_PRIVATE unw_addr_space_t create_addr_space(); XBT_PRIVATE void* create_context(unw_addr_space_t as, pid_t pid); } } namespace simgrid { namespace mc { class UnwindContext { simgrid::mc::AddressSpace* addressSpace_ = nullptr; simgrid::mc::Process* process_ = nullptr; unw_context_t unwindContext_; public: UnwindContext() {} ~UnwindContext() { clear(); } void initialize(simgrid::mc::Process* process, unw_context_t* c); void clear(); unw_cursor_t cursor(); private: // Methods and virtual table for libunwind static int find_proc_info(unw_addr_space_t as, unw_word_t ip, unw_proc_info_t *pip, int need_unwind_info, void* arg) noexcept; static void put_unwind_info(unw_addr_space_t as, unw_proc_info_t *pip, void* arg) noexcept; static int get_dyn_info_list_addr(unw_addr_space_t as, unw_word_t *dilap, void* arg) noexcept; static int access_mem(unw_addr_space_t as, unw_word_t addr, unw_word_t *valp, int write, void* arg) noexcept; static void* get_reg(unw_context_t* context, unw_regnum_t regnum) noexcept; static int access_reg(unw_addr_space_t as, unw_regnum_t regnum, unw_word_t *valp, int write, void* arg) noexcept; static int access_fpreg(unw_addr_space_t as, unw_regnum_t regnum, unw_fpreg_t *fpvalp, int write, void* arg) noexcept; static int resume(unw_addr_space_t as, unw_cursor_t *cp, void* arg) noexcept; static int get_proc_name(unw_addr_space_t as, unw_word_t addr, char *bufp, size_t buf_len, unw_word_t *offp, void* arg) noexcept; static unw_accessors_t accessors; public: // Create a libunwind address space: static unw_addr_space_t createUnwindAddressSpace(); }; void MC_dump_stack_unw(FILE* file, unw_cursor_t cursor); void dumpStack(FILE* file, unw_cursor_t cursor); void dumpStack(FILE* file, pid_t pid); } } #endif SimGrid-3.14.159/src/mc/DwarfExpression.cpp000644 001750 001750 00000014777 13031031270 022123 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include "src/mc/mc_private.h" #include "src/mc/LocationList.hpp" #include "src/mc/AddressSpace.hpp" #include "src/mc/Frame.hpp" #include "src/mc/ObjectInformation.hpp" #include "src/mc/DwarfExpression.hpp" #include "src/mc/mc_dwarf.hpp" using simgrid::mc::remote; namespace simgrid { namespace dwarf { evaluation_error::~evaluation_error() noexcept(true) {} void execute( const Dwarf_Op* ops, std::size_t n, const ExpressionContext& context, ExpressionStack& stack) { for (size_t i = 0; i != n; ++i) { const Dwarf_Op *op = ops + i; std::uint8_t atom = op->atom; switch (atom) { // Registers: case DW_OP_breg0: case DW_OP_breg1: case DW_OP_breg2: case DW_OP_breg3: case DW_OP_breg4: case DW_OP_breg5: case DW_OP_breg6: case DW_OP_breg7: case DW_OP_breg8: case DW_OP_breg9: case DW_OP_breg10: case DW_OP_breg11: case DW_OP_breg12: case DW_OP_breg13: case DW_OP_breg14: case DW_OP_breg15: case DW_OP_breg16: case DW_OP_breg17: case DW_OP_breg18: case DW_OP_breg19: case DW_OP_breg20: case DW_OP_breg21: case DW_OP_breg22: case DW_OP_breg23: case DW_OP_breg24: case DW_OP_breg25: case DW_OP_breg26: case DW_OP_breg27: case DW_OP_breg28: case DW_OP_breg29: case DW_OP_breg30: case DW_OP_breg31:{ // Push register + constant: int register_id = simgrid::dwarf::dwarf_register_to_libunwind( op->atom - DW_OP_breg0); unw_word_t res; if (!context.cursor) throw evaluation_error("Missing stack context"); unw_get_reg(context.cursor, register_id, &res); stack.push(res + op->number); break; } // Push the CFA (Canonical Frame Address): case DW_OP_call_frame_cfa: { /* See 6.4 of DWARF4 (http://dwarfstd.org/doc/DWARF4.pdf#page=140): * * > Typically, the CFA is defined to be the value of the stack * > pointer at the call site in the previous frame (which may be * > different from its value on entry to the current frame). * * We need to unwind the frame in order to get the SP of the parent * frame. * * Warning: the CFA returned by libunwind (UNW_X86_64_RSP, etc.) * is the SP of the *current* frame. */ if (!context.cursor) throw evaluation_error("Missint cursor"); // Get frame: unw_cursor_t cursor = *(context.cursor); unw_step(&cursor); unw_word_t res; unw_get_reg(&cursor, UNW_REG_SP, &res); stack.push(res); break; } // Frame base: case DW_OP_fbreg: stack.push((std::uintptr_t) context.frame_base + op->number); break; // ***** Constants: // Short constant literals: case DW_OP_lit0: case DW_OP_lit1: case DW_OP_lit2: case DW_OP_lit3: case DW_OP_lit4: case DW_OP_lit5: case DW_OP_lit6: case DW_OP_lit7: case DW_OP_lit8: case DW_OP_lit9: case DW_OP_lit10: case DW_OP_lit11: case DW_OP_lit12: case DW_OP_lit13: case DW_OP_lit14: case DW_OP_lit15: case DW_OP_lit16: case DW_OP_lit17: case DW_OP_lit18: case DW_OP_lit19: case DW_OP_lit20: case DW_OP_lit21: case DW_OP_lit22: case DW_OP_lit23: case DW_OP_lit24: case DW_OP_lit25: case DW_OP_lit26: case DW_OP_lit27: case DW_OP_lit28: case DW_OP_lit29: case DW_OP_lit30: case DW_OP_lit31: // Push a literal/constant on the stack: stack.push(atom - DW_OP_lit0); break; // Address from the base address of this ELF object. // Push the address on the stack (base_address + argument). case DW_OP_addr: { if (!context.object_info) throw evaluation_error("No base address"); Dwarf_Off addr = (Dwarf_Off) (std::uintptr_t) context.object_info->base_address() + op->number; stack.push(addr); break; } // General constants: // Push the constant argument on the stack. case DW_OP_const1u: case DW_OP_const2u: case DW_OP_const4u: case DW_OP_const8u: case DW_OP_const1s: case DW_OP_const2s: case DW_OP_const4s: case DW_OP_const8s: case DW_OP_constu: case DW_OP_consts: stack.push(op->number); break; // ***** Stack manipulation: // Push another copy/duplicate the value at the top of the stack: case DW_OP_dup: stack.dup(); break; // Pop/drop the top of the stack: case DW_OP_drop: stack.pop(); break; case DW_OP_swap: stack.swap(); break; // Duplicate the value under the top of the stack: case DW_OP_over: stack.push(stack.top(1)); break; // ***** Operations: // Those usually take the top of the stack and the next value as argument // and replace the top of the stack with the computed value // (stack.top() += stack.before_top()). case DW_OP_plus: stack.push(stack.pop() + stack.pop()); break; case DW_OP_mul: stack.push(stack.pop() * stack.pop()); break; case DW_OP_plus_uconst: stack.top() += op->number; break; case DW_OP_not: stack.top() = ~stack.top(); break; case DW_OP_neg: stack.top() = - (intptr_t) stack.top(); break; case DW_OP_minus: stack.push(stack.pop() - stack.pop()); break; case DW_OP_and: stack.push(stack.pop() & stack.pop()); break; case DW_OP_or: stack.push(stack.pop() | stack.pop()); break; case DW_OP_xor: stack.push(stack.pop() ^ stack.pop()); break; case DW_OP_nop: break; // ***** Deference (memory fetch) case DW_OP_deref_size: throw evaluation_error("Unsupported operation"); case DW_OP_deref: // Computed address: if (!context.address_space) throw evaluation_error("Missing address space"); context.address_space->read_bytes( &stack.top(), sizeof(uintptr_t), remote(stack.top()), context.process_index); break; // Not handled: default: throw evaluation_error("Unsupported operation"); } } } } } SimGrid-3.14.159/src/mc/mc_record.cpp000644 001750 001750 00000006274 13031031270 020726 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include #include #include "simgrid/simix.h" #include "src/kernel/context/Context.hpp" #include "src/simix/ActorImpl.hpp" #include "src/simix/smx_private.h" #include "src/mc/mc_replay.h" #include "src/mc/mc_record.h" #include "src/mc/mc_base.h" #include "src/mc/Transition.hpp" #if HAVE_MC #include "src/mc/mc_request.h" #include "src/mc/mc_private.h" #include "src/mc/mc_state.h" #include "src/mc/mc_smx.h" #include "src/mc/Checker.hpp" #endif XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_record, mc, " Logging specific to MC record/replay facility"); extern "C" { char* MC_record_path = nullptr; } namespace simgrid { namespace mc { void replay(RecordTrace const& trace) { simgrid::mc::wait_for_requests(); for (simgrid::mc::Transition const& transition : trace) { XBT_DEBUG("Executing %i$%i", transition.pid, transition.argument); // Choose a request: smx_actor_t process = SIMIX_process_from_PID(transition.pid); if (!process) xbt_die("Unexpected process."); smx_simcall_t simcall = &(process->simcall); if(!simcall || simcall->call == SIMCALL_NONE) xbt_die("No simcall for this process."); if (!simgrid::mc::request_is_visible(simcall) || !simgrid::mc::request_is_enabled(simcall)) xbt_die("Unexpected simcall."); // Execute the request: SIMIX_simcall_handle(simcall, transition.argument); simgrid::mc::wait_for_requests(); } } void replay(const char* path_string) { simgrid::mc::processes_time.resize(SIMIX_process_get_maxpid()); simgrid::mc::RecordTrace trace = simgrid::mc::parseRecordTrace(path_string); simgrid::mc::replay(trace); simgrid::mc::processes_time.clear(); } RecordTrace parseRecordTrace(const char* data) { RecordTrace res; XBT_INFO("path=%s", data); if (data == nullptr || data[0] == '\0') throw std::runtime_error("Could not parse record path"); const char* current = data; while (*current) { simgrid::mc::Transition item; int count = sscanf(current, "%u/%u", &item.pid, &item.argument); if(count != 2 && count != 1) throw std::runtime_error("Could not parse record path"); res.push_back(item); // Find next chunk: const char* end = std::strchr(current, ';'); if(end == nullptr) break; else current = end + 1; } return res; } #if HAVE_MC std::string traceToString(simgrid::mc::RecordTrace const& trace) { std::ostringstream stream; for (auto i = trace.begin(); i != trace.end(); ++i) { if (i != trace.begin()) stream << ';'; stream << i->pid; if (i->argument) stream << '/' << i->argument; } return stream.str(); } void dumpRecordPath() { if (MC_record_is_active()) { RecordTrace trace = mc_model_checker->getChecker()->getRecordTrace(); XBT_INFO("Path = %s", traceToString(trace).c_str()); } } #endif } } SimGrid-3.14.159/src/mc/RemotePtr.hpp000644 001750 001750 00000010511 13031031270 020704 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_REMOTE_PTR_HPP #define SIMGRID_MC_REMOTE_PTR_HPP #include #include #include #include #include namespace simgrid { namespace mc { /** HACK, A value from another process * * This represents a value from another process: * * * constructor/destructor are disabled; * * * raw memory copy (std::memcpy) is used to copy Remote; * * * raw memory comparison is used to compare them; * * * when T is a trivial type, Remote is convertible to a T. * * We currently only handle the case where the type has the same layout * in the current process and in the target process: we don't handle * cross-architecture (such as 32-bit/64-bit access). */ template union Remote { private: T buffer; public: Remote() {} ~Remote() {} Remote(T& p) { std::memcpy(&buffer, &p, sizeof(buffer)); } Remote(Remote const& that) { std::memcpy(&buffer, &that.buffer, sizeof(buffer)); } Remote& operator=(Remote const& that) { std::memcpy(&buffer, &that.buffer, sizeof(buffer)); return *this; } T* getBuffer() { return &buffer; } const T* getBuffer() const { return &buffer; } std::size_t getBufferSize() const { return sizeof(T); } operator T() const { static_assert(std::is_trivial::value, "Cannot convert non trivial type"); return buffer; } void clear() { std::memset(static_cast(&buffer), 0, sizeof(T)); } }; /** Pointer to a remote address-space (process, snapshot) * * With this we can clearly identify the expected type of an address in the * remote process while avoiding to use native local pointers. * * Some operators (+/-) assume use the size of the underlying element. This * only works if the target applications is using the same target: it won't * work for example, when inspecting a 32 bit application from a 64 bit * model-checker. * * We do not actually store the target address space because we can * always detect it in context. This way `RemotePtr` is as efficient * as a `uint64_t`. */ template class RemotePtr { std::uint64_t address_; public: RemotePtr() : address_(0) {} RemotePtr(std::uint64_t address) : address_(address) {} RemotePtr(T* address) : address_((std::uintptr_t)address) {} RemotePtr(Remote p) : RemotePtr(*p.getBuffer()) {} std::uint64_t address() const { return address_; } /** Turn into a local pointer * (if the remote process is not, in fact, remote) */ T* local() const { return (T*) address_; } operator bool() const { return address_; } bool operator!() const { return !address_; } operator RemotePtr() const { return RemotePtr(address_); } RemotePtr operator+(std::uint64_t n) const { return RemotePtr(address_ + n * sizeof(T)); } RemotePtr operator-(std::uint64_t n) const { return RemotePtr(address_ - n * sizeof(T)); } RemotePtr& operator+=(std::uint64_t n) { address_ += n * sizeof(T); return *this; } RemotePtr& operator-=(std::uint64_t n) { address_ -= n * sizeof(T); return *this; } }; template bool operator<(RemotePtr const& x, RemotePtr const& y) { return x.address() < y.address(); } template bool operator>(RemotePtr const& x, RemotePtr const& y) { return x.address() > y.address(); } template bool operator>=(RemotePtr const& x, RemotePtr const& y) { return x.address() >= y.address(); } template bool operator<=(RemotePtr const& x, RemotePtr const& y) { return x.address() <= y.address(); } template bool operator==(RemotePtr const& x, RemotePtr const& y) { return x.address() == y.address(); } template bool operator!=(RemotePtr const& x, RemotePtr const& y) { return x.address() != y.address(); } template inline RemotePtr remote(T *p) { return RemotePtr(p); } template inline RemotePtr remote(uint64_t p) { return RemotePtr(p); } } } #endif SimGrid-3.14.159/src/mc/mc_checkpoint.cpp000644 001750 001750 00000047651 13031031270 021603 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #ifndef WIN32 #include #endif #include "src/internal_config.h" #include "src/mc/mc_private.h" #include "xbt/module.h" #include #include #include "src/smpi/private.h" #include "src/xbt/mmalloc/mmprivate.h" #include "src/simix/smx_private.h" #include #include #include "src/mc/mc_private.h" #include #include "src/mc/mc_snapshot.h" #include "src/mc/mc_mmu.h" #include "src/mc/mc_unw.h" #include "src/mc/mc_protocol.h" #include "src/mc/mc_smx.h" #include "mc_hash.hpp" #include "src/mc/RegionSnapshot.hpp" #include "src/mc/ObjectInformation.hpp" #include "src/mc/Frame.hpp" #include "src/mc/Variable.hpp" using simgrid::mc::remote; XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_checkpoint, mc, "Logging specific to mc_checkpoint"); namespace simgrid { namespace mc { /************************************ Free functions **************************************/ /*****************************************************************************************/ /** @brief Restore a region from a snapshot * * @param region Target region */ static void restore(mc_mem_region_t region) { switch(region->storage_type()) { case simgrid::mc::StorageType::NoData: default: xbt_die("Storage type not supported"); break; case simgrid::mc::StorageType::Flat: mc_model_checker->process().write_bytes(region->flat_data().get(), region->size(), region->permanent_address()); break; case simgrid::mc::StorageType::Chunked: mc_region_restore_sparse(&mc_model_checker->process(), region); break; case simgrid::mc::StorageType::Privatized: for (auto& p : region->privatized_data()) restore(&p); break; } } #if HAVE_SMPI RegionSnapshot privatized_region( RegionType region_type, void *start_addr, void* permanent_addr, std::size_t size ) { size_t process_count = MC_smpi_process_count(); // Read smpi_privatisation_regions from MCed: smpi_privatisation_region_t remote_smpi_privatisation_regions; mc_model_checker->process().read_variable( "smpi_privatisation_regions", &remote_smpi_privatisation_regions, sizeof(remote_smpi_privatisation_regions)); s_smpi_privatisation_region_t privatisation_regions[process_count]; mc_model_checker->process().read_bytes( &privatisation_regions, sizeof(privatisation_regions), remote(remote_smpi_privatisation_regions)); std::vector data; data.reserve(process_count); for (size_t i = 0; i < process_count; i++) data.push_back(simgrid::mc::region(region_type, start_addr, privatisation_regions[i].address, size)); simgrid::mc::RegionSnapshot region = simgrid::mc::RegionSnapshot( region_type, start_addr, permanent_addr, size); region.privatized_data(std::move(data)); return region; } #endif static void add_region(int index, simgrid::mc::Snapshot* snapshot, simgrid::mc::RegionType type, simgrid::mc::ObjectInformation* object_info, void *start_addr, void* permanent_addr, std::size_t size) { if (type == simgrid::mc::RegionType::Data) xbt_assert(object_info, "Missing object info for object."); else if (type == simgrid::mc::RegionType::Heap) xbt_assert(!object_info, "Unexpected object info for heap region."); simgrid::mc::RegionSnapshot region; #if HAVE_SMPI const bool privatization_aware = object_info && mc_model_checker->process().privatized(*object_info); if (privatization_aware && MC_smpi_process_count()) region = simgrid::mc::privatized_region( type, start_addr, permanent_addr, size); else #endif region = simgrid::mc::region(type, start_addr, permanent_addr, size); region.object_info(object_info); snapshot->snapshot_regions[index] = std::unique_ptr( new simgrid::mc::RegionSnapshot(std::move(region))); return; } static void get_memory_regions(simgrid::mc::Process* process, simgrid::mc::Snapshot* snapshot) { const size_t n = process->object_infos.size(); snapshot->snapshot_regions.resize(n + 1); int i = 0; for (auto const& object_info : process->object_infos) add_region(i++, snapshot, simgrid::mc::RegionType::Data, object_info.get(), object_info->start_rw, object_info->start_rw, object_info->end_rw - object_info->start_rw); xbt_mheap_t heap = process->get_heap(); void *start_heap = heap->base; void *end_heap = heap->breakval; add_region(n, snapshot, simgrid::mc::RegionType::Heap, nullptr, start_heap, start_heap, (char *) end_heap - (char *) start_heap); snapshot->heap_bytes_used = mmalloc_get_bytes_used_remote( heap->heaplimit, process->get_malloc_info()); #if HAVE_SMPI if (mc_model_checker->process().privatized() && MC_smpi_process_count()) // snapshot->privatization_index = smpi_loaded_page mc_model_checker->process().read_variable( "smpi_loaded_page", &snapshot->privatization_index, sizeof(snapshot->privatization_index)); else #endif snapshot->privatization_index = simgrid::mc::ProcessIndexMissing; } #define PROT_RWX (PROT_READ | PROT_WRITE | PROT_EXEC) #define PROT_RW (PROT_READ | PROT_WRITE) #define PROT_RX (PROT_READ | PROT_EXEC) /** \brief Fills the position of the segments (executable, read-only, read/write). * */ // TODO, use the ELF segment information for more robustness void find_object_address( std::vector const& maps, simgrid::mc::ObjectInformation* result) { char* name = xbt_basename(result->file_name.c_str()); for (size_t i = 0; i < maps.size(); ++i) { simgrid::xbt::VmMap const& reg = maps[i]; if (maps[i].pathname.empty()) continue; char* map_basename = xbt_basename(maps[i].pathname.c_str()); if (strcmp(name, map_basename) != 0) { free(map_basename); continue; } free(map_basename); // This is the non-GNU_RELRO-part of the data segment: if (reg.prot == PROT_RW) { xbt_assert(!result->start_rw, "Multiple read-write segments for %s, not supported", maps[i].pathname.c_str()); result->start_rw = (char*) reg.start_addr; result->end_rw = (char*) reg.end_addr; // The next VMA might be end of the data segment: if (i + 1 < maps.size() && maps[i + 1].pathname.empty() && maps[i + 1].prot == PROT_RW && maps[i + 1].start_addr == reg.end_addr) result->end_rw = (char*) maps[i + 1].end_addr; } // This is the text segment: else if (reg.prot == PROT_RX) { xbt_assert(!result->start_exec, "Multiple executable segments for %s, not supported", maps[i].pathname.c_str()); result->start_exec = (char*) reg.start_addr; result->end_exec = (char*) reg.end_addr; // The next VMA might be end of the data segment: if (i + 1 < maps.size() && maps[i + 1].pathname.empty() && maps[i + 1].prot == PROT_RW && maps[i + 1].start_addr == reg.end_addr) { result->start_rw = (char*) maps[i + 1].start_addr; result->end_rw = (char*) maps[i + 1].end_addr; } } // This is the GNU_RELRO-part of the data segment: else if (reg.prot == PROT_READ) { xbt_assert(!result->start_ro, "Multiple read only segments for %s, not supported", maps[i].pathname.c_str()); result->start_ro = (char*) reg.start_addr; result->end_ro = (char*) reg.end_addr; } } result->start = result->start_rw; if ((const void*) result->start_ro < result->start) result->start = result->start_ro; if ((const void*) result->start_exec < result->start) result->start = result->start_exec; result->end = result->end_rw; if (result->end_ro && (const void*) result->end_ro > result->end) result->end = result->end_ro; if (result->end_exec && (const void*) result->end_exec > result->end) result->end = result->end_exec; xbt_assert(result->start_exec || result->start_rw || result->start_ro); free(name); } /************************************* Take Snapshot ************************************/ /****************************************************************************************/ /** \brief Checks whether the variable is in scope for a given IP. * * A variable may be defined only from a given value of IP. * * \param var Variable description * \param scope Scope description * \param ip Instruction pointer * \return true if the variable is valid * */ static bool valid_variable(simgrid::mc::Variable* var, simgrid::mc::Frame* scope, const void *ip) { // The variable is not yet valid: if (scope->range.begin() + var->start_scope > (std::uint64_t) ip) return false; else return true; } static void fill_local_variables_values(mc_stack_frame_t stack_frame, simgrid::mc::Frame* scope, int process_index, std::vector& result) { simgrid::mc::Process* process = &mc_model_checker->process(); if (!scope || !scope->range.contain(stack_frame->ip)) return; for(simgrid::mc::Variable& current_variable : scope->variables) { if (!valid_variable(¤t_variable, scope, (void *) stack_frame->ip)) continue; int region_type; // FIXME, get rid of `region_type` if ((long) stack_frame->ip > (long) process->libsimgrid_info->start_exec) region_type = 1; else region_type = 2; s_local_variable_t new_var; new_var.subprogram = stack_frame->frame; new_var.ip = stack_frame->ip; new_var.name = current_variable.name; new_var.type = current_variable.type; new_var.region = region_type; new_var.address = nullptr; if (current_variable.address != nullptr) new_var.address = current_variable.address; else if (!current_variable.location_list.empty()) { simgrid::dwarf::Location location = simgrid::dwarf::resolve( current_variable.location_list, current_variable.object_info, &(stack_frame->unw_cursor), (void *) stack_frame->frame_base, &mc_model_checker->process(), process_index); if (!location.in_memory()) xbt_die("Cannot handle non-address variable"); new_var.address = location.address(); } else xbt_die("No address"); result.push_back(std::move(new_var)); } // Recursive processing of nested scopes: for(simgrid::mc::Frame& nested_scope : scope->scopes) fill_local_variables_values( stack_frame, &nested_scope, process_index, result); } static std::vector get_local_variables_values( std::vector& stack_frames, int process_index) { std::vector variables; for (s_mc_stack_frame_t& stack_frame : stack_frames) fill_local_variables_values(&stack_frame, stack_frame.frame, process_index, variables); return variables; } static std::vector unwind_stack_frames(simgrid::mc::UnwindContext* stack_context) { simgrid::mc::Process* process = &mc_model_checker->process(); std::vector result; unw_cursor_t c = stack_context->cursor(); // TODO, check condition check (unw_init_local==0 means end of frame) while (1) { s_mc_stack_frame_t stack_frame; stack_frame.unw_cursor = c; unw_word_t ip, sp; unw_get_reg(&c, UNW_REG_IP, &ip); unw_get_reg(&c, UNW_REG_SP, &sp); stack_frame.ip = ip; stack_frame.sp = sp; // TODO, use real addresses in frame_t instead of fixing it here simgrid::mc::Frame* frame = process->find_function(remote(ip)); stack_frame.frame = frame; if (frame) { stack_frame.frame_name = frame->name; stack_frame.frame_base = (unw_word_t) frame->frame_base(c); } else { stack_frame.frame_base = 0; stack_frame.frame_name = std::string(); } result.push_back(std::move(stack_frame)); /* Stop before context switch with maestro */ if (frame != nullptr && frame->name == "smx_ctx_sysv_wrapper") break; int ret = unw_step(&c); if (ret == 0) xbt_die("Unexpected end of stack."); else if (ret < 0) xbt_die("Error while unwinding stack"); } if (result.empty()) { XBT_INFO("unw_init_local failed"); xbt_abort(); } return result; }; static std::vector take_snapshot_stacks(simgrid::mc::Snapshot* snapshot) { std::vector res; for (auto const& stack : mc_model_checker->process().stack_areas()) { s_mc_snapshot_stack_t st; // Read the context from remote process: unw_context_t context; mc_model_checker->process().read_bytes( &context, sizeof(context), remote(stack.context)); st.context.initialize(&mc_model_checker->process(), &context); st.stack_frames = unwind_stack_frames(&st.context); st.local_variables = get_local_variables_values(st.stack_frames, stack.process_index); st.process_index = stack.process_index; unw_word_t sp = st.stack_frames[0].sp; res.push_back(std::move(st)); size_t stack_size = (char*) stack.address + stack.size - (char*) sp; snapshot->stack_sizes.push_back(stack_size); } return res; } static void snapshot_handle_ignore(simgrid::mc::Snapshot* snapshot) { xbt_assert(snapshot->process()); // Copy the memory: for (auto const& region : mc_model_checker->process().ignored_regions()) { s_mc_snapshot_ignored_data_t ignored_data; ignored_data.start = (void*)region.addr; ignored_data.data.resize(region.size); // TODO, we should do this once per privatization segment: snapshot->process()->read_bytes( ignored_data.data.data(), region.size, remote(region.addr), simgrid::mc::ProcessIndexDisabled); snapshot->ignored_data.push_back(std::move(ignored_data)); } // Zero the memory: for(auto const& region : mc_model_checker->process().ignored_regions()) snapshot->process()->clear_bytes(remote(region.addr), region.size); } static void snapshot_ignore_restore(simgrid::mc::Snapshot* snapshot) { for (auto const& ignored_data : snapshot->ignored_data) snapshot->process()->write_bytes( ignored_data.data.data(), ignored_data.data.size(), remote(ignored_data.start)); } static std::vector get_current_fds(pid_t pid) { const size_t fd_dir_path_size = 20; char fd_dir_path[fd_dir_path_size]; int res = snprintf(fd_dir_path, fd_dir_path_size, "/proc/%lli/fd", (long long int) pid); xbt_assert(res >= 0); if ((size_t) res > fd_dir_path_size) xbt_die("Unexpected buffer is too small for fd_dir_path"); DIR* fd_dir = opendir(fd_dir_path); if (fd_dir == nullptr) xbt_die("Cannot open directory '/proc/self/fd'\n"); std::vector fds; struct dirent* fd_number; while ((fd_number = readdir(fd_dir))) { int fd_value = xbt_str_parse_int(fd_number->d_name, "Found a non-numerical FD: %s. Freaking out!"); if(fd_value < 3) continue; const size_t source_size = 25; char source[25]; int res = snprintf(source, source_size, "/proc/%lli/fd/%s", (long long int) pid, fd_number->d_name); xbt_assert(res >= 0); if ((size_t) res > source_size) xbt_die("Unexpected buffer is too small for fd %s", fd_number->d_name); const size_t link_size = 200; char link[200]; res = readlink(source, link, link_size); if (res<0) xbt_die("Could not read link for %s", source); if (res==200) xbt_die("Buffer to small for link of %s", source); link[res] = '\0'; #if HAVE_SMPI if(smpi_is_privatisation_file(link)) continue; #endif // This is (probably) the DIR* we are reading: // TODO, read all the file entries at once and close the DIR.* if(strcmp(fd_dir_path, link) == 0) continue; // We don't handle them. // It does not mean we should silently ignore them however. if (strncmp(link, "pipe:", std::strlen("pipe:")) == 0 || strncmp(link, "socket:", std::strlen("socket:")) == 0) continue; // If dot_output enabled, do not handle the corresponding file if (dot_output != nullptr) { char* link_basename = xbt_basename(link); if (strcmp(link_basename, _sg_mc_dot_output_file) == 0) { free(link_basename); continue; } free(link_basename); } // This is probably a shared memory used by lttng-ust: if(strncmp("/dev/shm/ust-shm-tmp-", link, std::strlen("/dev/shm/ust-shm-tmp-"))==0) continue; // Add an entry for this FD in the snapshot: s_fd_infos_t fd; fd.filename = std::string(link); fd.number = fd_value; fd.flags = fcntl(fd_value, F_GETFL) | fcntl(fd_value, F_GETFD) ; fd.current_position = lseek(fd_value, 0, SEEK_CUR); fds.push_back(std::move(fd)); } closedir (fd_dir); return fds; } std::shared_ptr take_snapshot(int num_state) { XBT_DEBUG("Taking snapshot %i", num_state); simgrid::mc::Process* mc_process = &mc_model_checker->process(); std::shared_ptr snapshot = std::make_shared(mc_process); snapshot->num_state = num_state; for (auto& p : mc_model_checker->process().simix_processes()) snapshot->enabled_processes.insert(p.copy.getBuffer()->pid); snapshot_handle_ignore(snapshot.get()); if (_sg_mc_snapshot_fds) snapshot->current_fds = get_current_fds(mc_model_checker->process().pid()); /* Save the std heap and the writable mapped pages of libsimgrid and binary */ get_memory_regions(mc_process, snapshot.get()); snapshot->to_ignore = mc_model_checker->process().ignored_heap(); if (_sg_mc_visited > 0 || strcmp(_sg_mc_property_file, "")) { snapshot->stacks = take_snapshot_stacks(snapshot.get()); if (_sg_mc_hash) snapshot->hash = simgrid::mc::hash(*snapshot); else snapshot->hash = 0; } else snapshot->hash = 0; snapshot_ignore_restore(snapshot.get()); return snapshot; } static inline void restore_snapshot_regions(simgrid::mc::Snapshot* snapshot) { for(std::unique_ptr const& region : snapshot->snapshot_regions) { // For privatized, variables we decided it was not necessary to take the snapshot: if (region) restore(region.get()); } #if HAVE_SMPI // TODO, send a message to implement this in the MCed process if(snapshot->privatization_index >= 0) { // Fix the privatization mmap: s_mc_restore_message message; message.type = MC_MESSAGE_RESTORE; message.index = snapshot->privatization_index; mc_model_checker->process().getChannel().send(message); } #endif } static inline void restore_snapshot_fds(simgrid::mc::Snapshot* snapshot) { xbt_die("FD snapshot not implemented in client/server mode."); for (auto const& fd : snapshot->current_fds) { int new_fd = open(fd.filename.c_str(), fd.flags); if (new_fd < 0) xbt_die("Could not reopen the file %s fo restoring the file descriptor", fd.filename.c_str()); if (new_fd != fd.number) { dup2(new_fd, fd.number); close(new_fd); }; lseek(fd.number, fd.current_position, SEEK_SET); } } void restore_snapshot(std::shared_ptr snapshot) { XBT_DEBUG("Restore snapshot %i", snapshot->num_state); restore_snapshot_regions(snapshot.get()); if (_sg_mc_snapshot_fds) restore_snapshot_fds(snapshot.get()); snapshot_ignore_restore(snapshot.get()); mc_model_checker->process().clear_cache(); } } } SimGrid-3.14.159/src/mc/mc_snapshot.cpp000644 001750 001750 00000023713 13031031270 021304 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include "src/internal_config.h" #include "src/smpi/private.h" #include "src/mc/mc_snapshot.h" #include "src/mc/mc_private.h" #include "src/mc/mc_mmu.h" #include "src/mc/PageStore.hpp" extern "C" { /** @brief Find the snapshoted region from a pointer * * @param addr Pointer * @param snapshot Snapshot * @param Snapshot region in the snapshot this pointer belongs to * (or nullptr if it does not belong to any snapshot region) * */ mc_mem_region_t mc_get_snapshot_region( const void* addr, const simgrid::mc::Snapshot* snapshot, int process_index) { size_t n = snapshot->snapshot_regions.size(); for (size_t i = 0; i != n; ++i) { mc_mem_region_t region = snapshot->snapshot_regions[i].get(); if (!(region && region->contain(simgrid::mc::remote(addr)))) continue; if (region->storage_type() == simgrid::mc::StorageType::Privatized) { #if HAVE_SMPI // Use the current process index of the snapshot: if (process_index == simgrid::mc::ProcessIndexDisabled) process_index = snapshot->privatization_index; if (process_index < 0) xbt_die("Missing process index"); if (process_index >= (int) region->privatized_data().size()) xbt_die("Invalid process index"); simgrid::mc::RegionSnapshot& priv_region = region->privatized_data()[process_index]; xbt_assert(priv_region.contain(simgrid::mc::remote(addr))); return &priv_region; #else xbt_die("Privatized region in a non SMPI build (this should not happen)"); #endif } return region; } return nullptr; } /** @brief Read memory from a snapshot region broken across fragmented pages * * @param addr Process (non-snapshot) address of the data * @param region Snapshot memory region where the data is located * @param target Buffer to store the value * @param size Size of the data to read in bytes * @return Pointer where the data is located (target buffer of original location) */ const void* MC_region_read_fragmented(mc_mem_region_t region, void* target, const void* addr, size_t size) { // Last byte of the memory area: void* end = (char*) addr + size - 1; // TODO, we assume the chunks are aligned to natural chunk boundaries. // We should remove this assumption. // Page of the last byte of the memory area: size_t page_end = simgrid::mc::mmu::split((std::uintptr_t) end).first; void* dest = target; if (dest==nullptr) xbt_die("Missing destination buffer for fragmented memory access"); // Read each page: while (simgrid::mc::mmu::split((std::uintptr_t) addr).first != page_end) { void* snapshot_addr = mc_translate_address_region_chunked((uintptr_t) addr, region); void* next_page = (void*) simgrid::mc::mmu::join( simgrid::mc::mmu::split((std::uintptr_t) addr).first + 1, 0); size_t readable = (char*) next_page - (char*) addr; memcpy(dest, snapshot_addr, readable); addr = (char*) addr + readable; dest = (char*) dest + readable; size -= readable; } // Read the end: void* snapshot_addr = mc_translate_address_region_chunked((uintptr_t)addr, region); memcpy(dest, snapshot_addr, size); return target; } /** Compare memory between snapshots (with known regions) * * @param addr1 Address in the first snapshot * @param snapshot2 Region of the address in the first snapshot * @param addr2 Address in the second snapshot * @param snapshot2 Region of the address in the second snapshot * @return same as memcmp * */ int MC_snapshot_region_memcmp( const void* addr1, mc_mem_region_t region1, const void* addr2, mc_mem_region_t region2, size_t size) { // Using alloca() for large allocations may trigger stack overflow: // use malloc if the buffer is too big. bool stack_alloc = size < 64; const bool region1_need_buffer = region1==nullptr || region1->storage_type()==simgrid::mc::StorageType::Flat; const bool region2_need_buffer = region2==nullptr || region2->storage_type()==simgrid::mc::StorageType::Flat; void* buffer1a = region1_need_buffer ? nullptr : stack_alloc ? alloca(size) : malloc(size); void* buffer2a = region2_need_buffer ? nullptr : stack_alloc ? alloca(size) : malloc(size); const void* buffer1 = MC_region_read(region1, buffer1a, addr1, size); const void* buffer2 = MC_region_read(region2, buffer2a, addr2, size); int res; if (buffer1 == buffer2) res = 0; else res = memcmp(buffer1, buffer2, size); if (!stack_alloc) { free(buffer1a); free(buffer2a); } return res; } /** Compare memory between snapshots * * @param addr1 Address in the first snapshot * @param snapshot1 First snapshot * @param addr2 Address in the second snapshot * @param snapshot2 Second snapshot * @return same as memcmp * */ int MC_snapshot_memcmp( const void* addr1, simgrid::mc::Snapshot* snapshot1, const void* addr2, simgrid::mc::Snapshot* snapshot2, int process_index, size_t size) { mc_mem_region_t region1 = mc_get_snapshot_region(addr1, snapshot1, process_index); mc_mem_region_t region2 = mc_get_snapshot_region(addr2, snapshot2, process_index); return MC_snapshot_region_memcmp(addr1, region1, addr2, region2, size); } namespace simgrid { namespace mc { Snapshot::Snapshot(Process* process) : AddressSpace(process), num_state(0), heap_bytes_used(0), enabled_processes(), privatization_index(0), hash(0) { } Snapshot::~Snapshot() { } const void* Snapshot::read_bytes(void* buffer, std::size_t size, RemotePtr address, int process_index, ReadOptions options) const { mc_mem_region_t region = mc_get_snapshot_region((void*)address.address(), this, process_index); if (region) { const void* res = MC_region_read(region, buffer, (void*)address.address(), size); if (buffer == res || options & ReadOptions::lazy()) return res; else { memcpy(buffer, res, size); return buffer; } } else return this->process()->read_bytes( buffer, size, address, process_index, options); } } } } #ifdef SIMGRID_TEST #include #include #include #include "src/mc/mc_private.h" #include "src/mc/mc_snapshot.h" #include "src/mc/mc_mmu.h" XBT_TEST_SUITE("mc_snapshot", "Snapshots"); static inline void init_memory(void* mem, size_t size) { char* dest = (char*) mem; for (size_t i = 0; i < size; ++i) { dest[i] = rand() & 255; } } static void test_snapshot(bool sparse_checkpoint); XBT_TEST_UNIT("flat_snapshot", test_flat_snapshots, "Test flat snapshots") { test_snapshot(0); } XBT_TEST_UNIT("page_snapshots", test_per_snpashots, "Test per-page snapshots") { test_snapshot(1); } static void test_snapshot(bool sparse_checkpoint) { xbt_test_add("Initialisation"); _sg_mc_sparse_checkpoint = sparse_checkpoint; xbt_assert(xbt_pagesize == getpagesize()); xbt_assert(1 << xbt_pagebits == xbt_pagesize); std::unique_ptr process(new simgrid::mc::Process(getpid(), -1)); process->init(); mc_model_checker = new ::simgrid::mc::ModelChecker(std::move(process)); for(int n=1; n!=256; ++n) { // Store region page(s): size_t byte_size = n * xbt_pagesize; void* source = mmap(nullptr, byte_size, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0); xbt_assert(source!=MAP_FAILED, "Could not allocate source memory"); // Init memory and take snapshots: init_memory(source, byte_size); simgrid::mc::RegionSnapshot region0 = simgrid::mc::sparse_region( simgrid::mc::RegionType::Unknown, source, source, byte_size); for(int i=0; i #include #include #include #include #include "src/mc/Process.hpp" #include "src/mc/mc_unw.h" /** \file * Libunwind namespace implementation using process_vm_readv. *. * This implem */ /** Partial structure of libunwind-ptrace context in order to get the PID * * HACK, The context type for libunwind-race is an opaque type. * We need to get the PID which is the first field. This is a hack * which might break if the libunwind-ptrace structure changes. */ struct _UPT_info { pid_t pid; // Other things; }; /** Get the PID of a `libunwind-ptrace` context */ static inline pid_t _UPT_getpid(void* arg) { struct _UPT_info* info = (_UPT_info*) arg; return info->pid; } /** Read from the memory, avoid using `ptrace` (libunwind method) */ static int access_mem(const unw_addr_space_t as, const unw_word_t addr, unw_word_t* const valp, const int write, void* const arg) { if (write) return - UNW_EINVAL; pid_t pid = _UPT_getpid(arg); size_t size = sizeof(unw_word_t); #if HAVE_PROCESS_VM_READV // process_vm_read implementation. // This is only available since Linux 3.2. struct iovec local = { valp, size }; struct iovec remote = { (void*) addr, size }; ssize_t s = process_vm_readv(pid, &local, 1, &remote, 1, 0); if (s >= 0) { if ((size_t) s != size) return - UNW_EINVAL; else return 0; } if (s < 0 && errno != ENOSYS) return - UNW_EINVAL; #endif // /proc/${pid}/mem implementation. // On recent kernels, we do not need to ptrace the target process. // On older kernels, it is necessary to ptrace the target process. size_t count = size; off_t off = (off_t) addr; char* buf = (char*) valp; int fd = simgrid::mc::open_vm(pid, O_RDONLY); if (fd < 0) return - UNW_EINVAL; while (1) { ssize_t s = pread(fd, buf, count, off); if (s == 0) { close(fd); return - UNW_EINVAL; } if (s == -1) break; count -= s; buf += s; off += s; if (count == 0) { close(fd); return 0; } } close(fd); // ptrace implementation. // We need to have PTRACE_ATTACH-ed it before. return _UPT_access_mem(as, addr, valp, write, arg); } namespace simgrid { namespace unw { /** Virtual table for our `libunwind-process_vm_readv` implementation. * * This implementation reuse most the code of `libunwind-ptrace` but * does not use ptrace() to read the target process memory by * `process_vm_readv()` or `/dev/${pid}/mem` if possible. * * Does not support any MC-specific behaviour (privatisation, snapshots) * and `ucontext_t`. * * It works with `void*` contexts allocated with `_UPT_create(pid)`. */ // TODO, we could get rid of this if we properly stop the model-checked // process before reading the memory. static unw_accessors_t accessors = { &_UPT_find_proc_info, &_UPT_put_unwind_info, &_UPT_get_dyn_info_list_addr, &access_mem, &_UPT_access_reg, &_UPT_access_fpreg, &_UPT_resume, &_UPT_get_proc_name }; unw_addr_space_t create_addr_space() { return unw_create_addr_space(&accessors, BYTE_ORDER); } void* create_context(unw_addr_space_t as, pid_t pid) { return _UPT_create(pid); } } } SimGrid-3.14.159/src/mc/mc_state.h000644 001750 001750 00000010045 13031031270 020224 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_STATE_H #define SIMGRID_MC_STATE_H #include #include #include #include #include #include "src/simix/smx_private.h" #include "src/kernel/activity/SynchroIo.hpp" #include "src/kernel/activity/SynchroComm.hpp" #include "src/kernel/activity/SynchroRaw.hpp" #include "src/kernel/activity/SynchroSleep.hpp" #include "src/kernel/activity/SynchroExec.hpp" #include "src/mc/mc_snapshot.h" #include "src/mc/mc_record.h" #include "src/mc/Transition.hpp" namespace simgrid { namespace mc { enum class PatternCommunicationType { none = 0, send = 1, receive = 2, }; struct PatternCommunication { int num = 0; smx_activity_t comm_addr; PatternCommunicationType type = PatternCommunicationType::send; unsigned long src_proc = 0; unsigned long dst_proc = 0; const char *src_host = nullptr; const char *dst_host = nullptr; std::string rdv; std::vector data; int tag = 0; int index = 0; PatternCommunication() { std::memset(&comm_addr, 0, sizeof(comm_addr)); } PatternCommunication dup() const { simgrid::mc::PatternCommunication res; // num? res.comm_addr = this->comm_addr; res.type = this->type; // src_proc? // dst_proc? res.dst_proc = this->dst_proc; res.dst_host = this->dst_host; res.rdv = this->rdv; res.data = this->data; // tag? res.index = this->index; return res; } }; /* On every state, each process has an entry of the following type */ class ProcessState { /* Possible exploration status of a process in a state */ enum class InterleavingType { /** We do not have to execute this process transitions */ disabled = 0, /** We still have to execute (some of) this process transitions */ interleave, /** We have already executed this process transitions */ done, }; /** Exploration control information */ InterleavingType state = InterleavingType::disabled; public: /** Number of times that the process was interleaved */ // TODO, make this private unsigned int interleave_count = 0; bool isDisabled() const { return this->state == InterleavingType::disabled; } bool isDone() const { return this->state == InterleavingType::done; } bool isToInterleave() const { return this->state == InterleavingType::interleave; } void interleave() { this->state = InterleavingType::interleave; this->interleave_count = 0; } void setDone() { this->state = InterleavingType::done; } }; /* A node in the exploration graph (kind-of) */ struct XBT_PRIVATE State { /** Sequential state number (used for debugging) */ int num = 0; /** State's exploration status by process */ std::vector processStates; Transition transition; /** The simcall which was executed */ s_smx_simcall_t executed_req; /* Internal translation of the simcall * * SIMCALL_COMM_TESTANY is translated to a SIMCALL_COMM_TEST * and SIMCALL_COMM_WAITANY to a SIMCALL_COMM_WAIT. */ s_smx_simcall_t internal_req; /* Can be used as a copy of the remote synchro object */ simgrid::mc::Remote internal_comm; /** Snapshot of system state (if needed) */ std::shared_ptr system_state; // For CommunicationDeterminismChecker std::vector> incomplete_comm_pattern; std::vector communicationIndices; State(); std::size_t interleaveSize() const; void interleave(smx_actor_t process) { this->processStates[process->pid].interleave(); } Transition getTransition() const; }; } } XBT_PRIVATE simgrid::mc::State* MC_state_new(unsigned long state_number); XBT_PRIVATE smx_simcall_t MC_state_get_request(simgrid::mc::State* state); #endif SimGrid-3.14.159/src/mc/Session.hpp000644 001750 001750 00000004425 13031031270 020415 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_SESSION_HPP #define SIMGRID_MC_SESSION_HPP #ifdef __linux__ #include #endif #include #include #include #include #include #include #include "src/mc/mc_forward.hpp" #include "src/mc/ModelChecker.hpp" namespace simgrid { namespace mc { /** A model-checking session * * This is expected to become the interface used by model-checking * algorithms to control the execution of the model-checked process * and the exploration of the execution graph. Model-checking * algorithms should be able to be written in high-level languages * (e.g. Python) using bindings on this interface. */ class Session { private: std::unique_ptr modelChecker_; std::shared_ptr initialSnapshot_; private: Session(pid_t pid, int socket); // No copy: Session(Session const&) = delete; Session& operator=(Session const&) = delete; public: ~Session(); void close(); public: void initialize(); void execute(Transition const& transition); void logState(); void restoreInitialState(); public: // static constructors /** Create a new session by forking * * This sets up the environment for the model-checked process * (environoment variables, sockets, etc.). * * The code is expected to `exec` the model-checker program. */ static Session* fork(std::function code); /** Spawn a model-checked process * * @param path full path of the executable * @param argv arguments for the model-checked process (NULL-terminated) */ static Session* spawnv(const char *path, char *const argv[]); /** Spawn a model-checked process (using PATH) * * @param file file name of the executable (found using `PATH`) * @param argv arguments for the model-checked process (NULL-terminated) */ static Session* spawnvp(const char *file, char *const argv[]); }; // Temporary extern simgrid::mc::Session* session; } } #endif SimGrid-3.14.159/src/mc/LocationList.cpp000644 001750 001750 00000006560 13031031270 021373 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2004-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include "src/mc/mc_dwarf.hpp" #include "src/mc/ObjectInformation.hpp" #include "src/mc/LocationList.hpp" namespace simgrid { namespace dwarf { /** Resolve a location expression */ Location resolve( simgrid::dwarf::DwarfExpression const& expression, simgrid::mc::ObjectInformation* object_info, unw_cursor_t * c, void *frame_pointer_address, simgrid::mc::AddressSpace* address_space, int process_index) { simgrid::dwarf::ExpressionContext context; context.frame_base = frame_pointer_address; context.cursor = c; context.address_space = address_space; context.object_info = object_info; context.process_index = process_index; if (!expression.empty() && expression[0].atom >= DW_OP_reg0 && expression[0].atom <= DW_OP_reg31) { int dwarf_register = expression[0].atom - DW_OP_reg0; xbt_assert(c, "Missing frame context for register operation DW_OP_reg%i", dwarf_register); return Location(dwarf_register_to_libunwind(dwarf_register)); } simgrid::dwarf::ExpressionStack stack; simgrid::dwarf::execute(expression, context, stack); return Location((void*) stack.top()); } // TODO, move this in a method of LocationList static simgrid::dwarf::DwarfExpression const* find_expression( simgrid::dwarf::LocationList const& locations, unw_word_t ip) { for (simgrid::dwarf::LocationListEntry const& entry : locations) if (entry.valid_for_ip(ip)) return &entry.expression(); return nullptr; } Location resolve( simgrid::dwarf::LocationList const& locations, simgrid::mc::ObjectInformation* object_info, unw_cursor_t * c, void *frame_pointer_address, simgrid::mc::AddressSpace* address_space, int process_index) { unw_word_t ip = 0; if (c && unw_get_reg(c, UNW_REG_IP, &ip)) xbt_die("Could not resolve IP"); simgrid::dwarf::DwarfExpression const* expression = find_expression(locations, ip); if (!expression) xbt_die("Could not resolve location"); return simgrid::dwarf::resolve( *expression, object_info, c, frame_pointer_address, address_space, process_index); } LocationList location_list( simgrid::mc::ObjectInformation& info, Dwarf_Attribute& attr) { LocationList locations; std::ptrdiff_t offset = 0; while (1) { Dwarf_Addr base, start, end; Dwarf_Op *ops; std::size_t len; offset = dwarf_getlocations( &attr, offset, &base, &start, &end, &ops, &len); if (offset == 0) break; else if (offset == -1) xbt_die("Error while loading location list"); std::uint64_t base_address = (std::uint64_t) info.base_address(); LocationListEntry::range_type range; if (start == 0) // If start == 0, this is not a location list: range = { 0, UINT64_MAX }; else range = { base_address + start, base_address + end }; locations.push_back({ DwarfExpression(ops, ops+len), range }); } return locations; } } }SimGrid-3.14.159/src/mc/mc_comm_pattern.h000644 001750 001750 00000004136 13031031270 021600 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_COMM_PATTERN_H #define SIMGRID_MC_COMM_PATTERN_H #include #include #include #include #include "src/simix/smx_private.h" #include "src/smpi/private.h" #include #include "src/mc/mc_state.h" namespace simgrid { namespace mc { struct PatternCommunicationList { unsigned int index_comm = 0; std::vector> list; }; } } SG_BEGIN_DECL() /** * Type: `xbt_dynar_t` */ extern XBT_PRIVATE xbt_dynar_t initial_communications_pattern; /** * Type: `xbt_dynar_t>` */ extern XBT_PRIVATE xbt_dynar_t incomplete_communications_pattern; typedef enum { MC_CALL_TYPE_NONE, MC_CALL_TYPE_SEND, MC_CALL_TYPE_RECV, MC_CALL_TYPE_WAIT, MC_CALL_TYPE_WAITANY, } e_mc_call_type_t; typedef enum { NONE_DIFF, TYPE_DIFF, RDV_DIFF, TAG_DIFF, SRC_PROC_DIFF, DST_PROC_DIFF, DATA_SIZE_DIFF, DATA_DIFF, } e_mc_comm_pattern_difference_t; static inline e_mc_call_type_t MC_get_call_type(smx_simcall_t req) { switch(req->call) { case SIMCALL_COMM_ISEND: return MC_CALL_TYPE_SEND; case SIMCALL_COMM_IRECV: return MC_CALL_TYPE_RECV; case SIMCALL_COMM_WAIT: return MC_CALL_TYPE_WAIT; case SIMCALL_COMM_WAITANY: return MC_CALL_TYPE_WAITANY; default: return MC_CALL_TYPE_NONE; } } XBT_PRIVATE void MC_handle_comm_pattern(e_mc_call_type_t call_type, smx_simcall_t request, int value, xbt_dynar_t current_pattern, int backtracking); XBT_PRIVATE void MC_restore_communications_pattern(simgrid::mc::State* state); XBT_PRIVATE void MC_state_copy_incomplete_communications_pattern(simgrid::mc::State* state); XBT_PRIVATE void MC_state_copy_index_communications_pattern(simgrid::mc::State* state); SG_END_DECL() #endif SimGrid-3.14.159/src/mc/Checker.cpp000644 001750 001750 00000001475 13031031270 020333 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include "src/mc/Checker.hpp" #include "src/mc/ModelChecker.hpp" namespace simgrid { namespace mc { Checker::Checker(Session& session) : session_(&session) { xbt_assert(mc_model_checker); xbt_assert(mc_model_checker->getChecker() == nullptr); mc_model_checker->setChecker(this); } Checker::~Checker() { } // virtual RecordTrace Checker::getRecordTrace() { return {}; } // virtual std::vector Checker::getTextualTrace() { return {}; } // virtual void Checker::logState() { } } } SimGrid-3.14.159/src/mc/Process.cpp000644 001750 001750 00000045561 13031031270 020411 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #define _FILE_OFFSET_BITS 64 #include #include #include #include #include #include #include #include #include #include #include // PROT_* #include #include #include #include #include #include #include #include "src/mc/mc_unw.h" #include "src/mc/mc_snapshot.h" #include "src/mc/mc_ignore.h" #include "src/mc/mc_smx.h" #include "src/mc/Process.hpp" #include "src/mc/AddressSpace.hpp" #include "src/mc/ObjectInformation.hpp" #include "src/mc/Variable.hpp" using simgrid::mc::remote; XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_process, mc, "MC process information"); // ***** Helper stuff namespace simgrid { namespace mc { #define SO_RE "\\.so[\\.0-9]*$" #define VERSION_RE "-[\\.0-9-]*$" // List of library which memory segments are not considered: static const char* const filtered_libraries[] = { #ifdef __linux__ "ld", #elif defined __FreeBSD__ "ld-elf", "ld-elf32", "libkvm", /* kernel data access library */ "libprocstat", /* process and file information retrieval */ "libthr", /* thread library */ "libutil", #endif "libasan", /* gcc sanitizers */ "libargp", /* workarounds for glibc-less systems */ "libtsan", "libubsan", "libbz2", "libboost_chrono", "libboost_context", "libboost_context-mt", "libboost_system", "libboost_thread", "libc", "libc++", "libcdt", "libcgraph", "libcxxrt", "libdl", "libdw", "libelf", "libevent", "libgcc_s", "liblua5.1", "liblua5.3", "liblzma", "libm", "libpthread", "librt", "libstdc++", "libunwind", "libunwind-x86_64", "libunwind-x86", "libunwind-ptrace", "libz"}; static bool is_simgrid_lib(const char* libname) { return !strcmp(libname, "libsimgrid"); } static bool is_filtered_lib(const char* libname) { for (const char* filtered_lib : filtered_libraries) if (strcmp(libname, filtered_lib)==0) return true; return false; } struct s_mc_memory_map_re { regex_t so_re; regex_t version_re; }; static char* get_lib_name(const char* pathname, struct s_mc_memory_map_re* res) { char* map_basename = xbt_basename(pathname); regmatch_t match; if(regexec(&res->so_re, map_basename, 1, &match, 0)) { free(map_basename); return nullptr; } char* libname = strndup(map_basename, match.rm_so); free(map_basename); map_basename = nullptr; // Strip the version suffix: if(libname && !regexec(&res->version_re, libname, 1, &match, 0)) { char* temp = libname; libname = strndup(temp, match.rm_so); free(temp); } return libname; } static ssize_t pread_whole(int fd, void *buf, size_t count, std::uint64_t offset) { char* buffer = (char*) buf; ssize_t real_count = count; while (count) { ssize_t res = pread(fd, buffer, count, (std::int64_t) offset); if (res > 0) { count -= res; buffer += res; offset += res; } else if (res==0) return -1; else if (errno != EINTR) { perror("pread_whole"); return -1; } } return real_count; } static ssize_t pwrite_whole(int fd, const void *buf, size_t count, off_t offset) { const char* buffer = (const char*) buf; ssize_t real_count = count; while (count) { ssize_t res = pwrite(fd, buffer, count, offset); if (res > 0) { count -= res; buffer += res; offset += res; } else if (res==0) return -1; else if (errno != EINTR) return -1; } return real_count; } static pthread_once_t zero_buffer_flag = PTHREAD_ONCE_INIT; static const void* zero_buffer; static const size_t zero_buffer_size = 10 * 4096; static void zero_buffer_init(void) { int fd = open("/dev/zero", O_RDONLY); if (fd<0) xbt_die("Could not open /dev/zero"); zero_buffer = mmap(nullptr, zero_buffer_size, PROT_READ, MAP_SHARED, fd, 0); if (zero_buffer == MAP_FAILED) xbt_die("Could not map the zero buffer"); close(fd); } int open_vm(pid_t pid, int flags) { const size_t buffer_size = 30; char buffer[buffer_size]; int res = snprintf(buffer, buffer_size, "/proc/%lli/mem", (long long) pid); if (res < 0 || (size_t) res >= buffer_size) { errno = ENAMETOOLONG; return -1; } return open(buffer, flags); } // ***** Process Process::Process(pid_t pid, int sockfd) : AddressSpace(this), pid_(pid), channel_(sockfd), running_(true) {} void Process::init() { this->memory_map_ = simgrid::xbt::get_memory_map(this->pid_); this->init_memory_map_info(); int fd = open_vm(this->pid_, O_RDWR); if (fd<0) xbt_die("Could not open file for process virtual address space"); this->memory_file = fd; // Read std_heap (is a struct mdesc*): simgrid::mc::Variable* std_heap_var = this->find_variable("__mmalloc_default_mdp"); if (!std_heap_var) xbt_die("No heap information in the target process"); if(!std_heap_var->address) xbt_die("No constant address for this variable"); this->read_bytes(&this->heap_address, sizeof(struct mdesc*), remote(std_heap_var->address), simgrid::mc::ProcessIndexDisabled); this->smx_process_infos.clear(); this->smx_old_process_infos.clear(); this->unw_addr_space = simgrid::mc::UnwindContext::createUnwindAddressSpace(); this->unw_underlying_addr_space = simgrid::unw::create_addr_space(); this->unw_underlying_context = simgrid::unw::create_context( this->unw_underlying_addr_space, this->pid_); } Process::~Process() { if (this->memory_file >= 0) close(this->memory_file); if (this->unw_underlying_addr_space != unw_local_addr_space) { unw_destroy_addr_space(this->unw_underlying_addr_space); _UPT_destroy(this->unw_underlying_context); } unw_destroy_addr_space(this->unw_addr_space); } /** Refresh the information about the process * * Do not use directly, this is used by the getters when appropriate * in order to have fresh data. */ void Process::refresh_heap() { // Read/dereference/refresh the std_heap pointer: if (!this->heap) this->heap = std::unique_ptr(new s_xbt_mheap_t()); this->read_bytes(this->heap.get(), sizeof(struct mdesc), remote(this->heap_address), simgrid::mc::ProcessIndexDisabled); this->cache_flags_ |= Process::cache_heap; } /** Refresh the information about the process * * Do not use direclty, this is used by the getters when appropriate * in order to have fresh data. * */ void Process::refresh_malloc_info() { // Refresh process->heapinfo: if (this->cache_flags_ & Process::cache_malloc) return; size_t count = this->heap->heaplimit + 1; if (this->heap_info.size() < count) this->heap_info.resize(count); this->read_bytes(this->heap_info.data(), count * sizeof(malloc_info), remote(this->heap->heapinfo), simgrid::mc::ProcessIndexDisabled); this->cache_flags_ |= Process::cache_malloc; } /** @brief Finds the range of the different memory segments and binary paths */ void Process::init_memory_map_info() { XBT_DEBUG("Get debug information ..."); this->maestro_stack_start_ = nullptr; this->maestro_stack_end_ = nullptr; this->object_infos.resize(0); this->binary_info = nullptr; this->libsimgrid_info = nullptr; struct s_mc_memory_map_re res; if(regcomp(&res.so_re, SO_RE, 0) || regcomp(&res.version_re, VERSION_RE, 0)) xbt_die(".so regexp did not compile"); std::vector const& maps = this->memory_map_; const char* current_name = nullptr; this->object_infos.clear(); for (size_t i=0; i < maps.size(); i++) { simgrid::xbt::VmMap const& reg = maps[i]; const char* pathname = maps[i].pathname.c_str(); // Nothing to do if (maps[i].pathname.empty()) { current_name = nullptr; continue; } // [stack], [vvar], [vsyscall], [vdso] ... if (pathname[0] == '[') { if ((reg.prot & PROT_WRITE) && !memcmp(pathname, "[stack]", 7)) { this->maestro_stack_start_ = remote(reg.start_addr); this->maestro_stack_end_ = remote(reg.end_addr); } current_name = nullptr; continue; } if (current_name && strcmp(current_name, pathname)==0) continue; current_name = pathname; if (!(reg.prot & PROT_READ) && (reg.prot & PROT_EXEC)) continue; const bool is_executable = !i; char* libname = nullptr; if (!is_executable) { libname = get_lib_name(pathname, &res); if(!libname) continue; if (is_filtered_lib(libname)) { free(libname); continue; } } std::shared_ptr info = simgrid::mc::createObjectInformation(this->memory_map_, pathname); this->object_infos.push_back(info); if (is_executable) this->binary_info = info; else if (libname && is_simgrid_lib(libname)) this->libsimgrid_info = info; free(libname); } regfree(&res.so_re); regfree(&res.version_re); // Resolve time (including across different objects): for (auto const& object_info : this->object_infos) postProcessObjectInformation(this, object_info.get()); xbt_assert(this->maestro_stack_start_, "Did not find maestro_stack_start"); xbt_assert(this->maestro_stack_end_, "Did not find maestro_stack_end"); XBT_DEBUG("Get debug information done !"); } std::shared_ptr Process::find_object_info(RemotePtr addr) const { for (auto const& object_info : this->object_infos) if (addr.address() >= (std::uint64_t)object_info->start && addr.address() <= (std::uint64_t)object_info->end) return object_info; return nullptr; } std::shared_ptr Process::find_object_info_exec(RemotePtr addr) const { for (std::shared_ptr const& info : this->object_infos) if (addr.address() >= (std::uint64_t) info->start_exec && addr.address() <= (std::uint64_t) info->end_exec) return info; return nullptr; } std::shared_ptr Process::find_object_info_rw(RemotePtr addr) const { for (std::shared_ptr const& info : this->object_infos) if (addr.address() >= (std::uint64_t)info->start_rw && addr.address() <= (std::uint64_t)info->end_rw) return info; return nullptr; } simgrid::mc::Frame* Process::find_function(RemotePtr ip) const { std::shared_ptr info = this->find_object_info_exec(ip); return info ? info->find_function((void*) ip.address()) : nullptr; } /** Find (one occurrence of) the named variable definition */ simgrid::mc::Variable* Process::find_variable(const char* name) const { // First lookup the variable in the executable shared object. // A global variable used directly by the executable code from a library // is reinstanciated in the executable memory .data/.bss. // We need to look up the variable in the executable first. if (this->binary_info) { std::shared_ptr const& info = this->binary_info; simgrid::mc::Variable* var = info->find_variable(name); if (var) return var; } for (std::shared_ptr const& info : this->object_infos) { simgrid::mc::Variable* var = info->find_variable(name); if (var) return var; } return nullptr; } void Process::read_variable(const char* name, void* target, size_t size) const { simgrid::mc::Variable* var = this->find_variable(name); if (!var->address) xbt_die("No simple location for this variable"); if (!var->type->full_type) xbt_die("Partial type for %s, cannot check size", name); if ((size_t) var->type->full_type->byte_size != size) xbt_die("Unexpected size for %s (expected %zi, was %zi)", name, size, (size_t) var->type->full_type->byte_size); this->read_bytes(target, size, remote(var->address)); } std::string Process::read_string(RemotePtr address) const { if (!address) return {}; // TODO, use std::vector with .data() in C++17 to avoid useless copies std::vector res(128); off_t off = 0; while (1) { ssize_t c = pread(this->memory_file, res.data() + off, res.size() - off, (off_t) address.address() + off); if (c == -1) { if (errno == EINTR) continue; else xbt_die("Could not read from from remote process"); } if (c==0) xbt_die("Could not read string from remote process"); void* p = memchr(res.data() + off, '\0', c); if (p) return std::string(res.data()); off += c; if (off == (off_t) res.size()) res.resize(res.size() * 2); } } const void *Process::read_bytes(void* buffer, std::size_t size, RemotePtr address, int process_index, ReadOptions options) const { if (process_index != simgrid::mc::ProcessIndexDisabled) { std::shared_ptr const& info = this->find_object_info_rw((void*)address.address()); // Segment overlap is not handled. #if HAVE_SMPI if (info.get() && this->privatized(*info)) { if (process_index < 0) xbt_die("Missing process index"); if (process_index >= (int) MC_smpi_process_count()) xbt_die("Invalid process index"); // Read smpi_privatisation_regions from MCed: smpi_privatisation_region_t remote_smpi_privatisation_regions = mc_model_checker->process().read_variable( "smpi_privatisation_regions"); s_smpi_privatisation_region_t privatisation_region = mc_model_checker->process().read( remote(remote_smpi_privatisation_regions + process_index)); // Address translation in the privatization segment: size_t offset = address.address() - (std::uint64_t)info->start_rw; address = remote((char*)privatisation_region.address + offset); } #endif } if (pread_whole(this->memory_file, buffer, size, address.address()) < 0) xbt_die("Read from process %lli failed", (long long) this->pid_); return buffer; } /** Write data to a process memory * * @param buffer local memory address (source) * @param len data size * @param address target process memory address (target) */ void Process::write_bytes(const void* buffer, size_t len, RemotePtr address) { if (pwrite_whole(this->memory_file, buffer, len, address.address()) < 0) xbt_die("Write to process %lli failed", (long long) this->pid_); } void Process::clear_bytes(RemotePtr address, size_t len) { pthread_once(&zero_buffer_flag, zero_buffer_init); while (len) { size_t s = len > zero_buffer_size ? zero_buffer_size : len; this->write_bytes(zero_buffer, s, address); address = remote((char*) address.address() + s); len -= s; } } void Process::ignore_region(std::uint64_t addr, std::size_t size) { IgnoredRegion region; region.addr = addr; region.size = size; if (ignored_regions_.empty()) { ignored_regions_.push_back(region); return; } unsigned int cursor = 0; IgnoredRegion* current_region = nullptr; int start = 0; int end = ignored_regions_.size() - 1; while (start <= end) { cursor = (start + end) / 2; current_region = &ignored_regions_[cursor]; if (current_region->addr == addr) { if (current_region->size == size) return; else if (current_region->size < size) start = cursor + 1; else end = cursor - 1; } else if (current_region->addr < addr) start = cursor + 1; else end = cursor - 1; } std::size_t position; if (current_region->addr == addr) { if (current_region->size < size) position = cursor + 1; else position = cursor; } else if (current_region->addr < addr) position = cursor + 1; else position = cursor; ignored_regions_.insert( ignored_regions_.begin() + position, region); } void Process::ignore_heap(IgnoredHeapRegion const& region) { if (ignored_heap_.empty()) { ignored_heap_.push_back(std::move(region)); return; } typedef std::vector::size_type size_type; size_type start = 0; size_type end = ignored_heap_.size() - 1; // Binary search the position of insertion: size_type cursor; while (start <= end) { cursor = start + (end - start) / 2; auto& current_region = ignored_heap_[cursor]; if (current_region.address == region.address) return; else if (current_region.address < region.address) start = cursor + 1; else if (cursor != 0) end = cursor - 1; // Avoid underflow: else break; } // Insert it mc_heap_ignore_region_t: if (ignored_heap_[cursor].address < region.address) ++cursor; ignored_heap_.insert( ignored_heap_.begin() + cursor, region); } void Process::unignore_heap(void *address, size_t size) { typedef std::vector::size_type size_type; size_type start = 0; size_type end = ignored_heap_.size() - 1; // Binary search: size_type cursor; while (start <= end) { cursor = (start + end) / 2; auto& region = ignored_heap_[cursor]; if (region.address == address) { ignored_heap_.erase(ignored_heap_.begin() + cursor); return; } else if (region.address < address) start = cursor + 1; else if ((char *) region.address <= ((char *) address + size)) { ignored_heap_.erase(ignored_heap_.begin() + cursor); return; } else if (cursor != 0) end = cursor - 1; // Avoid underflow: else break; } } void Process::ignore_local_variable(const char *var_name, const char *frame_name) { if (frame_name != nullptr && strcmp(frame_name, "*") == 0) frame_name = nullptr; for (std::shared_ptr const& info : this->object_infos) info->remove_local_variable(var_name, frame_name); } std::vector& Process::simix_processes() { this->refresh_simix(); return smx_process_infos; } std::vector& Process::old_simix_processes() { this->refresh_simix(); return smx_old_process_infos; } void Process::dumpStack() { unw_addr_space_t as = unw_create_addr_space(&_UPT_accessors, BYTE_ORDER); if (as == nullptr) { XBT_ERROR("Could not initialize ptrace address space"); return; } void* context = _UPT_create(this->pid_); if (context == nullptr) { unw_destroy_addr_space(as); XBT_ERROR("Could not initialize ptrace context"); return; } unw_cursor_t cursor; if (unw_init_remote(&cursor, as, context) != 0) { _UPT_destroy(context); unw_destroy_addr_space(as); XBT_ERROR("Could not initialiez ptrace cursor"); return; } simgrid::mc::dumpStack(stderr, cursor); _UPT_destroy(context); unw_destroy_addr_space(as); return; } } } SimGrid-3.14.159/src/mc/mc_dwarf.hpp000644 001750 001750 00000001655 13031031270 020556 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #if !defined(SIMGRID_MC_DWARF_HPP) #define SIMGRID_MC_DWARF_HPP #include #include #include #include #define DW_LANG_Objc DW_LANG_ObjC /* fix spelling error in older dwarf.h */ #include #include "src/mc/mc_forward.hpp" namespace simgrid { namespace dwarf { XBT_PRIVATE const char* attrname(int attr); XBT_PRIVATE const char* tagname(int tag); XBT_PRIVATE void* resolve_member( const void* base, simgrid::mc::Type* type, simgrid::mc::Member* member, simgrid::mc::AddressSpace* snapshot, int process_index); XBT_PRIVATE int dwarf_register_to_libunwind(int dwarf_register); } } #endif SimGrid-3.14.159/src/mc/SafetyChecker.hpp000644 001750 001750 00000002473 13031031270 021513 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_SAFETY_CHECKER_HPP #define SIMGRID_MC_SAFETY_CHECKER_HPP #include #include #include #include #include "src/mc/mc_forward.hpp" #include "src/mc/Checker.hpp" #include "src/mc/VisitedState.hpp" namespace simgrid { namespace mc { class XBT_PRIVATE SafetyChecker : public Checker { simgrid::mc::ReductionMode reductionMode_ = simgrid::mc::ReductionMode::unset; public: SafetyChecker(Session& session); ~SafetyChecker(); int run() override; RecordTrace getRecordTrace() override; std::vector getTextualTrace() override; void logState() override; private: // Temp void init(); bool checkNonTermination(simgrid::mc::State* current_state); int backtrack(); void restoreState(); private: /** Stack representing the position in the exploration graph */ std::list> stack_; simgrid::mc::VisitedStates visitedStates_; std::unique_ptr visitedState_; unsigned long expandedStatesCount_ = 0; }; } } #endif SimGrid-3.14.159/src/mc/CommunicationDeterminismChecker.cpp000644 001750 001750 00000055664 13031031270 025273 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include "src/mc/mc_state.h" #include "src/mc/mc_comm_pattern.h" #include "src/mc/mc_request.h" #include "src/mc/mc_safety.h" #include "src/mc/mc_private.h" #include "src/mc/mc_record.h" #include "src/mc/mc_smx.h" #include "src/mc/Client.hpp" #include "src/mc/CommunicationDeterminismChecker.hpp" #include "src/mc/mc_exit.h" #include "src/mc/VisitedState.hpp" #include "src/mc/Transition.hpp" using simgrid::mc::remote; XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_comm_determinism, mc, "Logging specific to MC communication determinism detection"); /********** Global variables **********/ xbt_dynar_t initial_communications_pattern; xbt_dynar_t incomplete_communications_pattern; /********** Static functions ***********/ static e_mc_comm_pattern_difference_t compare_comm_pattern(simgrid::mc::PatternCommunication* comm1, simgrid::mc::PatternCommunication* comm2) { if(comm1->type != comm2->type) return TYPE_DIFF; if (comm1->rdv != comm2->rdv) return RDV_DIFF; if (comm1->src_proc != comm2->src_proc) return SRC_PROC_DIFF; if (comm1->dst_proc != comm2->dst_proc) return DST_PROC_DIFF; if (comm1->tag != comm2->tag) return TAG_DIFF; if (comm1->data.size() != comm2->data.size()) return DATA_SIZE_DIFF; if (comm1->data != comm2->data) return DATA_DIFF; return NONE_DIFF; } static char* print_determinism_result(e_mc_comm_pattern_difference_t diff, int process, simgrid::mc::PatternCommunication* comm, unsigned int cursor) { char *type, *res; if (comm->type == simgrid::mc::PatternCommunicationType::send) type = bprintf("The send communications pattern of the process %d is different!", process - 1); else type = bprintf("The recv communications pattern of the process %d is different!", process - 1); switch(diff) { case TYPE_DIFF: res = bprintf("%s Different type for communication #%d", type, cursor); break; case RDV_DIFF: res = bprintf("%s Different rdv for communication #%d", type, cursor); break; case TAG_DIFF: res = bprintf("%s Different tag for communication #%d", type, cursor); break; case SRC_PROC_DIFF: res = bprintf("%s Different source for communication #%d", type, cursor); break; case DST_PROC_DIFF: res = bprintf("%s Different destination for communication #%d", type, cursor); break; case DATA_SIZE_DIFF: res = bprintf("%s\n Different data size for communication #%d", type, cursor); break; case DATA_DIFF: res = bprintf("%s\n Different data for communication #%d", type, cursor); break; default: res = nullptr; break; } return res; } static void update_comm_pattern( simgrid::mc::PatternCommunication* comm_pattern, simgrid::mc::RemotePtr comm_addr) { // HACK, type punning simgrid::mc::Remote temp_comm; mc_model_checker->process().read(temp_comm, comm_addr); simgrid::kernel::activity::Comm* comm = static_cast(temp_comm.getBuffer()); smx_actor_t src_proc = mc_model_checker->process().resolveProcess( simgrid::mc::remote(comm->src_proc)); smx_actor_t dst_proc = mc_model_checker->process().resolveProcess( simgrid::mc::remote(comm->dst_proc)); comm_pattern->src_proc = src_proc->pid; comm_pattern->dst_proc = dst_proc->pid; comm_pattern->src_host = MC_smx_process_get_host_name(src_proc); comm_pattern->dst_host = MC_smx_process_get_host_name(dst_proc); if (comm_pattern->data.size() == 0 && comm->src_buff != nullptr) { size_t buff_size; mc_model_checker->process().read( &buff_size, remote(comm->dst_buff_size)); comm_pattern->data.resize(buff_size); mc_model_checker->process().read_bytes( comm_pattern->data.data(), comm_pattern->data.size(), remote(comm->src_buff)); } } namespace simgrid { namespace mc { void CommunicationDeterminismChecker::deterministic_comm_pattern( int process, simgrid::mc::PatternCommunication* comm, int backtracking) { simgrid::mc::PatternCommunicationList* list = xbt_dynar_get_as(initial_communications_pattern, process, simgrid::mc::PatternCommunicationList*); if(!backtracking){ e_mc_comm_pattern_difference_t diff = compare_comm_pattern(list->list[list->index_comm].get(), comm); if (diff != NONE_DIFF) { if (comm->type == simgrid::mc::PatternCommunicationType::send) { this->send_deterministic = 0; if (this->send_diff != nullptr) xbt_free(this->send_diff); this->send_diff = print_determinism_result(diff, process, comm, list->index_comm + 1); } else { this->recv_deterministic = 0; if (this->recv_diff != nullptr) xbt_free(this->recv_diff); this->recv_diff = print_determinism_result(diff, process, comm, list->index_comm + 1); } if(_sg_mc_send_determinism && !this->send_deterministic){ XBT_INFO("*********************************************************"); XBT_INFO("***** Non-send-deterministic communications pattern *****"); XBT_INFO("*********************************************************"); XBT_INFO("%s", this->send_diff); xbt_free(this->send_diff); this->send_diff = nullptr; simgrid::mc::session->logState(); mc_model_checker->exit(SIMGRID_MC_EXIT_NON_DETERMINISM); }else if(_sg_mc_comms_determinism && (!this->send_deterministic && !this->recv_deterministic)) { XBT_INFO("****************************************************"); XBT_INFO("***** Non-deterministic communications pattern *****"); XBT_INFO("****************************************************"); XBT_INFO("%s", this->send_diff); XBT_INFO("%s", this->recv_diff); xbt_free(this->send_diff); this->send_diff = nullptr; xbt_free(this->recv_diff); this->recv_diff = nullptr; simgrid::mc::session->logState(); mc_model_checker->exit(SIMGRID_MC_EXIT_NON_DETERMINISM); } } } } /********** Non Static functions ***********/ void CommunicationDeterminismChecker::get_comm_pattern(xbt_dynar_t list, smx_simcall_t request, e_mc_call_type_t call_type, int backtracking) { const smx_actor_t issuer = MC_smx_simcall_get_issuer(request); simgrid::mc::PatternCommunicationList* initial_pattern = xbt_dynar_get_as( initial_communications_pattern, issuer->pid, simgrid::mc::PatternCommunicationList*); xbt_dynar_t incomplete_pattern = xbt_dynar_get_as( incomplete_communications_pattern, issuer->pid, xbt_dynar_t); std::unique_ptr pattern = std::unique_ptr( new simgrid::mc::PatternCommunication()); pattern->index = initial_pattern->index_comm + xbt_dynar_length(incomplete_pattern); if (call_type == MC_CALL_TYPE_SEND) { /* Create comm pattern */ pattern->type = simgrid::mc::PatternCommunicationType::send; pattern->comm_addr = simcall_comm_isend__get__result(request); simgrid::mc::Remote temp_synchro; mc_model_checker->process().read(temp_synchro, remote( static_cast(pattern->comm_addr))); simgrid::kernel::activity::Comm* synchro = static_cast(temp_synchro.getBuffer()); char* remote_name = mc_model_checker->process().read( (std::uint64_t)(synchro->mbox ? &synchro->mbox->name : &synchro->mbox_cpy->name)); pattern->rdv = mc_model_checker->process().read_string(remote_name); pattern->src_proc = mc_model_checker->process().resolveProcess( simgrid::mc::remote(synchro->src_proc))->pid; pattern->src_host = MC_smx_process_get_host_name(issuer); struct s_smpi_mpi_request mpi_request = mc_model_checker->process().read( (std::uint64_t) simcall_comm_isend__get__data(request)); pattern->tag = mpi_request.tag; if (synchro->src_buff != nullptr){ pattern->data.resize(synchro->src_buff_size); mc_model_checker->process().read_bytes( pattern->data.data(), pattern->data.size(), remote(synchro->src_buff)); } if(mpi_request.detached){ if (!this->initial_communications_pattern_done) { /* Store comm pattern */ simgrid::mc::PatternCommunicationList* list = xbt_dynar_get_as( initial_communications_pattern, pattern->src_proc, simgrid::mc::PatternCommunicationList*); list->list.push_back(std::move(pattern)); } else { /* Evaluate comm determinism */ this->deterministic_comm_pattern(pattern->src_proc, pattern.get(), backtracking); xbt_dynar_get_as( initial_communications_pattern, pattern->src_proc, simgrid::mc::PatternCommunicationList* )->index_comm++; } return; } } else if (call_type == MC_CALL_TYPE_RECV) { pattern->type = simgrid::mc::PatternCommunicationType::receive; pattern->comm_addr = simcall_comm_irecv__get__result(request); struct s_smpi_mpi_request mpi_request; mc_model_checker->process().read( &mpi_request, remote((struct s_smpi_mpi_request*)simcall_comm_irecv__get__data(request))); pattern->tag = mpi_request.tag; simgrid::mc::Remote temp_comm; mc_model_checker->process().read(temp_comm, remote( static_cast(pattern->comm_addr))); simgrid::kernel::activity::Comm* comm = temp_comm.getBuffer(); char* remote_name; mc_model_checker->process().read(&remote_name, remote(comm->mbox ? &comm->mbox->name : &comm->mbox_cpy->name)); pattern->rdv = mc_model_checker->process().read_string(remote_name); pattern->dst_proc = mc_model_checker->process().resolveProcess( simgrid::mc::remote(comm->dst_proc))->pid; pattern->dst_host = MC_smx_process_get_host_name(issuer); } else xbt_die("Unexpected call_type %i", (int) call_type); XBT_DEBUG("Insert incomplete comm pattern %p for process %lu", pattern.get(), issuer->pid); xbt_dynar_t dynar = xbt_dynar_get_as(incomplete_communications_pattern, issuer->pid, xbt_dynar_t); simgrid::mc::PatternCommunication* pattern2 = pattern.release(); xbt_dynar_push(dynar, &pattern2); } void CommunicationDeterminismChecker::complete_comm_pattern( xbt_dynar_t list, simgrid::mc::RemotePtr comm_addr, unsigned int issuer, int backtracking) { simgrid::mc::PatternCommunication* current_comm_pattern; unsigned int cursor = 0; std::unique_ptr comm_pattern; int completed = 0; /* Complete comm pattern */ xbt_dynar_foreach(xbt_dynar_get_as(incomplete_communications_pattern, issuer, xbt_dynar_t), cursor, current_comm_pattern) if (remote(current_comm_pattern->comm_addr) == comm_addr) { update_comm_pattern(current_comm_pattern, comm_addr); completed = 1; simgrid::mc::PatternCommunication* temp; xbt_dynar_remove_at( xbt_dynar_get_as(incomplete_communications_pattern, issuer, xbt_dynar_t), cursor, &temp); comm_pattern = std::unique_ptr(temp); XBT_DEBUG("Remove incomplete comm pattern for process %u at cursor %u", issuer, cursor); break; } if(!completed) xbt_die("Corresponding communication not found!"); simgrid::mc::PatternCommunicationList* pattern = xbt_dynar_get_as( initial_communications_pattern, issuer, simgrid::mc::PatternCommunicationList*); if (!this->initial_communications_pattern_done) /* Store comm pattern */ pattern->list.push_back(std::move(comm_pattern)); else { /* Evaluate comm determinism */ this->deterministic_comm_pattern(issuer, comm_pattern.get(), backtracking); pattern->index_comm++; } } CommunicationDeterminismChecker::CommunicationDeterminismChecker(Session& session) : Checker(session) { } CommunicationDeterminismChecker::~CommunicationDeterminismChecker() { } RecordTrace CommunicationDeterminismChecker::getRecordTrace() // override { RecordTrace res; for (auto const& state : stack_) res.push_back(state->getTransition()); return res; } std::vector CommunicationDeterminismChecker::getTextualTrace() // override { std::vector trace; for (auto const& state : stack_) { smx_simcall_t req = &state->executed_req; if (req) trace.push_back(simgrid::mc::request_to_string( req, state->transition.argument, simgrid::mc::RequestType::executed)); } return trace; } void CommunicationDeterminismChecker::logState() // override { Checker::logState(); if (_sg_mc_comms_determinism && !this->recv_deterministic && this->send_deterministic) { XBT_INFO("******************************************************"); XBT_INFO("**** Only-send-deterministic communication pattern ****"); XBT_INFO("******************************************************"); XBT_INFO("%s", this->recv_diff); } else if(_sg_mc_comms_determinism && !this->send_deterministic && this->recv_deterministic) { XBT_INFO("******************************************************"); XBT_INFO("**** Only-recv-deterministic communication pattern ****"); XBT_INFO("******************************************************"); XBT_INFO("%s", this->send_diff); } XBT_INFO("Expanded states = %lu", expandedStatesCount_); XBT_INFO("Visited states = %lu", mc_model_checker->visited_states); XBT_INFO("Executed transitions = %lu", mc_model_checker->executed_transitions); XBT_INFO("Send-deterministic : %s", !this->send_deterministic ? "No" : "Yes"); if (_sg_mc_comms_determinism) XBT_INFO("Recv-deterministic : %s", !this->recv_deterministic ? "No" : "Yes"); } void CommunicationDeterminismChecker::prepare() { int i; const int maxpid = MC_smx_get_maxpid(); // Create initial_communications_pattern elements: initial_communications_pattern = simgrid::xbt::newDeleteDynar(); for (i=0; i < maxpid; i++){ simgrid::mc::PatternCommunicationList* process_list_pattern = new simgrid::mc::PatternCommunicationList(); xbt_dynar_insert_at(initial_communications_pattern, i, &process_list_pattern); } // Create incomplete_communications_pattern elements: incomplete_communications_pattern = xbt_dynar_new(sizeof(xbt_dynar_t), xbt_dynar_free_voidp); for (i=0; i < maxpid; i++){ xbt_dynar_t process_pattern = xbt_dynar_new(sizeof(simgrid::mc::PatternCommunication*), nullptr); xbt_dynar_insert_at(incomplete_communications_pattern, i, &process_pattern); } std::unique_ptr initial_state = std::unique_ptr(MC_state_new(++expandedStatesCount_)); XBT_DEBUG("********* Start communication determinism verification *********"); /* Get an enabled process and insert it in the interleave set of the initial state */ for (auto& p : mc_model_checker->process().simix_processes()) if (simgrid::mc::process_is_enabled(p.copy.getBuffer())) initial_state->interleave(p.copy.getBuffer()); stack_.push_back(std::move(initial_state)); } static inline bool all_communications_are_finished() { for (size_t current_process = 1; current_process < MC_smx_get_maxpid(); current_process++) { xbt_dynar_t pattern = xbt_dynar_get_as( incomplete_communications_pattern, current_process, xbt_dynar_t); if (!xbt_dynar_is_empty(pattern)) { XBT_DEBUG("Some communications are not finished, cannot stop the exploration ! State not visited."); return false; } } return true; } void CommunicationDeterminismChecker::restoreState() { /* Intermediate backtracking */ { simgrid::mc::State* state = stack_.back().get(); if (state->system_state) { simgrid::mc::restore_snapshot(state->system_state); MC_restore_communications_pattern(state); return; } } /* Restore the initial state */ simgrid::mc::session->restoreInitialState(); // int n = xbt_dynar_length(incomplete_communications_pattern); unsigned n = MC_smx_get_maxpid(); assert(n == xbt_dynar_length(incomplete_communications_pattern)); assert(n == xbt_dynar_length(initial_communications_pattern)); for (unsigned j=0; j < n ; j++) { xbt_dynar_reset((xbt_dynar_t)xbt_dynar_get_as(incomplete_communications_pattern, j, xbt_dynar_t)); xbt_dynar_get_as(initial_communications_pattern, j, simgrid::mc::PatternCommunicationList*)->index_comm = 0; } /* Traverse the stack from the state at position start and re-execute the transitions */ for (std::unique_ptr const& state : stack_) { if (state == stack_.back()) break; int req_num = state->transition.argument; smx_simcall_t saved_req = &state->executed_req; xbt_assert(saved_req); /* because we got a copy of the executed request, we have to fetch the real one, pointed by the request field of the issuer process */ const smx_actor_t issuer = MC_smx_simcall_get_issuer(saved_req); smx_simcall_t req = &issuer->simcall; /* TODO : handle test and testany simcalls */ e_mc_call_type_t call = MC_get_call_type(req); mc_model_checker->handle_simcall(state->transition); MC_handle_comm_pattern(call, req, req_num, nullptr, 1); mc_model_checker->wait_for_requests(); /* Update statistics */ mc_model_checker->visited_states++; mc_model_checker->executed_transitions++; } } int CommunicationDeterminismChecker::main(void) { std::unique_ptr visited_state = nullptr; smx_simcall_t req = nullptr; while (!stack_.empty()) { /* Get current state */ simgrid::mc::State* state = stack_.back().get(); XBT_DEBUG("**************************************************"); XBT_DEBUG("Exploration depth = %zi (state = %d, interleaved processes = %zd)", stack_.size(), state->num, state->interleaveSize()); /* Update statistics */ mc_model_checker->visited_states++; if (stack_.size() <= (std::size_t) _sg_mc_max_depth && (req = MC_state_get_request(state)) != nullptr && (visited_state == nullptr)) { int req_num = state->transition.argument; XBT_DEBUG("Execute: %s", simgrid::mc::request_to_string( req, req_num, simgrid::mc::RequestType::simix).c_str()); std::string req_str; if (dot_output != nullptr) req_str = simgrid::mc::request_get_dot_output(req, req_num); mc_model_checker->executed_transitions++; /* TODO : handle test and testany simcalls */ e_mc_call_type_t call = MC_CALL_TYPE_NONE; if (_sg_mc_comms_determinism || _sg_mc_send_determinism) call = MC_get_call_type(req); /* Answer the request */ mc_model_checker->handle_simcall(state->transition); /* After this call req is no longer useful */ if (!this->initial_communications_pattern_done) MC_handle_comm_pattern(call, req, req_num, initial_communications_pattern, 0); else MC_handle_comm_pattern(call, req, req_num, nullptr, 0); /* Wait for requests (schedules processes) */ mc_model_checker->wait_for_requests(); /* Create the new expanded state */ std::unique_ptr next_state = std::unique_ptr(MC_state_new(++expandedStatesCount_)); /* If comm determinism verification, we cannot stop the exploration if some communications are not finished (at least, data are transferred). These communications are incomplete and they cannot be analyzed and compared with the initial pattern. */ bool compare_snapshots = all_communications_are_finished() && this->initial_communications_pattern_done; if (_sg_mc_visited == 0 || (visited_state = visitedStates_.addVisitedState( expandedStatesCount_, next_state.get(), compare_snapshots)) == nullptr) { /* Get enabled processes and insert them in the interleave set of the next state */ for (auto& p : mc_model_checker->process().simix_processes()) if (simgrid::mc::process_is_enabled(p.copy.getBuffer())) next_state->interleave(p.copy.getBuffer()); if (dot_output != nullptr) fprintf(dot_output, "\"%d\" -> \"%d\" [%s];\n", state->num, next_state->num, req_str.c_str()); } else if (dot_output != nullptr) fprintf(dot_output, "\"%d\" -> \"%d\" [%s];\n", state->num, visited_state->other_num == -1 ? visited_state->num : visited_state->other_num, req_str.c_str()); stack_.push_back(std::move(next_state)); } else { if (stack_.size() > (std::size_t) _sg_mc_max_depth) XBT_WARN("/!\\ Max depth reached ! /!\\ "); else if (visited_state != nullptr) XBT_DEBUG("State already visited (equal to state %d), exploration stopped on this path.", visited_state->other_num == -1 ? visited_state->num : visited_state->other_num); else XBT_DEBUG("There are no more processes to interleave. (depth %zi)", stack_.size()); if (!this->initial_communications_pattern_done) this->initial_communications_pattern_done = 1; /* Trash the current state, no longer needed */ XBT_DEBUG("Delete state %d at depth %zi", state->num, stack_.size()); stack_.pop_back(); visited_state = nullptr; /* Check for deadlocks */ if (mc_model_checker->checkDeadlock()) { MC_show_deadlock(); return SIMGRID_MC_EXIT_DEADLOCK; } while (!stack_.empty()) { std::unique_ptr state = std::move(stack_.back()); stack_.pop_back(); if (state->interleaveSize() && stack_.size() < (std::size_t) _sg_mc_max_depth) { /* We found a back-tracking point, let's loop */ XBT_DEBUG("Back-tracking to state %d at depth %zi", state->num, stack_.size() + 1); stack_.push_back(std::move(state)); this->restoreState(); XBT_DEBUG("Back-tracking to state %d at depth %zi done", stack_.back()->num, stack_.size()); break; } else { XBT_DEBUG("Delete state %d at depth %zi", state->num, stack_.size() + 1); } } } } simgrid::mc::session->logState(); return SIMGRID_MC_EXIT_SUCCESS; } int CommunicationDeterminismChecker::run() { XBT_INFO("Check communication determinism"); simgrid::mc::session->initialize(); this->prepare(); this->initial_communications_pattern_done = 0; this->recv_deterministic = 1; this->send_deterministic = 1; this->recv_diff = nullptr; this->send_diff = nullptr; int res = this->main(); return res; } Checker* createCommunicationDeterminismChecker(Session& session) { return new CommunicationDeterminismChecker(session); } } } SimGrid-3.14.159/src/mc/Checker.hpp000644 001750 001750 00000004132 13031031270 020331 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_CHECKER_HPP #define SIMGRID_MC_CHECKER_HPP #include #include #include #include "src/mc/mc_forward.hpp" #include "src/mc/mc_record.h" #include "src/mc/Session.hpp" namespace simgrid { namespace mc { /** A model-checking algorithm * * This is an abstract base class used to group the data, state, configuration * of a model-checking algorithm. * * Implementing this interface will probably not be really mandatory, * you might be able to write your model-checking algorithm as plain * imperative code instead. * * It is expected to interact with the model-checking core through the * `Session` interface (but currently the `Session` interface does not * have all the necessary features). */ // abstract class Checker { Session* session_; public: Checker(Session& session); // No copy: Checker(Checker const&) = delete; Checker& operator=(Checker const&) = delete; virtual ~Checker(); /** Main function of this algorithm */ virtual int run() = 0; /* These methods are callbacks called by the model-checking engine * to get and display information about the current state of the * model-checking algorithm: */ /** Show the current trace/stack * * Could this be handled in the Session/ModelChecker instead? */ virtual RecordTrace getRecordTrace(); /** Generate a textual execution trace of the simulated application */ virtual std::vector getTextualTrace(); /** Log additional information about the state of the model-checker */ virtual void logState(); protected: Session& getSession() { return *session_; } }; XBT_PUBLIC() Checker* createLivenessChecker(Session& session); XBT_PUBLIC() Checker* createSafetyChecker(Session& session); XBT_PUBLIC() Checker* createCommunicationDeterminismChecker(Session& session); } } #endif SimGrid-3.14.159/src/mc/mc_exit.h000644 001750 001750 00000001207 13031031271 020056 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_EXIT_HPP #define SIMGRID_MC_EXIT_HPP #define SIMGRID_MC_EXIT_SUCCESS 0 #define SIMGRID_MC_EXIT_SAFETY 1 #define SIMGRID_MC_EXIT_LIVENESS 2 #define SIMGRID_MC_EXIT_DEADLOCK 3 #define SIMGRID_MC_EXIT_NON_TERMINATION 4 #define SIMGRID_MC_EXIT_NON_DETERMINISM 5 #define SIMGRID_MC_EXIT_PROGRAM_CRASH 6 #define SIMGRID_MC_EXIT_ERROR 63 #endif SimGrid-3.14.159/src/mc/mc_client_api.cpp000644 001750 001750 00000006063 13031031270 021553 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include "src/mc/mc_record.h" #include "src/mc/mc_private.h" #include "src/mc/mc_ignore.h" #include "src/mc/mc_protocol.h" #include "src/mc/Client.hpp" #include "src/mc/ModelChecker.hpp" /** @file mc_client_api.cpp * * This is the implementation of the API used by the user simulated program to * communicate with the MC (declared in modelchecker.h). */ XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_client_api, mc, "Public API for the model-checked application"); // MC_random() is in mc_base.cpp void MC_assert(int prop) { xbt_assert(mc_model_checker == nullptr); if (MC_is_active() && !prop) simgrid::mc::Client::get()->reportAssertionFailure(); } void MC_cut(void) { xbt_assert(mc_model_checker == nullptr); if (!MC_is_active()) return; // FIXME, We want to do this in the model-checker: xbt_die("MC_cut() not implemented"); } void MC_ignore(void* addr, size_t size) { xbt_assert(mc_model_checker == nullptr); if (!MC_is_active()) return; simgrid::mc::Client::get()->ignoreMemory(addr, size); } void MC_automaton_new_propositional_symbol(const char *id, int(*fct)(void)) { xbt_assert(mc_model_checker == nullptr); if (!MC_is_active()) return; xbt_die("Support for client-side function proposition is not implemented: " "use MC_automaton_new_propositional_symbol_pointer instead."); } void MC_automaton_new_propositional_symbol_pointer(const char *name, int* value) { xbt_assert(mc_model_checker == nullptr); if (!MC_is_active()) return; simgrid::mc::Client::get()->declareSymbol(name, value); } /** @brief Register a stack in the model checker * * The stacks are allocated in the heap. The MC handle them especially * when we analyze/compare the content of the heap so it must be told where * they are with this function. * * @param stack * @param process Process owning the stack * @param context * @param size Size of the stack */ void MC_register_stack_area(void *stack, smx_actor_t process, ucontext_t* context, size_t size) { xbt_assert(mc_model_checker == nullptr); if (!MC_is_active()) return; simgrid::mc::Client::get()->declareStack(stack, size, process, context); } void MC_ignore_global_variable(const char *name) { xbt_assert(mc_model_checker == nullptr); if (!MC_is_active()) return; // TODO, send a message to the model_checker xbt_die("Unimplemented"); } void MC_ignore_heap(void *address, size_t size) { xbt_assert(mc_model_checker == nullptr); if (!MC_is_active()) return; simgrid::mc::Client::get()->ignoreHeap(address, size); } void MC_remove_ignore_heap(void *address, size_t size) { xbt_assert(mc_model_checker == nullptr); if (!MC_is_active()) return; simgrid::mc::Client::get()->unignoreHeap(address, size); } SimGrid-3.14.159/src/mc/LivenessChecker.hpp000644 001750 001750 00000006020 13031031270 022040 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_LIVENESS_CHECKER_HPP #define SIMGRID_MC_LIVENESS_CHECKER_HPP #include #include #include #include #include #include #include #include #include #include "src/mc/mc_state.h" #include "src/mc/Checker.hpp" SG_BEGIN_DECL() SG_END_DECL() namespace simgrid { namespace mc { struct XBT_PRIVATE Pair { int num = 0; bool search_cycle = false; std::shared_ptr graph_state = nullptr; /* System state included */ xbt_automaton_state_t automaton_state = nullptr; std::shared_ptr> atomic_propositions; int requests = 0; int depth = 0; bool exploration_started = false; Pair(unsigned long expanded_pairs); ~Pair(); Pair(Pair const&) = delete; Pair& operator=(Pair const&) = delete; }; struct XBT_PRIVATE VisitedPair { int num = 0; int other_num = 0; /* Dot output for */ std::shared_ptr graph_state = nullptr; /* System state included */ xbt_automaton_state_t automaton_state = nullptr; std::shared_ptr> atomic_propositions; std::size_t heap_bytes_used = 0; int nb_processes = 0; VisitedPair( int pair_num, xbt_automaton_state_t automaton_state, std::shared_ptr> atomic_propositions, std::shared_ptr graph_state); ~VisitedPair(); }; class XBT_PRIVATE LivenessChecker : public Checker { public: LivenessChecker(Session& session); ~LivenessChecker(); int run() override; RecordTrace getRecordTrace() override; std::vector getTextualTrace() override; void logState() override; private: int main(); void prepare(); int compare(simgrid::mc::VisitedPair* state1, simgrid::mc::VisitedPair* state2); std::shared_ptr> getPropositionValues(); std::shared_ptr insertAcceptancePair(simgrid::mc::Pair* pair); int insertVisitedPair(std::shared_ptr visited_pair, simgrid::mc::Pair* pair); void showAcceptanceCycle(std::size_t depth); void replay(); void removeAcceptancePair(int pair_num); void purgeVisitedPairs(); void backtrack(); std::shared_ptr newPair(Pair* pair, xbt_automaton_state_t state, std::shared_ptr> propositions); private: // A stack of (application_state, automaton_state) pairs for DFS exploration: std::list> explorationStack_; std::list> acceptancePairs_; std::list> visitedPairs_; unsigned long visitedPairsCount_ = 0; unsigned long expandedPairsCount_ = 0; unsigned long expandedStatesCount_ = 0; int previousPair_ = 0; std::string previousRequest_; }; } } #endif SimGrid-3.14.159/src/mc/mc_base.cpp000644 001750 001750 00000016713 13031031270 020361 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include "src/mc/mc_base.h" #include "src/simix/smx_private.h" #include "src/mc/mc_replay.h" #include "mc/mc.h" #include "src/mc/mc_protocol.h" #include "src/kernel/activity/ActivityImpl.hpp" #include "src/kernel/activity/SynchroIo.hpp" #include "src/kernel/activity/SynchroComm.hpp" #include "src/kernel/activity/SynchroRaw.hpp" #include "src/kernel/activity/SynchroSleep.hpp" #include "src/kernel/activity/SynchroExec.hpp" #if HAVE_MC #include "src/mc/mc_request.h" #include "src/mc/Process.hpp" #include "src/mc/ModelChecker.hpp" #include "src/mc/mc_smx.h" #endif #if HAVE_MC using simgrid::mc::remote; #endif XBT_LOG_NEW_CATEGORY(mc, "All MC categories"); int MC_random(int min, int max) { #if HAVE_MC xbt_assert(mc_model_checker == nullptr); /* TODO, if the MC is disabled we do not really need to make a simcall for * this :) */ #endif return simcall_mc_random(min, max); } namespace simgrid { namespace mc { void wait_for_requests(void) { #if HAVE_MC xbt_assert(mc_model_checker == nullptr); #endif smx_actor_t process; smx_simcall_t req; unsigned int iter; while (!xbt_dynar_is_empty(simix_global->process_to_run)) { SIMIX_process_runall(); xbt_dynar_foreach(simix_global->process_that_ran, iter, process) { req = &process->simcall; if (req->call != SIMCALL_NONE && !simgrid::mc::request_is_visible(req)) SIMIX_simcall_handle(req, 0); } } } /** @brief returns if there this transition can proceed in a finite amount of time * * It is used in the model-checker to not get into self-deadlock where it would execute a never ending transition. * * Only WAIT operations (on comm, on mutex, etc) can ever return false because they could lock the MC exploration. * Wait operations are OK and return true in only two situations: * - if the wait will succeed immediately (if both peer of the comm are there already or if the mutex is available) * - if a timeout is provided, because we can fire the timeout if the transition is not ready without blocking in this transition for ever. * */ // Called from both MCer and MCed: bool request_is_enabled(smx_simcall_t req) { unsigned int index = 0; // TODO, add support for the subtypes? switch (req->call) { case SIMCALL_NONE: return false; case SIMCALL_SEM_ACQUIRE: xbt_die("Don't use semaphores in model-checked code, it's not supported yet"); case SIMCALL_COND_WAIT: xbt_die("Don't use condition variables in model-checked code, it's not supported yet"); case SIMCALL_COMM_WAIT: { /* FIXME: check also that src and dst processes are not suspended */ simgrid::kernel::activity::Comm *act = static_cast(simcall_comm_wait__get__comm(req)); #if HAVE_MC // Fetch from MCed memory: // HACK, type puning simgrid::mc::Remote temp_comm; if (mc_model_checker != nullptr) { mc_model_checker->process().read(temp_comm, remote(act)); act = static_cast(temp_comm.getBuffer()); } #endif if (simcall_comm_wait__get__timeout(req) >= 0) { /* If it has a timeout it will be always be enabled, because even if the * communication is not ready, it can timeout and won't block. */ if (_sg_mc_timeout == 1) return true; } /* On the other hand if it hasn't a timeout, check if the comm is ready.*/ else if (act->detached && act->src_proc == nullptr && act->type == SIMIX_COMM_READY) return (act->dst_proc != nullptr); return (act->src_proc && act->dst_proc); } case SIMCALL_COMM_WAITANY: { xbt_dynar_t comms; simgrid::kernel::activity::Comm *act = static_cast(simcall_comm_wait__get__comm(req)); #if HAVE_MC s_xbt_dynar_t comms_buffer; size_t buffer_size = 0; if (mc_model_checker != nullptr) { // Read dynar: mc_model_checker->process().read( &comms_buffer, remote(simcall_comm_waitany__get__comms(req))); assert(comms_buffer.elmsize == sizeof(act)); buffer_size = comms_buffer.elmsize * comms_buffer.used; comms = &comms_buffer; } else comms = simcall_comm_waitany__get__comms(req); // Read all the dynar buffer: char buffer[buffer_size]; if (mc_model_checker != nullptr) mc_model_checker->process().read_bytes(buffer, sizeof(buffer), remote(comms->data)); #else comms = simcall_comm_waitany__get__comms(req); #endif for (index = 0; index < comms->used; ++index) { #if HAVE_MC // Fetch act from MCed memory: // HACK, type puning simgrid::mc::Remote temp_comm; if (mc_model_checker != nullptr) { memcpy(&act, buffer + comms->elmsize * index, sizeof(act)); mc_model_checker->process().read(temp_comm, remote(act)); act = static_cast(temp_comm.getBuffer()); } else #endif act = xbt_dynar_get_as(comms, index, simgrid::kernel::activity::Comm*); if (act->src_proc && act->dst_proc) return true; } return false; } case SIMCALL_MUTEX_LOCK: { smx_mutex_t mutex = simcall_mutex_lock__get__mutex(req); #if HAVE_MC simgrid::mc::Remote temp_mutex; if (mc_model_checker != nullptr) { mc_model_checker->process().read(temp_mutex.getBuffer(), remote(mutex)); mutex = temp_mutex.getBuffer(); } #endif if(mutex->owner == nullptr) return true; #if HAVE_MC else if (mc_model_checker != nullptr) { simgrid::mc::Process& modelchecked = mc_model_checker->process(); // TODO, *(mutex->owner) :/ return modelchecked.resolveProcess(simgrid::mc::remote(mutex->owner))->pid == modelchecked.resolveProcess(simgrid::mc::remote(req->issuer))->pid; } #endif else return mutex->owner->pid == req->issuer->pid; } default: /* The rest of the requests are always enabled */ return true; } } bool request_is_visible(smx_simcall_t req) { return req->call == SIMCALL_COMM_ISEND || req->call == SIMCALL_COMM_IRECV || req->call == SIMCALL_COMM_WAIT || req->call == SIMCALL_COMM_WAITANY || req->call == SIMCALL_COMM_TEST || req->call == SIMCALL_COMM_TESTANY || req->call == SIMCALL_MC_RANDOM || req->call == SIMCALL_MUTEX_LOCK || req->call == SIMCALL_MUTEX_TRYLOCK ; } } } static int prng_random(int min, int max) { unsigned long output_size = ((unsigned long) max - (unsigned long) min) + 1; unsigned long input_size = (unsigned long) RAND_MAX + 1; unsigned long reject_size = input_size % output_size; unsigned long accept_size = input_size - reject_size; // module*accept_size // Use rejection in order to avoid skew unsigned long x; do { #ifndef _WIN32 x = (unsigned long) random(); #else x = (unsigned long) rand(); #endif } while( x >= accept_size ); return min + (x % output_size); } int simcall_HANDLER_mc_random(smx_simcall_t simcall, int min, int max) { if (!MC_is_active() && !MC_record_path) return prng_random(min, max); return simcall->mc_value; } SimGrid-3.14.159/src/mc/Channel.cpp000644 001750 001750 00000002223 13031031270 020327 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2015-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include "src/mc/Channel.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_Channel, mc, "MC interprocess communication"); namespace simgrid { namespace mc { Channel::~Channel() { if (this->socket_ >=0) close(this->socket_); } int Channel::send(const void* message, size_t size) const { XBT_DEBUG("Send %s", MC_message_type_name(*(e_mc_message_type*) message)); while (::send(this->socket_, message, size, 0) == -1) if (errno == EINTR) continue; else return errno; return 0; } ssize_t Channel::receive(void* message, size_t size, bool block) const { int res = recv(this->socket_, message, size, block ? 0 : MSG_DONTWAIT); if (res != -1) XBT_DEBUG("Receive %s", MC_message_type_name(*(e_mc_message_type*) message)); return res; } } } SimGrid-3.14.159/src/mc/mc_xbt.hpp000644 001750 001750 00000001325 13031031271 020243 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_XBT_HPP #define SIMGRID_MC_XBT_HPP #include #include #include "src/mc/RemotePtr.hpp" #include "src/mc/AddressSpace.hpp" namespace simgrid { namespace mc { XBT_PRIVATE void read_element(AddressSpace const& as, void* local, RemotePtr addr, std::size_t i, std::size_t len); XBT_PRIVATE std::size_t read_length( AddressSpace const& as, RemotePtr addr); } } #endif SimGrid-3.14.159/src/mc/mc_dwarf.cpp000644 001750 001750 00000130010 13031031270 020535 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include #define DW_LANG_Objc DW_LANG_ObjC /* fix spelling error in older dwarf.h */ #include #include #include #include #include "src/simgrid/util.hpp" #include #include #include "src/mc/mc_private.h" #include "src/mc/mc_dwarf.hpp" #include "src/mc/Process.hpp" #include "src/mc/ObjectInformation.hpp" #include "src/mc/Variable.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_dwarf, mc, "DWARF processing"); /** \brief The default DW_TAG_lower_bound for a given DW_AT_language. * * The default for a given language is defined in the DWARF spec. * * \param language constant as defined by the DWARf spec */ static uint64_t MC_dwarf_default_lower_bound(int lang); /** \brief Computes the the element_count of a DW_TAG_enumeration_type DIE * * This is the number of elements in a given array dimension. * * A reference of the compilation unit (DW_TAG_compile_unit) is * needed because the default lower bound (when there is no DW_AT_lower_bound) * depends of the language of the compilation unit (DW_AT_language). * * \param die DIE for the DW_TAG_enumeration_type or DW_TAG_subrange_type * \param unit DIE of the DW_TAG_compile_unit */ static uint64_t MC_dwarf_subrange_element_count(Dwarf_Die * die, Dwarf_Die * unit); /** \brief Computes the number of elements of a given DW_TAG_array_type. * * \param die DIE for the DW_TAG_array_type */ static uint64_t MC_dwarf_array_element_count(Dwarf_Die * die, Dwarf_Die * unit); /** \brief Process a DIE * * \param info the resulting object fot the library/binary file (output) * \param die the current DIE * \param unit the DIE of the compile unit of the current DIE * \param frame containing frame if any */ static void MC_dwarf_handle_die(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns); /** \brief Process a type DIE */ static void MC_dwarf_handle_type_die(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns); /** \brief Calls MC_dwarf_handle_die on all children of the given die * * \param info the resulting object fot the library/binary file (output) * \param die the current DIE * \param unit the DIE of the compile unit of the current DIE * \param frame containing frame if any */ static void MC_dwarf_handle_children(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns); /** \brief Handle a variable (DW_TAG_variable or other) * * \param info the resulting object fot the library/binary file (output) * \param die the current DIE * \param unit the DIE of the compile unit of the current DIE * \param frame containing frame if any */ static void MC_dwarf_handle_variable_die(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns); /** \brief Get the DW_TAG_type of the DIE * * \param die DIE * \return DW_TAG_type attribute as a new string (nullptr if none) */ static std::uint64_t MC_dwarf_at_type(Dwarf_Die * die); namespace simgrid { namespace dwarf { enum class TagClass { Unknown, Type, Subprogram, Variable, Scope, Namespace }; /*** Class of forms defined in the DWARF standard */ enum class FormClass { Unknown, Address, // Location in the program's address space Block, // Arbitrary block of bytes Constant, String, Flag, // Boolean value Reference, // Reference to another DIE ExprLoc, // DWARF expression/location description LinePtr, LocListPtr, MacPtr, RangeListPtr }; static TagClass classify_tag(int tag) { switch (tag) { case DW_TAG_array_type: case DW_TAG_class_type: case DW_TAG_enumeration_type: case DW_TAG_typedef: case DW_TAG_pointer_type: case DW_TAG_reference_type: case DW_TAG_rvalue_reference_type: case DW_TAG_string_type: case DW_TAG_structure_type: case DW_TAG_subroutine_type: case DW_TAG_union_type: case DW_TAG_ptr_to_member_type: case DW_TAG_set_type: case DW_TAG_subrange_type: case DW_TAG_base_type: case DW_TAG_const_type: case DW_TAG_file_type: case DW_TAG_packed_type: case DW_TAG_volatile_type: case DW_TAG_restrict_type: case DW_TAG_interface_type: case DW_TAG_unspecified_type: case DW_TAG_shared_type: return TagClass::Type; case DW_TAG_subprogram: return TagClass::Subprogram; case DW_TAG_variable: case DW_TAG_formal_parameter: return TagClass::Variable; case DW_TAG_lexical_block: case DW_TAG_try_block: case DW_TAG_catch_block: case DW_TAG_inlined_subroutine: case DW_TAG_with_stmt: return TagClass::Scope; case DW_TAG_namespace: return TagClass::Namespace; default: return TagClass::Unknown; } } /** \brief Find the DWARF data class for a given DWARF data form * * This mapping is defined in the DWARF spec. * * \param form The form (values taken from the DWARF spec) * \return An internal representation for the corresponding class * */ static FormClass classify_form(int form) { switch (form) { case DW_FORM_addr: return FormClass::Address; case DW_FORM_block2: case DW_FORM_block4: case DW_FORM_block: case DW_FORM_block1: return FormClass::Block; case DW_FORM_data1: case DW_FORM_data2: case DW_FORM_data4: case DW_FORM_data8: case DW_FORM_udata: case DW_FORM_sdata: return FormClass::Constant; case DW_FORM_string: case DW_FORM_strp: return FormClass::String; case DW_FORM_ref_addr: case DW_FORM_ref1: case DW_FORM_ref2: case DW_FORM_ref4: case DW_FORM_ref8: case DW_FORM_ref_udata: return FormClass::Reference; case DW_FORM_flag: case DW_FORM_flag_present: return FormClass::Flag; case DW_FORM_exprloc: return FormClass::ExprLoc; // TODO sec offset // TODO indirect default: return FormClass::Unknown; } } /** \brief Get the name of the tag of a given DIE * * \param die DIE * \return name of the tag of this DIE */ inline XBT_PRIVATE const char *tagname(Dwarf_Die * die) { return simgrid::dwarf::tagname(dwarf_tag(die)); } } } // ***** Attributes /** \brief Get an attribute of a given DIE as a string * * \param die the DIE * \param attribute attribute * \return value of the given attribute of the given DIE */ static const char *MC_dwarf_attr_integrate_string(Dwarf_Die * die, int attribute) { Dwarf_Attribute attr; if (!dwarf_attr_integrate(die, attribute, &attr)) return nullptr; else return dwarf_formstring(&attr); } /** \brief Get the linkage name of a DIE. * * Use either DW_AT_linkage_name or DW_AT_MIPS_linkage_name. * DW_AT_linkage_name is standardized since DWARF 4. * Before this version of DWARF, the MIPS extensions * DW_AT_MIPS_linkage_name is used (at least by GCC). * * \param the DIE * \return linkage name of the given DIE (or nullptr) * */ static const char *MC_dwarf_at_linkage_name(Dwarf_Die * die) { const char *name = MC_dwarf_attr_integrate_string(die, DW_AT_linkage_name); if (!name) name = MC_dwarf_attr_integrate_string(die, DW_AT_MIPS_linkage_name); return name; } static Dwarf_Off MC_dwarf_attr_dieoffset(Dwarf_Die * die, int attribute) { Dwarf_Attribute attr; if (dwarf_hasattr_integrate(die, attribute) == 0) return 0; dwarf_attr_integrate(die, attribute, &attr); Dwarf_Die subtype_die; if (dwarf_formref_die(&attr, &subtype_die) == nullptr) xbt_die("Could not find DIE"); return dwarf_dieoffset(&subtype_die); } static Dwarf_Off MC_dwarf_attr_integrate_dieoffset(Dwarf_Die * die, int attribute) { Dwarf_Attribute attr; if (dwarf_hasattr_integrate(die, attribute) == 0) return 0; dwarf_attr_integrate(die, DW_AT_type, &attr); Dwarf_Die subtype_die; if (dwarf_formref_die(&attr, &subtype_die) == nullptr) xbt_die("Could not find DIE"); return dwarf_dieoffset(&subtype_die); } /** \brief Find the type/subtype (DW_AT_type) for a DIE * * \param dit the DIE * \return DW_AT_type reference as a global offset in hexadecimal (or nullptr) */ static std::uint64_t MC_dwarf_at_type(Dwarf_Die * die) { return MC_dwarf_attr_integrate_dieoffset(die, DW_AT_type); } static uint64_t MC_dwarf_attr_integrate_addr(Dwarf_Die * die, int attribute) { Dwarf_Attribute attr; if (dwarf_attr_integrate(die, attribute, &attr) == nullptr) return 0; Dwarf_Addr value; if (dwarf_formaddr(&attr, &value) == 0) return (uint64_t) value; else return 0; } static uint64_t MC_dwarf_attr_integrate_uint(Dwarf_Die * die, int attribute, uint64_t default_value) { Dwarf_Attribute attr; if (dwarf_attr_integrate(die, attribute, &attr) == nullptr) return default_value; Dwarf_Word value; return dwarf_formudata(dwarf_attr_integrate(die, attribute, &attr), &value) == 0 ? (uint64_t) value : default_value; } static bool MC_dwarf_attr_flag(Dwarf_Die * die, int attribute, bool integrate) { Dwarf_Attribute attr; if ((integrate ? dwarf_attr_integrate(die, attribute, &attr) : dwarf_attr(die, attribute, &attr)) == 0) return false; bool result; if (dwarf_formflag(&attr, &result)) xbt_die("Unexpected form for attribute %s", simgrid::dwarf::attrname(attribute)); return result; } /** @brief Find the default lower bound for a given language * * The default lower bound of an array (when DW_TAG_lower_bound * is missing) depends on the language of the compilation unit. * * @param lang Language of the compilation unit (values defined in the DWARF spec) * @return Default lower bound of an array in this compilation unit * */ static uint64_t MC_dwarf_default_lower_bound(int lang) { switch (lang) { case DW_LANG_C: case DW_LANG_C89: case DW_LANG_C99: case DW_LANG_C_plus_plus: case DW_LANG_D: case DW_LANG_Java: case DW_LANG_ObjC: case DW_LANG_ObjC_plus_plus: case DW_LANG_Python: case DW_LANG_UPC: return 0; case DW_LANG_Ada83: case DW_LANG_Ada95: case DW_LANG_Fortran77: case DW_LANG_Fortran90: case DW_LANG_Fortran95: case DW_LANG_Modula2: case DW_LANG_Pascal83: case DW_LANG_PL1: case DW_LANG_Cobol74: case DW_LANG_Cobol85: return 1; default: xbt_die("No default DW_TAG_lower_bound for language %i and none given", lang); return 0; } } /** \brief Finds the number of elements in a DW_TAG_subrange_type or DW_TAG_enumeration_type DIE * * \param die the DIE * \param unit DIE of the compilation unit * \return number of elements in the range * */ static uint64_t MC_dwarf_subrange_element_count(Dwarf_Die * die, Dwarf_Die * unit) { xbt_assert(dwarf_tag(die) == DW_TAG_enumeration_type || dwarf_tag(die) == DW_TAG_subrange_type, "MC_dwarf_subrange_element_count called with DIE of type %s", simgrid::dwarf::tagname(die)); // Use DW_TAG_count if present: if (dwarf_hasattr_integrate(die, DW_AT_count)) return MC_dwarf_attr_integrate_uint(die, DW_AT_count, 0); // Otherwise compute DW_TAG_upper_bound-DW_TAG_lower_bound + 1: if (!dwarf_hasattr_integrate(die, DW_AT_upper_bound)) // This is not really 0, but the code expects this (we do not know): return 0; uint64_t upper_bound = MC_dwarf_attr_integrate_uint(die, DW_AT_upper_bound, -1); uint64_t lower_bound = 0; if (dwarf_hasattr_integrate(die, DW_AT_lower_bound)) lower_bound = MC_dwarf_attr_integrate_uint(die, DW_AT_lower_bound, -1); else lower_bound = MC_dwarf_default_lower_bound(dwarf_srclang(unit)); return upper_bound - lower_bound + 1; } /** \brief Finds the number of elements in a array type (DW_TAG_array_type) * * The compilation unit might be needed because the default lower * bound depends on the language of the compilation unit. * * \param die the DIE of the DW_TAG_array_type * \param unit the DIE of the compilation unit * \return number of elements in this array type * */ static uint64_t MC_dwarf_array_element_count(Dwarf_Die * die, Dwarf_Die * unit) { xbt_assert(dwarf_tag(die) == DW_TAG_array_type, "MC_dwarf_array_element_count called with DIE of type %s", simgrid::dwarf::tagname(die)); int result = 1; Dwarf_Die child; int res; for (res = dwarf_child(die, &child); res == 0; res = dwarf_siblingof(&child, &child)) { int child_tag = dwarf_tag(&child); if (child_tag == DW_TAG_subrange_type || child_tag == DW_TAG_enumeration_type) result *= MC_dwarf_subrange_element_count(&child, unit); } return result; } // ***** Variable /** Sort the variable by name and address. * * We could use boost::container::flat_set instead. */ static bool MC_compare_variable( simgrid::mc::Variable const& a, simgrid::mc::Variable const& b) { int cmp = strcmp(a.name.c_str(), b.name.c_str()); if (cmp < 0) return true; else if (cmp > 0) return false; else return a.address < b.address; } // ***** simgrid::mc::Type* /** \brief Initialize the location of a member of a type * (DW_AT_data_member_location of a DW_TAG_member). * * \param type a type (struct, class) * \param member the member of the type * \param child DIE of the member (DW_TAG_member) */ static void MC_dwarf_fill_member_location( simgrid::mc::Type* type, simgrid::mc::Member* member, Dwarf_Die * child) { if (dwarf_hasattr(child, DW_AT_data_bit_offset)) xbt_die("Can't groke DW_AT_data_bit_offset."); if (!dwarf_hasattr_integrate(child, DW_AT_data_member_location)) { if (type->type == DW_TAG_union_type) return; xbt_die ("Missing DW_AT_data_member_location field in DW_TAG_member %s of type <%" PRIx64 ">%s", member->name.c_str(), (uint64_t) type->id, type->name.c_str()); } Dwarf_Attribute attr; dwarf_attr_integrate(child, DW_AT_data_member_location, &attr); int form = dwarf_whatform(&attr); simgrid::dwarf::FormClass form_class = simgrid::dwarf::classify_form(form); switch (form_class) { case simgrid::dwarf::FormClass::ExprLoc: case simgrid::dwarf::FormClass::Block: // Location expression: { Dwarf_Op *expr; size_t len; if (dwarf_getlocation(&attr, &expr, &len)) xbt_die ("Could not read location expression DW_AT_data_member_location in DW_TAG_member %s of type <%" PRIx64 ">%s", MC_dwarf_attr_integrate_string(child, DW_AT_name), (uint64_t) type->id, type->name.c_str()); member->location_expression = simgrid::dwarf::DwarfExpression(expr, expr+len); break; } case simgrid::dwarf::FormClass::Constant: // Offset from the base address of the object: { Dwarf_Word offset; if (!dwarf_formudata(&attr, &offset)) member->offset(offset); else xbt_die("Cannot get %s location <%" PRIx64 ">%s", MC_dwarf_attr_integrate_string(child, DW_AT_name), (uint64_t) type->id, type->name.c_str()); break; } case simgrid::dwarf::FormClass::LocListPtr: // Reference to a location list: // TODO case simgrid::dwarf::FormClass::Reference: // It's supposed to be possible in DWARF2 but I couldn't find its semantic // in the spec. default: xbt_die("Can't handle form class (%i) / form 0x%x as DW_AT_member_location", (int) form_class, form); } } /** \brief Populate the list of members of a type * * \param info ELF object containing the type DIE * \param die DIE of the type * \param unit DIE of the compilation unit containing the type DIE * \param type the type */ static void MC_dwarf_add_members(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Type* type) { int res; Dwarf_Die child; xbt_assert(type->members.empty()); for (res = dwarf_child(die, &child); res == 0; res = dwarf_siblingof(&child, &child)) { int tag = dwarf_tag(&child); if (tag == DW_TAG_member || tag == DW_TAG_inheritance) { // Skip declarations: if (MC_dwarf_attr_flag(&child, DW_AT_declaration, false)) continue; // Skip compile time constants: if (dwarf_hasattr(&child, DW_AT_const_value)) continue; // TODO, we should use another type (because is is not a type but a member) simgrid::mc::Member member; if (tag == DW_TAG_inheritance) member.flags |= simgrid::mc::Member::INHERITANCE_FLAG; const char *name = MC_dwarf_attr_integrate_string(&child, DW_AT_name); if (name) member.name = name; // Those base names are used by GCC and clang for virtual table pointers // respectively ("__vptr$ClassName", "__vptr.ClassName"): if (boost::algorithm::starts_with(member.name, "__vptr$") || boost::algorithm::starts_with(member.name, "__vptr.")) member.flags |= simgrid::mc::Member::VIRTUAL_POINTER_FLAG; // A cleaner solution would be to check against the type: // --- // tag: DW_TAG_member // name: "_vptr$Foo" // type: // # Type for a pointer to a vtable // tag: DW_TAG_pointer_type // type: // # Type for a vtable: // tag: DW_TAG_pointer_type // name: "__vtbl_ptr_type" // type: // tag: DW_TAG_subroutine_type // type: // tag: DW_TAG_base_type // name: "int" // --- member.byte_size = MC_dwarf_attr_integrate_uint(&child, DW_AT_byte_size, 0); member.type_id = MC_dwarf_at_type(&child); if (dwarf_hasattr(&child, DW_AT_data_bit_offset)) xbt_die("Can't groke DW_AT_data_bit_offset."); MC_dwarf_fill_member_location(type, &member, &child); if (!member.type_id) xbt_die("Missing type for member %s of <%" PRIx64 ">%s", member.name.c_str(), (uint64_t) type->id, type->name.c_str()); type->members.push_back(std::move(member)); } } } /** \brief Create a MC type object from a DIE * * \param info current object info object * \param DIE (for a given type); * \param unit compilation unit of the current DIE * \return MC representation of the type */ static simgrid::mc::Type MC_dwarf_die_to_type( simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns) { simgrid::mc::Type type; type.type = dwarf_tag(die); type.name = std::string(); type.element_count = -1; // Global Offset type.id = dwarf_dieoffset(die); const char *prefix = ""; switch (type.type) { case DW_TAG_structure_type: prefix = "struct "; break; case DW_TAG_union_type: prefix = "union "; break; case DW_TAG_class_type: prefix = "class "; break; default: prefix = ""; } const char *name = MC_dwarf_attr_integrate_string(die, DW_AT_name); if (name != nullptr) { char* full_name = ns ? bprintf("%s%s::%s", prefix, ns, name) : bprintf("%s%s", prefix, name); type.name = std::string(full_name); free(full_name); } type.type_id = MC_dwarf_at_type(die); // Some compilers do not emit DW_AT_byte_size for pointer_type, // so we fill this. We currently assume that the model-checked process is in // the same architecture.. if (type.type == DW_TAG_pointer_type) type.byte_size = sizeof(void*); // Computation of the byte_size; if (dwarf_hasattr_integrate(die, DW_AT_byte_size)) type.byte_size = MC_dwarf_attr_integrate_uint(die, DW_AT_byte_size, 0); else if (type.type == DW_TAG_array_type || type.type == DW_TAG_structure_type || type.type == DW_TAG_class_type) { Dwarf_Word size; if (dwarf_aggregate_size(die, &size) == 0) type.byte_size = size; } switch (type.type) { case DW_TAG_array_type: type.element_count = MC_dwarf_array_element_count(die, unit); // TODO, handle DW_byte_stride and (not) DW_bit_stride break; case DW_TAG_pointer_type: case DW_TAG_reference_type: case DW_TAG_rvalue_reference_type: break; case DW_TAG_structure_type: case DW_TAG_union_type: case DW_TAG_class_type: MC_dwarf_add_members(info, die, unit, &type); char *new_ns = ns == nullptr ? xbt_strdup(type.name.c_str()) : bprintf("%s::%s", ns, name); MC_dwarf_handle_children(info, die, unit, frame, new_ns); free(new_ns); break; } return type; } static void MC_dwarf_handle_type_die(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns) { simgrid::mc::Type type = MC_dwarf_die_to_type(info, die, unit, frame, ns); auto& t = (info->types[type.id] = std::move(type)); if (!t.name.empty() && type.byte_size != 0) info->full_types_by_name[t.name] = &t; } static int mc_anonymous_variable_index = 0; static std::unique_ptr MC_die_to_variable( simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns) { // Skip declarations: if (MC_dwarf_attr_flag(die, DW_AT_declaration, false)) return nullptr; // Skip compile time constants: if (dwarf_hasattr(die, DW_AT_const_value)) return nullptr; Dwarf_Attribute attr_location; if (dwarf_attr(die, DW_AT_location, &attr_location) == nullptr) // No location: do not add it ? return nullptr; std::unique_ptr variable = std::unique_ptr(new simgrid::mc::Variable()); variable->id = dwarf_dieoffset(die); variable->global = frame == nullptr; // Can be override base on DW_AT_location variable->object_info = info; const char *name = MC_dwarf_attr_integrate_string(die, DW_AT_name); if (name) variable->name = name; variable->type_id = MC_dwarf_at_type(die); int form = dwarf_whatform(&attr_location); simgrid::dwarf::FormClass form_class; if (form == DW_FORM_sec_offset) form_class = simgrid::dwarf::FormClass::Constant; else form_class = simgrid::dwarf::classify_form(form); switch (form_class) { case simgrid::dwarf::FormClass::ExprLoc: case simgrid::dwarf::FormClass::Block: // Location expression: { Dwarf_Op *expr; size_t len; if (dwarf_getlocation(&attr_location, &expr, &len)) { xbt_die( "Could not read location expression in DW_AT_location " "of variable <%" PRIx64 ">%s", (uint64_t) variable->id, variable->name.c_str()); } if (len == 1 && expr[0].atom == DW_OP_addr) { variable->global = true; uintptr_t offset = (uintptr_t) expr[0].number; uintptr_t base = (uintptr_t) info->base_address(); variable->address = (void *) (base + offset); } else variable->location_list = { simgrid::dwarf::DwarfExpression(expr, expr + len) }; break; } case simgrid::dwarf::FormClass::LocListPtr: case simgrid::dwarf::FormClass::Constant: // Reference to location list: variable->location_list = simgrid::dwarf::location_list( *info, attr_location); break; default: xbt_die("Unexpected form 0x%x (%i), class 0x%x (%i) list for location " "in <%" PRIx64 ">%s", form, form, (int) form_class, (int) form_class, (uint64_t) variable->id, variable->name.c_str()); } // Handle start_scope: if (dwarf_hasattr(die, DW_AT_start_scope)) { Dwarf_Attribute attr; dwarf_attr(die, DW_AT_start_scope, &attr); int form = dwarf_whatform(&attr); simgrid::dwarf::FormClass form_class = simgrid::dwarf::classify_form(form); switch (form_class) { case simgrid::dwarf::FormClass::Constant: { Dwarf_Word value; variable->start_scope = dwarf_formudata(&attr, &value) == 0 ? (size_t) value : 0; break; } case simgrid::dwarf::FormClass::RangeListPtr: // TODO default: xbt_die ("Unhandled form 0x%x, class 0x%X for DW_AT_start_scope of variable %s", form, (int) form_class, name == nullptr ? "?" : name); } } if (ns && variable->global) variable->name = std::string(ns) + "::" + variable->name; // The current code needs a variable name, // generate a fake one: if (variable->name.empty()) variable->name = "@anonymous#" + std::to_string(mc_anonymous_variable_index++); return variable; } static void MC_dwarf_handle_variable_die(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns) { std::unique_ptr variable = MC_die_to_variable(info, die, unit, frame, ns); if (!variable) return; // Those arrays are sorted later: else if (variable->global) info->global_variables.push_back(std::move(*variable)); else if (frame != nullptr) frame->variables.push_back(std::move(*variable)); else xbt_die("No frame for this local variable"); } static void MC_dwarf_handle_scope_die(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* parent_frame, const char *ns) { // TODO, handle DW_TAG_type/DW_TAG_location for DW_TAG_with_stmt int tag = dwarf_tag(die); simgrid::dwarf::TagClass klass = simgrid::dwarf::classify_tag(tag); // (Template) Subprogram declaration: if (klass == simgrid::dwarf::TagClass::Subprogram && MC_dwarf_attr_flag(die, DW_AT_declaration, false)) return; if (klass == simgrid::dwarf::TagClass::Scope) xbt_assert(parent_frame, "No parent scope for this scope"); simgrid::mc::Frame frame; frame.tag = tag; frame.id = dwarf_dieoffset(die); frame.object_info = info; if (klass == simgrid::dwarf::TagClass::Subprogram) { const char *name = MC_dwarf_attr_integrate_string(die, DW_AT_name); if (name && ns) frame.name = std::string(ns) + "::" + name; else if (name) frame.name = name; } frame.abstract_origin_id = MC_dwarf_attr_dieoffset(die, DW_AT_abstract_origin); // This is the base address for DWARF addresses. // Relocated addresses are offset from this base address. // See DWARF4 spec 7.5 std::uint64_t base = (std::uint64_t) info->base_address(); // TODO, support DW_AT_ranges uint64_t low_pc = MC_dwarf_attr_integrate_addr(die, DW_AT_low_pc); frame.range.begin() = low_pc ? (std::uint64_t) base + low_pc : 0; if (low_pc) { // DW_AT_high_pc: Dwarf_Attribute attr; if (!dwarf_attr_integrate(die, DW_AT_high_pc, &attr)) xbt_die("Missing DW_AT_high_pc matching with DW_AT_low_pc"); Dwarf_Sword offset; Dwarf_Addr high_pc; switch (simgrid::dwarf::classify_form(dwarf_whatform(&attr))) { // DW_AT_high_pc if an offset from the low_pc: case simgrid::dwarf::FormClass::Constant: if (dwarf_formsdata(&attr, &offset) != 0) xbt_die("Could not read constant"); frame.range.end() = frame.range.begin() + offset; break; // DW_AT_high_pc is a relocatable address: case simgrid::dwarf::FormClass::Address: if (dwarf_formaddr(&attr, &high_pc) != 0) xbt_die("Could not read address"); frame.range.end() = base + high_pc; break; default: xbt_die("Unexpected class for DW_AT_high_pc"); } } if (klass == simgrid::dwarf::TagClass::Subprogram) { Dwarf_Attribute attr_frame_base; if (dwarf_attr_integrate(die, DW_AT_frame_base, &attr_frame_base)) frame.frame_base_location = simgrid::dwarf::location_list(*info, attr_frame_base); } // Handle children: MC_dwarf_handle_children(info, die, unit, &frame, ns); // We sort them in order to have an (somewhat) efficient by name // lookup: boost::range::sort(frame.variables, MC_compare_variable); // Register it: if (klass == simgrid::dwarf::TagClass::Subprogram) info->subprograms[frame.id] = std::move(frame); else if (klass == simgrid::dwarf::TagClass::Scope) parent_frame->scopes.push_back(std::move(frame)); } static void mc_dwarf_handle_namespace_die(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns) { const char *name = MC_dwarf_attr_integrate_string(die, DW_AT_name); if (frame) xbt_die("Unexpected namespace in a subprogram"); char *new_ns = ns == nullptr ? xbt_strdup(name) : bprintf("%s::%s", ns, name); MC_dwarf_handle_children(info, die, unit, frame, new_ns); xbt_free(new_ns); } static void MC_dwarf_handle_children(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns) { // For each child DIE: Dwarf_Die child; int res; for (res = dwarf_child(die, &child); res == 0; res = dwarf_siblingof(&child, &child)) MC_dwarf_handle_die(info, &child, unit, frame, ns); } static void MC_dwarf_handle_die(simgrid::mc::ObjectInformation* info, Dwarf_Die * die, Dwarf_Die * unit, simgrid::mc::Frame* frame, const char *ns) { int tag = dwarf_tag(die); simgrid::dwarf::TagClass klass = simgrid::dwarf::classify_tag(tag); switch (klass) { // Type: case simgrid::dwarf::TagClass::Type: MC_dwarf_handle_type_die(info, die, unit, frame, ns); break; // Subprogram or scope: case simgrid::dwarf::TagClass::Subprogram: case simgrid::dwarf::TagClass::Scope: MC_dwarf_handle_scope_die(info, die, unit, frame, ns); return; // Variable: case simgrid::dwarf::TagClass::Variable: MC_dwarf_handle_variable_die(info, die, unit, frame, ns); break; case simgrid::dwarf::TagClass::Namespace: mc_dwarf_handle_namespace_die(info, die, unit, frame, ns); break; default: break; } } static Elf64_Half get_type(Elf* elf) { Elf64_Ehdr* ehdr64 = elf64_getehdr(elf); if (ehdr64) return ehdr64->e_type; Elf32_Ehdr* ehdr32 = elf32_getehdr(elf); if (ehdr32) return ehdr32->e_type; xbt_die("Could not get ELF heeader"); } static void read_dwarf_info(simgrid::mc::ObjectInformation* info, Dwarf* dwarf) { // For each compilation unit: Dwarf_Off offset = 0; Dwarf_Off next_offset = 0; size_t length; while (dwarf_nextcu(dwarf, offset, &next_offset, &length, nullptr, nullptr, nullptr) == 0) { Dwarf_Die unit_die; if (dwarf_offdie(dwarf, offset + length, &unit_die) != nullptr) MC_dwarf_handle_children(info, &unit_die, &unit_die, nullptr, nullptr); offset = next_offset; } } /** Get the build-id (NT_GNU_BUILD_ID) from the ELF file * * This build-id may is used to locate an external debug (DWARF) file * for this ELF file. * * @param elf libelf handle for an ELF file * @return build-id for this ELF file (or an empty vector if none is found) */ static std::vector get_build_id(Elf* elf) { #ifdef __linux // Summary: the GNU build ID is stored in a ("GNU, NT_GNU_BUILD_ID) note // found in a PT_NOTE entry in the program header table. size_t phnum; if (elf_getphdrnum (elf, &phnum) != 0) xbt_die("Could not read program headers"); // Iterate over the program headers and find the PT_NOTE ones: for (size_t i = 0; i < phnum; ++i) { GElf_Phdr phdr_temp; GElf_Phdr *phdr = gelf_getphdr(elf, i, &phdr_temp); if (phdr->p_type != PT_NOTE) continue; Elf_Data* data = elf_getdata_rawchunk(elf, phdr->p_offset, phdr->p_filesz, ELF_T_NHDR); // Iterate over the notes and find the NT_GNU_BUILD_ID one: size_t pos = 0; while (pos < data->d_size) { GElf_Nhdr nhdr; // Location of the name within Elf_Data: size_t name_pos; size_t desc_pos; pos = gelf_getnote(data, pos, &nhdr, &name_pos, &desc_pos); // A build ID note is identified by the pair ("GNU", NT_GNU_BUILD_ID) // (a namespace and a type within this namespace): if (nhdr.n_type == NT_GNU_BUILD_ID && nhdr.n_namesz == sizeof("GNU") && memcmp((char*) data->d_buf + name_pos, "GNU", sizeof("GNU")) == 0) { XBT_DEBUG("Found GNU/NT_GNU_BUILD_ID note"); char* start = (char*) data->d_buf + desc_pos; char* end = (char*) start + nhdr.n_descsz; return std::vector(start, end); } } } #endif return std::vector(); } static char hexdigits[16] = { '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f' }; /** Binary data to hexadecimal */ static inline std::array to_hex(std::uint8_t byte) { // Horrid double braces! // Apparently, this is needed in C++11 (not in C++14). return { { hexdigits[byte >> 4], hexdigits[byte & 0xF] } }; } /** Binary data to hexadecimal */ static std::string to_hex(const char* data, std::size_t count) { std::string res; res.resize(2*count); for (std::size_t i = 0; i < count; i++) { std::array hex_byte = to_hex(data[i]); for (int j = 0; j < 2; ++j) res[2 * i + j] = hex_byte[j]; } return res; } /** Binary data to hexadecimal */ static std::string to_hex(std::vector const& data) { return to_hex(data.data(), data.size()); } /** Base directories for external debug files */ static const char* debug_paths[] = { "/usr/lib/debug/", "/usr/local/lib/debug/", }; /** Locate an external debug file from the NT_GNU_BUILD_ID * * This is one of the mechanisms used for * [separate debug files](https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html). */ // Example: // /usr/lib/debug/.build-id/0b/dc77f1c29aea2b14ff5acd9a19ab3175ffdeae.debug static std::string find_by_build_id(std::vector id) { std::string filename; std::string hex = to_hex(id); for (const char* debug_path : debug_paths) { // Example: filename = std::string(debug_path) + ".build-id/" + to_hex(id.data(), 1) + '/' + to_hex(id.data() + 1, id.size() - 1) + ".debug"; XBT_DEBUG("Checking debug file: %s", filename.c_str()); if (access(filename.c_str(), F_OK) == 0) { XBT_DEBUG("Found debug file: %s\n", hex.c_str()); return filename; } } XBT_DEBUG("Not debuf info found for build ID %s\n", hex.data()); return std::string(); } /** \brief Populate the debugging informations of the given ELF object * * Read the DWARf information of the EFFL object and populate the * lists of types, variables, functions. */ static void MC_load_dwarf(simgrid::mc::ObjectInformation* info) { if (elf_version(EV_CURRENT) == EV_NONE) xbt_die("libelf initialization error"); // Open the ELF file: int fd = open(info->file_name.c_str(), O_RDONLY); if (fd < 0) xbt_die("Could not open file %s", info->file_name.c_str()); Elf* elf = elf_begin(fd, ELF_C_READ, nullptr); if (elf == nullptr) xbt_die("Not an ELF file"); Elf_Kind kind = elf_kind(elf); if (kind != ELF_K_ELF) xbt_die("Not an ELF file"); // Remember if this is a `ET_EXEC` (fixed location) or `ET_DYN`: Elf64_Half type = get_type(elf); if (type == ET_EXEC) info->flags |= simgrid::mc::ObjectInformation::Executable; // Read DWARF debug information in the file: Dwarf* dwarf = dwarf_begin_elf (elf, DWARF_C_READ, nullptr); if (dwarf != nullptr) { read_dwarf_info(info, dwarf); dwarf_end(dwarf); elf_end(elf); close(fd); return; } dwarf_end(dwarf); // If there was no DWARF in the file, try to find it in a separate file. // Different methods might be used to store the DWARF informations: // * GNU NT_GNU_BUILD_ID; // * .gnu_debuglink. // See https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html // for reference of what we are doing. // Try with NT_GNU_BUILD_ID: we find the build ID in the ELF file and then // use this ID to find the file in some known locations in the filesystem. std::vector build_id = get_build_id(elf); if (!build_id.empty()) { elf_end(elf); close(fd); // Find the debug file using the build id: std::string debug_file = find_by_build_id(build_id); if (debug_file.empty()) { std::string hex = to_hex(build_id); xbt_die("Missing debug info for %s with build-id %s\n" "You might want to install the suitable debugging package.\n", info->file_name.c_str(), hex.c_str()); } // Load the DWARF info from this file: XBT_DEBUG("Load DWARF for %s from %s", info->file_name.c_str(), debug_file.c_str()); fd = open(debug_file.c_str(), O_RDONLY); if (fd < 0) xbt_die("Could not open file %s", debug_file.c_str()); Dwarf* dwarf = dwarf_begin(fd, DWARF_C_READ); if (dwarf == nullptr) xbt_die("No DWARF info in %s for %s", debug_file.c_str(), info->file_name.c_str()); read_dwarf_info(info, dwarf); dwarf_end(dwarf); close(fd); return; } // TODO, try to find DWARF info using .gnu_debuglink. elf_end(elf); close(fd); xbt_die("Debugging information not found for %s\n" "Try recompiling with -g\n", info->file_name.c_str()); } // ***** Functions index static int MC_compare_frame_index_items(simgrid::mc::FunctionIndexEntry* a, simgrid::mc::FunctionIndexEntry* b) { if (a->low_pc < b->low_pc) return -1; else if (a->low_pc == b->low_pc) return 0; else return 1; } static void MC_make_functions_index(simgrid::mc::ObjectInformation* info) { info->functions_index.clear(); for (auto& e : info->subprograms) { if (e.second.range.begin() == 0) continue; simgrid::mc::FunctionIndexEntry entry; entry.low_pc = (void*) e.second.range.begin(); entry.function = &e.second; info->functions_index.push_back(entry); } info->functions_index.shrink_to_fit(); // Sort the array by low_pc: boost::range::sort(info->functions_index, [](simgrid::mc::FunctionIndexEntry const& a, simgrid::mc::FunctionIndexEntry const& b) { return a.low_pc < b.low_pc; }); } static void MC_post_process_variables(simgrid::mc::ObjectInformation* info) { // Someone needs this to be sorted but who? boost::range::sort(info->global_variables, MC_compare_variable); for(simgrid::mc::Variable& variable : info->global_variables) if (variable.type_id) variable.type = simgrid::util::find_map_ptr( info->types, variable.type_id); } static void mc_post_process_scope(simgrid::mc::ObjectInformation* info, simgrid::mc::Frame* scope) { if (scope->tag == DW_TAG_inlined_subroutine) { // Attach correct namespaced name in inlined subroutine: auto i = info->subprograms.find(scope->abstract_origin_id); xbt_assert(i != info->subprograms.end(), "Could not lookup abstract origin %" PRIx64, (std::uint64_t) scope->abstract_origin_id); scope->name = i->second.name; } // Direct: for (simgrid::mc::Variable& variable : scope->variables) if (variable.type_id) variable.type = simgrid::util::find_map_ptr( info->types, variable.type_id); // Recursive post-processing of nested-scopes: for (simgrid::mc::Frame& nested_scope : scope->scopes) mc_post_process_scope(info, &nested_scope); } static simgrid::mc::Type* MC_resolve_type( simgrid::mc::ObjectInformation* info, unsigned type_id) { if (!type_id) return nullptr; simgrid::mc::Type* type = simgrid::util::find_map_ptr(info->types, type_id); if (type == nullptr) return nullptr; // We already have the information on the type: if (type->byte_size != 0) return type; // Don't have a name, we can't find a more complete version: if (type->name.empty()) return type; // Try to find a more complete description of the type: // We need to fix in order to support C++. simgrid::mc::Type** subtype = simgrid::util::find_map_ptr( info->full_types_by_name, type->name); if (subtype) type = *subtype; return type; } static void MC_post_process_types(simgrid::mc::ObjectInformation* info) { // Lookup "subtype" field: for(auto& i : info->types) { i.second.subtype = MC_resolve_type(info, i.second.type_id); for (simgrid::mc::Member& member : i.second.members) member.type = MC_resolve_type(info, member.type_id); } } namespace simgrid { namespace mc { /** \brief Finds informations about a given shared object/executable */ std::shared_ptr createObjectInformation( std::vector const& maps, const char *name) { std::shared_ptr result = std::make_shared(); result->file_name = name; simgrid::mc::find_object_address(maps, result.get()); MC_load_dwarf(result.get()); MC_post_process_variables(result.get()); MC_post_process_types(result.get()); for (auto& entry : result.get()->subprograms) mc_post_process_scope(result.get(), &entry.second); MC_make_functions_index(result.get()); return result; } /*************************************************************************/ void postProcessObjectInformation(simgrid::mc::Process* process, simgrid::mc::ObjectInformation* info) { for (auto& i : info->types) { simgrid::mc::Type* type = &(i.second); simgrid::mc::Type* subtype = type; while (subtype->type == DW_TAG_typedef || subtype->type == DW_TAG_volatile_type || subtype->type == DW_TAG_const_type) if (subtype->subtype) subtype = subtype->subtype; else break; // Resolve full_type: if (!subtype->name.empty() && subtype->byte_size == 0) for (auto const& object_info : process->object_infos) { auto i = object_info->full_types_by_name.find(subtype->name); if (i != object_info->full_types_by_name.end() && !i->second->name.empty() && i->second->byte_size) { type->full_type = i->second; break; } } else type->full_type = subtype; } } } } namespace simgrid { namespace dwarf { /** Convert a DWARF register into a libunwind register * * DWARF and libunwind does not use the same convention for numbering the * registers on some architectures. The function makes the necessary * conversion. */ int dwarf_register_to_libunwind(int dwarf_register) { #if defined(__x86_64__) // It seems for this arch, DWARF and libunwind agree in the numbering: return dwarf_register; #elif defined(__i386__) // Couldn't find the authoritative source of information for this. // This is inspired from http://source.winehq.org/source/dlls/dbghelp/cpu_i386.c#L517. switch (dwarf_register) { case 0: return UNW_X86_EAX; case 1: return UNW_X86_ECX; case 2: return UNW_X86_EDX; case 3: return UNW_X86_EBX; case 4: return UNW_X86_ESP; case 5: return UNW_X86_EBP; case 6: return UNW_X86_ESI; case 7: return UNW_X86_EDI; case 8: return UNW_X86_EIP; case 9: return UNW_X86_EFLAGS; case 10: return UNW_X86_CS; case 11: return UNW_X86_SS; case 12: return UNW_X86_DS; case 13: return UNW_X86_ES; case 14: return UNW_X86_FS; case 15: return UNW_X86_GS; case 16: return UNW_X86_ST0; case 17: return UNW_X86_ST1; case 18: return UNW_X86_ST2; case 19: return UNW_X86_ST3; case 20: return UNW_X86_ST4; case 21: return UNW_X86_ST5; case 22: return UNW_X86_ST6; case 23: return UNW_X86_ST7; default: xbt_die("Bad/unknown register number."); } #else #error This architecture is not supported yet for DWARF expression evaluation. #endif } } } SimGrid-3.14.159/src/mc/mc_comm_pattern.cpp000644 001750 001750 00000007461 13031031270 022137 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include "src/mc/mc_comm_pattern.h" #include "src/mc/mc_smx.h" #include "src/mc/mc_xbt.hpp" #include "src/mc/Checker.hpp" #include "src/mc/CommunicationDeterminismChecker.hpp" using simgrid::mc::remote; XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_comm_pattern, mc, "Logging specific to MC communication patterns"); static void MC_patterns_copy(xbt_dynar_t dest, std::vector const& source) { xbt_dynar_reset(dest); for (simgrid::mc::PatternCommunication const& comm : source) { simgrid::mc::PatternCommunication* copy_comm = new simgrid::mc::PatternCommunication(comm.dup()); xbt_dynar_push(dest, ©_comm); } } void MC_restore_communications_pattern(simgrid::mc::State* state) { simgrid::mc::PatternCommunicationList* list_process_comm; unsigned int cursor; xbt_dynar_foreach(initial_communications_pattern, cursor, list_process_comm) list_process_comm->index_comm = state->communicationIndices[cursor]; for (unsigned i = 0; i < MC_smx_get_maxpid(); i++) MC_patterns_copy( xbt_dynar_get_as(incomplete_communications_pattern, i, xbt_dynar_t), state->incomplete_comm_pattern[i] ); } void MC_state_copy_incomplete_communications_pattern(simgrid::mc::State* state) { state->incomplete_comm_pattern.clear(); for (unsigned i=0; i < MC_smx_get_maxpid(); i++) { xbt_dynar_t patterns = xbt_dynar_get_as(incomplete_communications_pattern, i, xbt_dynar_t); std::vector res; simgrid::mc::PatternCommunication* comm; unsigned int cursor; xbt_dynar_foreach(patterns, cursor, comm) res.push_back(comm->dup()); state->incomplete_comm_pattern.push_back(std::move(res)); } } void MC_state_copy_index_communications_pattern(simgrid::mc::State* state) { state->communicationIndices.clear(); simgrid::mc::PatternCommunicationList* list_process_comm; unsigned int cursor; xbt_dynar_foreach(initial_communications_pattern, cursor, list_process_comm) state->communicationIndices.push_back(list_process_comm->index_comm); } void MC_handle_comm_pattern( e_mc_call_type_t call_type, smx_simcall_t req, int value, xbt_dynar_t pattern, int backtracking) { // HACK, do not rely on the Checker implementation outside of it simgrid::mc::CommunicationDeterminismChecker* checker = (simgrid::mc::CommunicationDeterminismChecker*) mc_model_checker->getChecker(); switch(call_type) { case MC_CALL_TYPE_NONE: break; case MC_CALL_TYPE_SEND: case MC_CALL_TYPE_RECV: checker->get_comm_pattern(pattern, req, call_type, backtracking); break; case MC_CALL_TYPE_WAIT: case MC_CALL_TYPE_WAITANY: { simgrid::mc::RemotePtr comm_addr = nullptr; if (call_type == MC_CALL_TYPE_WAIT) comm_addr = remote(static_cast( simcall_comm_wait__get__comm(req))); else { simgrid::kernel::activity::Comm* addr; // comm_addr = REMOTE(xbt_dynar_get_as(simcall_comm_waitany__get__comms(req), value, smx_synchro_t)): simgrid::mc::read_element(mc_model_checker->process(), &addr, remote(simcall_comm_waitany__get__comms(req)), value, sizeof(comm_addr)); comm_addr = remote(addr); } checker->complete_comm_pattern(pattern, comm_addr, MC_smx_simcall_get_issuer(req)->pid, backtracking); } break; default: xbt_die("Unexpected call type %i", (int)call_type); } } SimGrid-3.14.159/src/mc/mc_page_snapshot.cpp000644 001750 001750 00000003325 13031031270 022275 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* MC interface: definitions that non-MC modules must see, but not the user */ #include // pread, pwrite #include "src/mc/PageStore.hpp" #include "src/mc/mc_mmu.h" #include "src/mc/mc_private.h" #include "src/mc/mc_snapshot.h" #include #include "src/mc/ChunkedData.hpp" using simgrid::mc::remote; extern "C" { /** @brief Restore a snapshot of a region * * If possible, the restoration will be incremental * (the modified pages will not be touched). * * @param start_addr * @param page_count Number of pages of the region * @param pagenos */ void mc_restore_page_snapshot_region(simgrid::mc::Process* process, void* start_addr, simgrid::mc::ChunkedData const& pages_copy) { for (size_t i = 0; i != pages_copy.page_count(); ++i) { // Otherwise, copy the page: void* target_page = (void*) simgrid::mc::mmu::join(i, (std::uintptr_t) start_addr); const void* source_page = pages_copy.page(i); process->write_bytes(source_page, xbt_pagesize, remote(target_page)); } } // ***** High level API void mc_region_restore_sparse(simgrid::mc::Process* process, mc_mem_region_t reg) { xbt_assert(((reg->permanent_address().address()) & (xbt_pagesize-1)) == 0, "Not at the beginning of a page"); xbt_assert(simgrid::mc::mmu::chunkCount(reg->size()) == reg->page_data().page_count()); mc_restore_page_snapshot_region(process, (void*) reg->permanent_address().address(), reg->page_data()); } } SimGrid-3.14.159/src/mc/Process.hpp000644 001750 001750 00000022560 13031031270 020410 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_PROCESS_H #define SIMGRID_MC_PROCESS_H #include #include #include #include #include #include #include #include #include #include #include "src/xbt/mmalloc/mmprivate.h" #include "src/mc/Channel.hpp" #include #include "src/simix/popping_private.h" #include "src/simix/smx_private.h" #include "src/xbt/memory_map.hpp" #include "src/mc/mc_forward.hpp" #include "src/mc/mc_base.h" #include "src/mc/RemotePtr.hpp" #include "src/mc/AddressSpace.hpp" #include "src/mc/mc_protocol.h" #include "src/mc/ObjectInformation.hpp" namespace simgrid { namespace mc { class SimixProcessInformation { public: /** MCed address of the process */ RemotePtr address = nullptr; Remote copy; /** Hostname (owned by `mc_modelchecker->hostnames`) */ const char* hostname = nullptr; std::string name; void clear() { name.clear(); address = nullptr; hostname = nullptr; } }; struct IgnoredRegion { std::uint64_t addr; std::size_t size; }; struct IgnoredHeapRegion { int block; int fragment; void *address; std::size_t size; }; /** Representation of a process * * This class is mixing a lot of different responsibilities and is tied * to SIMIX. It should probably be split into different classes. * * Responsibilities: * * - reading from the process memory (`AddressSpace`); * - accessing the system state of the process (heap, …); * - storing the SIMIX state of the process; * - privatization; * - communication with the model-checked process; * - stack unwinding; * - etc. */ class Process final : public AddressSpace { private: // Those flags are used to track down which cached information // is still up to date and which information needs to be updated. static constexpr int cache_none = 0; static constexpr int cache_heap = 1; static constexpr int cache_malloc = 2; static constexpr int cache_simix_processes = 4; public: Process(pid_t pid, int sockfd); ~Process(); void init(); Process(Process const&) = delete; Process(Process &&) = delete; Process& operator=(Process const&) = delete; Process& operator=(Process &&) = delete; // Read memory: const void* read_bytes(void* buffer, std::size_t size, RemotePtr address, int process_index = ProcessIndexAny, ReadOptions options = ReadOptions::none()) const override; void read_variable(const char* name, void* target, size_t size) const; template void read_variable(const char* name, T* target) const { read_variable(name, target, sizeof(*target)); } template Remote read_variable(const char *name) const { Remote res; read_variable(name, res.getBuffer(), sizeof(T)); return res; } std::string read_string(RemotePtr address) const; std::string read_string(RemotePtr address, std::size_t len) const { return AddressSpace::read_string(address, len); } // Write memory: void write_bytes(const void* buffer, size_t len, RemotePtr address); void clear_bytes(RemotePtr address, size_t len); // Debug information: std::shared_ptr find_object_info(RemotePtr addr) const; std::shared_ptr find_object_info_exec(RemotePtr addr) const; std::shared_ptr find_object_info_rw(RemotePtr addr) const; simgrid::mc::Frame* find_function(RemotePtr ip) const; simgrid::mc::Variable* find_variable(const char* name) const; // Heap access: xbt_mheap_t get_heap() { if (!(this->cache_flags_ & Process::cache_heap)) this->refresh_heap(); return this->heap.get(); } malloc_info* get_malloc_info() { if (!(this->cache_flags_ & Process::cache_malloc)) this->refresh_malloc_info(); return this->heap_info.data(); } void clear_cache() { this->cache_flags_ = Process::cache_none; } Channel const& getChannel() const { return channel_; } Channel& getChannel() { return channel_; } std::vector const& ignored_regions() const { return ignored_regions_; } void ignore_region(std::uint64_t address, std::size_t size); pid_t pid() const { return pid_; } bool in_maestro_stack(RemotePtr p) const { return p >= this->maestro_stack_start_ && p < this->maestro_stack_end_; } bool running() const { return running_; } void terminate() { running_ = false; } bool privatized(ObjectInformation const& info) const { return privatized_ && info.executable(); } bool privatized() const { return privatized_; } void privatized(bool privatized) { privatized_ = privatized; } void ignore_global_variable(const char* name) { for (std::shared_ptr const& info : this->object_infos) info->remove_global_variable(name); } std::vector& stack_areas() { return stack_areas_; } std::vector const& stack_areas() const { return stack_areas_; } std::vector const& ignored_heap() const { return ignored_heap_; } void ignore_heap(IgnoredHeapRegion const& region); void unignore_heap(void *address, size_t size); void ignore_local_variable(const char *var_name, const char *frame_name); std::vector& simix_processes(); std::vector& old_simix_processes(); /** Get a local description of a remote SIMIX process */ simgrid::mc::SimixProcessInformation* resolveProcessInfo( simgrid::mc::RemotePtr process) { xbt_assert(mc_model_checker != nullptr); if (!process) return nullptr; this->refresh_simix(); for (auto& process_info : this->smx_process_infos) if (process_info.address == process) return &process_info; for (auto& process_info : this->smx_old_process_infos) if (process_info.address == process) return &process_info; return nullptr; } /** Get a local copy of the SIMIX process structure */ simgrid::simix::ActorImpl* resolveProcess(simgrid::mc::RemotePtr process) { simgrid::mc::SimixProcessInformation* process_info = this->resolveProcessInfo(process); if (process_info) return process_info->copy.getBuffer(); else return nullptr; } void dumpStack(); private: void init_memory_map_info(); void refresh_heap(); void refresh_malloc_info(); void refresh_simix(); private: pid_t pid_ = -1; Channel channel_; bool running_ = false; std::vector memory_map_; RemotePtr maestro_stack_start_, maestro_stack_end_; int memory_file = -1; std::vector ignored_regions_; bool privatized_ = false; std::vector stack_areas_; std::vector ignored_heap_; public: // object info // TODO, make private (first, objectify simgrid::mc::ObjectInformation*) std::vector> object_infos; std::shared_ptr libsimgrid_info; std::shared_ptr binary_info; public: // Copies of MCed SMX data structures /** Copy of `simix_global->process_list` * * See mc_smx.c. */ std::vector smx_process_infos; /** Copy of `simix_global->process_to_destroy` * * See mc_smx.c. */ std::vector smx_old_process_infos; private: /** State of the cache (which variables are up to date) */ int cache_flags_ = Process::cache_none; public: /** Address of the heap structure in the MCed process. */ void* heap_address; /** Copy of the heap structure of the process * * This is refreshed with the `MC_process_refresh` call. * This is not used if the process is the current one: * use `get_heap_info()` in order to use it. */ std::unique_ptr heap; /** Copy of the allocation info structure * * This is refreshed with the `MC_process_refresh` call. * This is not used if the process is the current one: * use `get_malloc_info()` in order to use it. */ std::vector heap_info; public: // Libunwind-data /** Full-featured MC-aware libunwind address space for the process * * This address space is using a simgrid::mc::UnwindContext* * (with simgrid::mc::Process* / simgrid::mc::AddressSpace* * and unw_context_t). */ unw_addr_space_t unw_addr_space; /** Underlying libunwind address-space * * The `find_proc_info`, `put_unwind_info`, `get_dyn_info_list_addr` * operations of the native MC address space is currently delegated * to this address space (either the local or a ptrace unwinder). */ unw_addr_space_t unw_underlying_addr_space; /** The corresponding context */ void* unw_underlying_context; }; /** Open a FD to a remote process memory (`/dev/$pid/mem`) */ XBT_PRIVATE int open_vm(pid_t pid, int flags); } } #endif SimGrid-3.14.159/src/mc/VisitedState.hpp000644 001750 001750 00000002065 13031031270 021400 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_VISITED_STATE_HPP #define SIMGRID_MC_VISITED_STATE_HPP #include #include #include "src/mc/mc_snapshot.h" namespace simgrid { namespace mc { struct XBT_PRIVATE VisitedState { std::shared_ptr system_state = nullptr; std::size_t heap_bytes_used = 0; int nb_processes = 0; int num = 0; int other_num = 0; // dot_output for VisitedState(unsigned long state_number); ~VisitedState(); }; class XBT_PRIVATE VisitedStates { std::vector> states_; public: void clear() { states_.clear(); } std::unique_ptr addVisitedState(unsigned long state_number, simgrid::mc::State* graph_state, bool compare_snpashots); private: void prune(); }; } } #endif SimGrid-3.14.159/src/mc/CommunicationDeterminismChecker.hpp000644 001750 001750 00000003501 13031031270 025257 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include "src/mc/mc_forward.hpp" #include "src/mc/Checker.hpp" #include "src/mc/VisitedState.hpp" #ifndef SIMGRID_MC_COMMUNICATION_DETERMINISM_CHECKER_HPP #define SIMGRID_MC_COMMUNICATION_DETERMINISM_CHECKER_HPP namespace simgrid { namespace mc { class XBT_PRIVATE CommunicationDeterminismChecker : public Checker { public: CommunicationDeterminismChecker(Session& session); ~CommunicationDeterminismChecker(); int run() override; RecordTrace getRecordTrace() override; std::vector getTextualTrace() override; private: void prepare(); int main(); void logState() override; void deterministic_comm_pattern(int process, simgrid::mc::PatternCommunication* comm, int backtracking); void restoreState(); public: // These are used by functions which should be moved in CommunicationDeterminismChecker: void get_comm_pattern(xbt_dynar_t list, smx_simcall_t request, e_mc_call_type_t call_type, int backtracking); void complete_comm_pattern( xbt_dynar_t list, simgrid::mc::RemotePtr comm_addr, unsigned int issuer, int backtracking); private: /** Stack representing the position in the exploration graph */ std::list> stack_; simgrid::mc::VisitedStates visitedStates_; unsigned long expandedStatesCount_ = 0; int initial_communications_pattern_done = 0; int recv_deterministic = 0; int send_deterministic = 0; char *send_diff = nullptr; char *recv_diff = nullptr; }; #endif } } SimGrid-3.14.159/src/mc/mc_smx.cpp000644 001750 001750 00000014705 13031031271 020256 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include #include #include #include #include #include "src/simix/smx_private.h" #include "src/mc/mc_smx.h" #include "src/mc/ModelChecker.hpp" using simgrid::mc::remote; /** HACK, Statically "upcast" a s_smx_actor_t into a SimixProcessInformation * * This gets 'processInfo' from '&processInfo->copy'. It upcasts in the * sense that we could achieve the same thing by having SimixProcessInformation * inherit from s_smx_actor_t but we don't really want to do that. */ static inline simgrid::mc::SimixProcessInformation* process_info_cast(smx_actor_t p) { simgrid::mc::SimixProcessInformation temp; std::size_t offset = (char*) temp.copy.getBuffer() - (char*)&temp; simgrid::mc::SimixProcessInformation* process_info = (simgrid::mc::SimixProcessInformation*) ((char*) p - offset); return process_info; } /** Load the remote swag of processes into a vector * * @param process MCed process * @param target Local vector (to be filled with copies of `s_smx_actor_t`) * @param remote_swag Address of the process SWAG in the remote list */ static void MC_process_refresh_simix_process_list( simgrid::mc::Process* process, std::vector& target, simgrid::mc::RemotePtr remote_swag) { target.clear(); // swag = REMOTE(*simix_global->process_list) s_xbt_swag_t swag; process->read_bytes(&swag, sizeof(swag), remote_swag); // Load each element of the vector from the MCed process: int i = 0; for (smx_actor_t p = (smx_actor_t) swag.head; p; ++i) { simgrid::mc::SimixProcessInformation info; info.address = p; info.hostname = nullptr; process->read_bytes(&info.copy, sizeof(info.copy), remote(p)); target.push_back(std::move(info)); // Lookup next process address: p = (smx_actor_t) xbt_swag_getNext(&info.copy, swag.offset); } assert(i == swag.count); } namespace simgrid { namespace mc { void Process::refresh_simix() { if (this->cache_flags_ & Process::cache_simix_processes) return; // TODO, avoid to reload `&simix_global`, `simix_global`, `*simix_global` static_assert(std::is_same< std::unique_ptr, decltype(simix_global) >::value, "Unexpected type for simix_global"); static_assert(sizeof(simix_global) == sizeof(simgrid::simix::Global*), "Bad size for simix_global"); // simix_global_p = REMOTE(simix_global.get()); RemotePtr simix_global_p = this->read_variable("simix_global"); // simix_global = REMOTE(*simix_global) Remote simix_global = this->read(simix_global_p); MC_process_refresh_simix_process_list( this, this->smx_process_infos, remote(simix_global.getBuffer()->process_list)); MC_process_refresh_simix_process_list( this, this->smx_old_process_infos, remote(simix_global.getBuffer()->process_to_destroy)); this->cache_flags_ |= Process::cache_simix_processes; } } } /** Get the issuer of a simcall (`req->issuer`) * * In split-process mode, it does the black magic necessary to get an address * of a (shallow) copy of the data structure the issuer SIMIX process in the local * address space. * * @param process the MCed process * @param req the simcall (copied in the local process) */ smx_actor_t MC_smx_simcall_get_issuer(s_smx_simcall_t const* req) { xbt_assert(mc_model_checker != nullptr); // This is the address of the smx_actor in the MCed process: auto address = simgrid::mc::remote(req->issuer); // Lookup by address: for (auto& p : mc_model_checker->process().simix_processes()) if (p.address == address) return p.copy.getBuffer(); for (auto& p : mc_model_checker->process().old_simix_processes()) if (p.address == address) return p.copy.getBuffer(); xbt_die("Issuer not found"); } const char* MC_smx_process_get_host_name(smx_actor_t p) { if (mc_model_checker == nullptr) return p->host->cname(); simgrid::mc::Process* process = &mc_model_checker->process(); /* HACK, Horrible hack to find the offset of the id in the simgrid::s4u::Host. Offsetof is not supported for non-POD types but this should work in practice for the targets currently supported by the MC as long as we do not add funny features to the Host class (such as virtual base). We are using a (C++11) unrestricted union in order to avoid any construction/destruction of the simgrid::s4u::Host. */ union fake_host { simgrid::s4u::Host host; fake_host() {} ~fake_host() {} }; fake_host foo; const size_t offset = (char*) &foo.host.name() - (char*) &foo.host; // Read the simgrid::xbt::string in the MCed process: simgrid::mc::SimixProcessInformation* info = process_info_cast(p); auto remote_string_address = remote( (simgrid::xbt::string_data*) ((char*) p->host + offset)); simgrid::xbt::string_data remote_string = process->read(remote_string_address); char hostname[remote_string.len]; process->read_bytes(hostname, remote_string.len + 1, remote(remote_string.data)); info->hostname = mc_model_checker->get_host_name(hostname).c_str(); return info->hostname; } const char* MC_smx_process_get_name(smx_actor_t p) { simgrid::mc::Process* process = &mc_model_checker->process(); if (mc_model_checker == nullptr) return p->name.c_str(); simgrid::mc::SimixProcessInformation* info = process_info_cast(p); if (info->name.empty()) { simgrid::xbt::string_data string_data = (simgrid::xbt::string_data&)p->name; info->name = process->read_string(remote(string_data.data), string_data.len); } return info->name.c_str(); } #if HAVE_SMPI int MC_smpi_process_count(void) { if (mc_model_checker == nullptr) return smpi_process_count(); int res; mc_model_checker->process().read_variable("process_count", &res, sizeof(res)); return res; } #endif unsigned long MC_smx_get_maxpid(void) { unsigned long maxpid; mc_model_checker->process().read_variable("simix_process_maxpid", &maxpid, sizeof(maxpid)); return maxpid; } SimGrid-3.14.159/src/mc/mc_request.h000644 001750 001750 00000002035 13031031270 020574 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_REQUEST_H #define SIMGRID_MC_REQUEST_H #include #include #include "src/simix/smx_private.h" namespace simgrid { namespace mc { enum class RequestType { simix, executed, internal, }; XBT_PRIVATE bool request_depend(smx_simcall_t req1, smx_simcall_t req2); XBT_PRIVATE std::string request_to_string(smx_simcall_t req, int value, simgrid::mc::RequestType type); XBT_PRIVATE bool request_is_enabled_by_idx(smx_simcall_t req, unsigned int idx); /** Is the process ready to execute its simcall? * * This is true if the request associated with the process is ready. */ XBT_PRIVATE bool process_is_enabled(smx_actor_t process); XBT_PRIVATE std::string request_get_dot_output(smx_simcall_t req, int value); } } #endif SimGrid-3.14.159/src/mc/mc_smx.h000644 001750 001750 00000003543 13031031271 017721 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_SMX_H #define SIMGRID_MC_SMX_H #include #include #include #include #include "src/smpi/private.h" #include "src/mc/Process.hpp" #include "src/mc/mc_protocol.h" /** @file * @brief (Cross-process, MCer/MCed) Access to SMX structures * * We copy some C data structure from the MCed process in the MCer process. * This is implemented by: * * - `model_checker->process.smx_process_infos` * (copy of `simix_global->process_list`); * * - `model_checker->process.smx_old_process_infos` * (copy of `simix_global->process_to_destroy`); * * - `model_checker->hostnames`. * * The process lists are currently refreshed each time MCed code is executed. * We don't try to give a persistent MCer address for a given MCed process. * For this reason, a MCer simgrid::mc::Process* is currently not reusable after * MCed code. */ SG_BEGIN_DECL() /** Get the issuer of a simcall (`req->issuer`) * * In split-process mode, it does the black magic necessary to get an address * of a (shallow) copy of the data structure the issuer SIMIX process in the local * address space. * * @param process the MCed process * @param req the simcall (copied in the local process) */ XBT_PRIVATE smx_actor_t MC_smx_simcall_get_issuer(s_smx_simcall_t const* req); XBT_PRIVATE const char* MC_smx_process_get_name(smx_actor_t p); XBT_PRIVATE const char* MC_smx_process_get_host_name(smx_actor_t p); XBT_PRIVATE int MC_smpi_process_count(void); XBT_PRIVATE unsigned long MC_smx_get_maxpid(void); SG_END_DECL() #endif SimGrid-3.14.159/src/mc/mc_global.cpp000644 001750 001750 00000011303 13031031270 020675 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include #include #include #include "mc_base.h" #include "mc/mc.h" #ifndef _WIN32 #include #include #include #endif #include "src/simix/ActorImpl.hpp" #if HAVE_MC #include #include "src/mc/mc_comm_pattern.h" #include "src/mc/mc_request.h" #include "src/mc/mc_safety.h" #include "src/mc/mc_snapshot.h" #include "src/mc/mc_private.h" #include "src/mc/mc_unw.h" #include "src/mc/mc_smx.h" #include "src/mc/Checker.hpp" #endif #include "src/mc/mc_record.h" #include "src/mc/mc_protocol.h" #include "src/mc/Client.hpp" #include "src/mc/Transition.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_global, mc, "Logging specific to MC (global)"); namespace simgrid { namespace mc { std::vector processes_time; } } #if HAVE_MC /* Liveness */ namespace simgrid { namespace mc { xbt_automaton_t property_automaton = nullptr; } } /* Dot output */ FILE *dot_output = nullptr; /******************************* Initialization of MC *******************************/ /*********************************************************************************/ void MC_init_dot_output() { dot_output = fopen(_sg_mc_dot_output_file, "w"); if (dot_output == nullptr) { perror("Error open dot output file"); xbt_abort(); } fprintf(dot_output, "digraph graphname{\n fixedsize=true; rankdir=TB; ranksep=.25; edge [fontsize=12]; node [fontsize=10, shape=circle,width=.5 ]; graph [resolution=20, fontsize=10];\n"); } /******************************* Core of MC *******************************/ /**************************************************************************/ void MC_run() { simgrid::mc::processes_time.resize(SIMIX_process_get_maxpid()); MC_ignore_heap(simgrid::mc::processes_time.data(), simgrid::mc::processes_time.size() * sizeof(simgrid::mc::processes_time[0])); smx_actor_t process; xbt_swag_foreach(process, simix_global->process_list) MC_ignore_heap(&(process->process_hookup), sizeof(process->process_hookup)); simgrid::mc::Client::get()->mainLoop(); simgrid::mc::processes_time.clear(); } void MC_show_deadlock(void) { XBT_INFO("**************************"); XBT_INFO("*** DEAD-LOCK DETECTED ***"); XBT_INFO("**************************"); XBT_INFO("Counter-example execution trace:"); for (auto& s : mc_model_checker->getChecker()->getTextualTrace()) XBT_INFO("%s", s.c_str()); simgrid::mc::session->logState(); } void MC_automaton_load(const char *file) { if (simgrid::mc::property_automaton == nullptr) simgrid::mc::property_automaton = xbt_automaton_new(); xbt_automaton_load(simgrid::mc::property_automaton, file); } namespace simgrid { namespace mc { void dumpStack(FILE* file, unw_cursor_t cursor) { int nframe = 0; char buffer[100]; unw_word_t off; do { const char * name = !unw_get_proc_name(&cursor, buffer, 100, &off) ? buffer : "?"; int status; // Unmangle C++ names: char* realname = abi::__cxa_demangle(name, 0, 0, &status); #if defined(__x86_64__) unw_word_t rip = 0; unw_word_t rsp = 0; unw_get_reg(&cursor, UNW_X86_64_RIP, &rip); unw_get_reg(&cursor, UNW_X86_64_RSP, &rsp); fprintf(file, " %i: %s (RIP=0x%" PRIx64 " RSP=0x%" PRIx64 ")\n", nframe, realname ? realname : name, (std::uint64_t) rip, (std::uint64_t) rsp); #else fprintf(file, " %i: %s\n", nframe, realname ? realname : name); #endif free(realname); ++nframe; } while(unw_step(&cursor)); } } } static void MC_dump_stacks(FILE* file) { int nstack = 0; for (auto const& stack : mc_model_checker->process().stack_areas()) { fprintf(file, "Stack %i:\n", nstack++); simgrid::mc::UnwindContext context; unw_context_t raw_context = (unw_context_t) mc_model_checker->process().read( simgrid::mc::remote((unw_context_t *)stack.context)); context.initialize(&mc_model_checker->process(), &raw_context); unw_cursor_t cursor = context.cursor(); simgrid::mc::dumpStack(file, cursor); } } #endif double MC_process_clock_get(smx_actor_t process) { if (simgrid::mc::processes_time.empty()) return 0; if (process != nullptr) return simgrid::mc::processes_time[process->pid]; return -1; } void MC_process_clock_add(smx_actor_t process, double amount) { simgrid::mc::processes_time[process->pid] += amount; } SimGrid-3.14.159/src/mc/PageStore.cpp000644 001750 001750 00000017746 13031031270 020670 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include // memcpy, memcmp #include #ifdef __FreeBSD__ # define MAP_POPULATE MAP_PREFAULT_READ #endif #include #include #include #include "src/internal_config.h" #include "src/mc/PageStore.hpp" #include "src/mc/mc_mmu.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_page_snapshot, mc, "Logging specific to mc_page_snapshot"); namespace simgrid { namespace mc { /** @brief Compute a hash for the given memory page * * The page is used before inserting the page in the page store * in order to find duplicate of this page in the page store. * * @param data Memory page * @return hash off the page */ static inline __attribute__ ((always_inline)) PageStore::hash_type mc_hash_page(const void* data) { const std::uint64_t* values = (const uint64_t*) data; std::size_t n = xbt_pagesize / sizeof(uint64_t); // This djb2: std::uint64_t hash = 5381; for (std::size_t i = 0; i != n; ++i) hash = ((hash << 5) + hash) + values[i]; return hash; } // ***** snapshot_page_manager PageStore::PageStore(size_t size) : memory_(nullptr), capacity_(0), top_index_(0) { // Using mmap in order to be able to expand the region // by relocating it somewhere else in the virtual memory // space: void* memory = ::mmap(nullptr, size << xbt_pagebits, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_POPULATE, -1, 0); if (memory == MAP_FAILED) xbt_die("Could not mmap initial snapshot pages."); this->top_index_ = 0; this->capacity_ = size; this->memory_ = memory; this->page_counts_.resize(size); } PageStore::~PageStore() { ::munmap(this->memory_, this->capacity_ << xbt_pagebits); } void PageStore::resize(std::size_t size) { size_t old_bytesize = this->capacity_ << xbt_pagebits; size_t new_bytesize = size << xbt_pagebits; void *new_memory; // Expand the memory region by moving it into another // virtual memory address if necessary: #if HAVE_MREMAP new_memory = mremap(this->memory_, old_bytesize, new_bytesize, MREMAP_MAYMOVE); if (new_memory == MAP_FAILED) xbt_die("Could not mremap snapshot pages."); #else if (new_bytesize > old_bytesize) { // Grow: first try to add new space after current map new_memory = mmap((char *)this->memory_ + old_bytesize, new_bytesize-old_bytesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_POPULATE, -1, 0); if (new_memory == MAP_FAILED) xbt_die("Could not mremap snapshot pages."); // Check if expanding worked if (new_memory != (char *)this->memory_ + old_bytesize) { // New memory segment could not be put at the end of this->memory_, // so cancel this one and try to rellocate everything and copy data munmap(new_memory, new_bytesize-old_bytesize); new_memory = mmap(nullptr, new_bytesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_POPULATE, -1, 0); if (new_memory == MAP_FAILED) xbt_die("Could not mremap snapshot pages."); memcpy(new_memory, this->memory_, old_bytesize); munmap(this->memory_, old_bytesize); } } else { // We don't have functions to shrink a mapping, so leave memory as // it is for now new_memory = this->memory_; } #endif this->capacity_ = size; this->memory_ = new_memory; this->page_counts_.resize(size, 0); } /** Allocate a free page * * @return index of the free page */ std::size_t PageStore::alloc_page() { if (this->free_pages_.empty()) { // Expand the region: if (this->top_index_ == this->capacity_) // All the pages are allocated, we need add more pages: this->resize(2 * this->capacity_); // Use a page from the top: return this->top_index_++; } else { // Use a page from free_pages_ (inside of the region): size_t res = this->free_pages_[this->free_pages_.size() - 1]; this->free_pages_.pop_back(); return res; } } void PageStore::remove_page(std::size_t pageno) { this->free_pages_.push_back(pageno); const void* page = this->get_page(pageno); hash_type hash = mc_hash_page(page); this->hash_index_[hash].erase(pageno); } /** Store a page in memory */ std::size_t PageStore::store_page(void* page) { xbt_assert(top_index_ <= this->capacity_, "top_index is not consistent"); // First, we check if a page with the same content is already in the page // store: // 1. compute the hash of the page; // 2. find pages with the same hash using `hash_index_`; // 3. find a page with the same content. hash_type hash = mc_hash_page(page); // Try to find a duplicate in set of pages with the same hash: page_set_type& page_set = this->hash_index_[hash]; for (size_t pageno : page_set) { const void* snapshot_page = this->get_page(pageno); if (memcmp(page, snapshot_page, xbt_pagesize) == 0) { // If a page with the same content is already in the page store it is // reused and its reference count is incremented. page_counts_[pageno]++; return pageno; } } // Otherwise, a new page is allocated in the page store and the content // of the page is `memcpy()`-ed to this new page. std::size_t pageno = alloc_page(); xbt_assert(this->page_counts_[pageno]==0, "Allocated page is already used"); void* snapshot_page = (void*) this->get_page(pageno); memcpy(snapshot_page, page, xbt_pagesize); page_set.insert(pageno); page_counts_[pageno]++; return pageno; } } } #ifdef SIMGRID_TEST #include #include #include #include #include #include "src/mc/PageStore.hpp" static int value = 0; static void new_content(void* data, std::size_t size) { ::memset(data, ++value, size); } static void* getpage() { return mmap(nullptr, getpagesize(), PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0); } XBT_TEST_SUITE("mc_page_store", "Page store"); XBT_TEST_UNIT("base", test_mc_page_store, "Test adding/removing pages in the store") { using simgrid::mc::PageStore; xbt_test_add("Init"); std::size_t pagesize = (size_t) getpagesize(); std::unique_ptr store = std::unique_ptr(new simgrid::mc::PageStore(500)); void* data = getpage(); xbt_test_assert(store->size()==0, "Bad size"); xbt_test_add("Store the page once"); new_content(data, pagesize); size_t pageno1 = store->store_page(data); xbt_test_assert(store->get_ref(pageno1)==1, "Bad refcount"); const void* copy = store->get_page(pageno1); xbt_test_assert(::memcmp(data, copy, pagesize)==0, "Page data should be the same"); xbt_test_assert(store->size()==1, "Bad size"); xbt_test_add("Store the same page again"); size_t pageno2 = store->store_page(data); xbt_test_assert(pageno1==pageno2, "Page should be the same"); xbt_test_assert(store->get_ref(pageno1)==2, "Bad refcount"); xbt_test_assert(store->size()==1, "Bad size"); xbt_test_add("Store a new page"); new_content(data, pagesize); size_t pageno3 = store->store_page(data); xbt_test_assert(pageno1 != pageno3, "New page should be different"); xbt_test_assert(store->size()==2, "Bad size"); xbt_test_add("Unref pages"); store->unref_page(pageno1); xbt_assert(store->get_ref(pageno1)==1, "Bad refcount"); xbt_assert(store->size()==2, "Bad size"); store->unref_page(pageno2); xbt_test_assert(store->size()==1, "Bad size"); xbt_test_add("Reallocate page"); new_content(data, pagesize); size_t pageno4 = store->store_page(data); xbt_test_assert(pageno1 == pageno4, "Page was not reused"); xbt_test_assert(store->get_ref(pageno4)==1, "Bad refcount"); xbt_test_assert(store->size()==2, "Bad size"); } #endif /* SIMGRID_TEST */ SimGrid-3.14.159/src/mc/Transition.hpp000644 001750 001750 00000002010 13031031271 021111 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2015-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_TRANSITION_HPP #define SIMGRID_MC_TRANSITION_HPP namespace simgrid { namespace mc { /** An element in the recorded path * * At each decision point, we need to record which process transition * is triggered and potentially which value is associated with this * transition. The value is used to find which communication is triggered * in things like waitany and for associating a given value of MC_random() * calls. */ struct Transition { int pid = 0; /* Which transition was executed for this simcall * * Some simcalls can lead to different transitions: * * * waitany/testany can trigger on different messages; * * * random can produce different values. */ int argument = 0; }; } } #endif SimGrid-3.14.159/src/mc/compare.cpp000644 001750 001750 00000162560 13031031270 020420 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2016. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /** \file compare.cpp Memory snapshooting and comparison */ #include #include #include #include #include #include #include #include #include #include #include #include "src/internal_config.h" #include "src/xbt/mmalloc/mmprivate.h" #include "src/xbt/ex_interface.h" #if HAVE_SMPI #include "src/smpi/private.h" #endif #include "src/mc/mc_forward.hpp" #include "src/mc/mc_safety.h" #include "src/mc/mc_private.h" #include "src/mc/mc_smx.h" #include "src/mc/mc_dwarf.hpp" #include "src/mc/Frame.hpp" #include "src/mc/ObjectInformation.hpp" #include "src/mc/Variable.hpp" #include "src/mc/mc_private.h" #include "src/mc/mc_snapshot.h" #include "src/mc/mc_dwarf.hpp" #include "src/mc/Type.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_compare, xbt, "Logging specific to mc_compare in mc"); namespace simgrid { namespace mc { struct HeapLocation; typedef std::array HeapLocationPair; typedef std::set HeapLocationPairs; struct HeapArea; struct ProcessComparisonState; struct StateComparator; static int compare_heap_area( StateComparator& state, int process_index, const void *area1, const void* area2, Snapshot* snapshot1, Snapshot* snapshot2, HeapLocationPairs* previous, Type* type, int pointer_level); } } using simgrid::mc::remote; /*********************************** Heap comparison ***********************************/ /***************************************************************************************/ namespace simgrid { namespace mc { struct HeapLocation { int block = 0; int fragment = 0; HeapLocation() {} HeapLocation(int block, int fragment = 0) : block(block), fragment(fragment) {} bool operator==(HeapLocation const& that) const { return block == that.block && fragment == that.fragment; } bool operator<(HeapLocation const& that) const { return std::make_pair(block, fragment) < std::make_pair(that.block, that.fragment); } }; static inline HeapLocationPair makeHeapLocationPair(int block1, int fragment1, int block2, int fragment2) { return simgrid::mc::HeapLocationPair{{ simgrid::mc::HeapLocation(block1, fragment1), simgrid::mc::HeapLocation(block2, fragment2) }}; } struct HeapArea : public HeapLocation { bool valid = false; int block = 0; int fragment = 0; HeapArea() {} HeapArea(int block) : valid(true), block(block) {} HeapArea(int block, int fragment = 0) : valid(true), block(block), fragment(fragment) {} }; struct ProcessComparisonState { std::vector* to_ignore = nullptr; std::vector equals_to; std::vector types; std::size_t heapsize = 0; void initHeapInformation(xbt_mheap_t heap, std::vector* i); }; namespace { /** A hash which works with more stuff * * It can hash pairs: the standard hash currently doesn't include this. */ template struct hash : public std::hash {}; template struct hash> { std::size_t operator()(std::pairconst& x) const { struct hash h1; struct hash h2; return h1(x.first) ^ h2(x.second); } }; } struct StateComparator { s_xbt_mheap_t std_heap_copy; std::size_t heaplimit; std::array processStates; std::unordered_set, hash>> compared_pointers; void clear() { compared_pointers.clear(); } int initHeapInformation( xbt_mheap_t heap1, xbt_mheap_t heap2, std::vector* i1, std::vector* i2); HeapArea& equals_to1_(std::size_t i, std::size_t j) { return processStates[0].equals_to[ MAX_FRAGMENT_PER_BLOCK * i + j]; } HeapArea& equals_to2_(std::size_t i, std::size_t j) { return processStates[1].equals_to[ MAX_FRAGMENT_PER_BLOCK * i + j]; } Type*& types1_(std::size_t i, std::size_t j) { return processStates[0].types[ MAX_FRAGMENT_PER_BLOCK * i + j]; } Type*& types2_(std::size_t i, std::size_t j) { return processStates[1].types[ MAX_FRAGMENT_PER_BLOCK * i + j]; } HeapArea const& equals_to1_(std::size_t i, std::size_t j) const { return processStates[0].equals_to[ MAX_FRAGMENT_PER_BLOCK * i + j]; } HeapArea const& equals_to2_(std::size_t i, std::size_t j) const { return processStates[1].equals_to[ MAX_FRAGMENT_PER_BLOCK * i + j]; } Type* const& types1_(std::size_t i, std::size_t j) const { return processStates[0].types[ MAX_FRAGMENT_PER_BLOCK * i + j]; } Type* const& types2_(std::size_t i, std::size_t j) const { return processStates[1].types[ MAX_FRAGMENT_PER_BLOCK * i + j]; } /** Check whether two blocks are known to be matching * * @param b1 Block of state 1 * @param b2 Block of state 2 * @return if the blocks are known to be matching */ bool blocksEqual(int b1, int b2) const { return this->equals_to1_(b1, 0).block == b2 && this->equals_to2_(b2, 0).block == b1; } /** Check whether two fragments are known to be matching * * @param b1 Block of state 1 * @param f1 Fragment of state 1 * @param b2 Block of state 2 * @param f2 Fragment of state 2 * @return if the fragments are known to be matching */ int fragmentsEqual(int b1, int f1, int b2, int f2) const { return this->equals_to1_(b1, f1).block == b2 && this->equals_to1_(b1, f1).fragment == f2 && this->equals_to2_(b2, f2).block == b1 && this->equals_to2_(b2, f2).fragment == f1; } void match_equals(HeapLocationPairs* list); }; } } /************************************************************************************/ static ssize_t heap_comparison_ignore_size( std::vector* ignore_list, const void *address) { int start = 0; int end = ignore_list->size() - 1; while (start <= end) { unsigned int cursor = (start + end) / 2; simgrid::mc::IgnoredHeapRegion const& region = (*ignore_list)[cursor]; if (region.address == address) return region.size; if (region.address < address) start = cursor + 1; if (region.address > address) end = cursor - 1; } return -1; } static bool is_stack(const void *address) { for (auto const& stack : mc_model_checker->process().stack_areas()) if (address == stack.address) return true; return false; } // TODO, this should depend on the snapshot? static bool is_block_stack(int block) { for (auto const& stack : mc_model_checker->process().stack_areas()) if (block == stack.block) return true; return false; } namespace simgrid { namespace mc { void StateComparator::match_equals(HeapLocationPairs* list) { for (auto const& pair : *list) { if (pair[0].fragment != -1) { this->equals_to1_(pair[0].block, pair[0].fragment) = simgrid::mc::HeapArea(pair[1].block, pair[1].fragment); this->equals_to2_(pair[1].block, pair[1].fragment) = simgrid::mc::HeapArea(pair[0].block, pair[0].fragment); } else { this->equals_to1_(pair[0].block, 0) = simgrid::mc::HeapArea(pair[1].block, pair[1].fragment); this->equals_to2_(pair[1].block, 0) = simgrid::mc::HeapArea(pair[0].block, pair[0].fragment); } } } void ProcessComparisonState::initHeapInformation(xbt_mheap_t heap, std::vector* i) { auto heaplimit = ((struct mdesc *) heap)->heaplimit; this->heapsize = ((struct mdesc *) heap)->heapsize; this->to_ignore = i; this->equals_to.assign(heaplimit * MAX_FRAGMENT_PER_BLOCK, HeapArea()); this->types.assign(heaplimit * MAX_FRAGMENT_PER_BLOCK, nullptr); } int StateComparator::initHeapInformation(xbt_mheap_t heap1, xbt_mheap_t heap2, std::vector* i1, std::vector* i2) { if ((((struct mdesc *) heap1)->heaplimit != ((struct mdesc *) heap2)->heaplimit) || ((((struct mdesc *) heap1)->heapsize != ((struct mdesc *) heap2)->heapsize))) return -1; this->heaplimit = ((struct mdesc *) heap1)->heaplimit; this->std_heap_copy = *mc_model_checker->process().get_heap(); this->processStates[0].initHeapInformation(heap1, i1); this->processStates[1].initHeapInformation(heap2, i2); return 0; } // TODO, have a robust way to find it in O(1) static inline mc_mem_region_t MC_get_heap_region(simgrid::mc::Snapshot* snapshot) { for (auto& region : snapshot->snapshot_regions) if (region->region_type() == simgrid::mc::RegionType::Heap) return region.get(); xbt_die("No heap region"); } static int mmalloc_compare_heap( simgrid::mc::StateComparator& state, simgrid::mc::Snapshot* snapshot1, simgrid::mc::Snapshot* snapshot2) { simgrid::mc::Process* process = &mc_model_checker->process(); /* Start comparison */ size_t i1, i2, j1, j2, k; void *addr_block1, *addr_block2, *addr_frag1, *addr_frag2; int nb_diff1 = 0, nb_diff2 = 0; int equal, res_compare = 0; /* Check busy blocks */ i1 = 1; malloc_info heapinfo_temp1, heapinfo_temp2; malloc_info heapinfo_temp2b; mc_mem_region_t heap_region1 = MC_get_heap_region(snapshot1); mc_mem_region_t heap_region2 = MC_get_heap_region(snapshot2); // This is the address of std_heap->heapinfo in the application process: void* heapinfo_address = &((xbt_mheap_t) process->heap_address)->heapinfo; // This is in snapshot do not use them directly: const malloc_info* heapinfos1 = snapshot1->read( (std::uint64_t)heapinfo_address, simgrid::mc::ProcessIndexMissing); const malloc_info* heapinfos2 = snapshot2->read( (std::uint64_t)heapinfo_address, simgrid::mc::ProcessIndexMissing); while (i1 < state.heaplimit) { const malloc_info* heapinfo1 = (const malloc_info*) MC_region_read(heap_region1, &heapinfo_temp1, &heapinfos1[i1], sizeof(malloc_info)); const malloc_info* heapinfo2 = (const malloc_info*) MC_region_read(heap_region2, &heapinfo_temp2, &heapinfos2[i1], sizeof(malloc_info)); if (heapinfo1->type == MMALLOC_TYPE_FREE || heapinfo1->type == MMALLOC_TYPE_HEAPINFO) { /* Free block */ i1 ++; continue; } if (heapinfo1->type < 0) { fprintf(stderr, "Unkown mmalloc block type.\n"); abort(); } addr_block1 = ((void *) (((ADDR2UINT(i1)) - 1) * BLOCKSIZE + (char *) state.std_heap_copy.heapbase)); if (heapinfo1->type == MMALLOC_TYPE_UNFRAGMENTED) { /* Large block */ if (is_stack(addr_block1)) { for (k = 0; k < heapinfo1->busy_block.size; k++) state.equals_to1_(i1 + k, 0) = HeapArea(i1, -1); for (k = 0; k < heapinfo2->busy_block.size; k++) state.equals_to2_(i1 + k, 0) = HeapArea(i1, -1); i1 += heapinfo1->busy_block.size; continue; } if (state.equals_to1_(i1, 0).valid) { i1++; continue; } i2 = 1; equal = 0; res_compare = 0; /* Try first to associate to same block in the other heap */ if (heapinfo2->type == heapinfo1->type && state.equals_to2_(i1, 0).valid == 0) { addr_block2 = (ADDR2UINT(i1) - 1) * BLOCKSIZE + (char *) state.std_heap_copy.heapbase; res_compare = compare_heap_area(state, simgrid::mc::ProcessIndexMissing, addr_block1, addr_block2, snapshot1, snapshot2, nullptr, nullptr, 0); if (res_compare != 1) { for (k = 1; k < heapinfo2->busy_block.size; k++) state.equals_to2_(i1 + k, 0) = HeapArea(i1, -1); for (k = 1; k < heapinfo1->busy_block.size; k++) state.equals_to1_(i1 + k, 0) = HeapArea(i1, -1); equal = 1; i1 += heapinfo1->busy_block.size; } } while (i2 < state.heaplimit && !equal) { addr_block2 = (ADDR2UINT(i2) - 1) * BLOCKSIZE + (char *) state.std_heap_copy.heapbase; if (i2 == i1) { i2++; continue; } const malloc_info* heapinfo2b = (const malloc_info*) MC_region_read(heap_region2, &heapinfo_temp2b, &heapinfos2[i2], sizeof(malloc_info)); if (heapinfo2b->type != MMALLOC_TYPE_UNFRAGMENTED) { i2++; continue; } if (state.equals_to2_(i2, 0).valid) { i2++; continue; } res_compare = compare_heap_area(state, simgrid::mc::ProcessIndexMissing, addr_block1, addr_block2, snapshot1, snapshot2, nullptr, nullptr, 0); if (res_compare != 1) { for (k = 1; k < heapinfo2b->busy_block.size; k++) state.equals_to2_(i2 + k, 0) = HeapArea(i1, -1); for (k = 1; k < heapinfo1->busy_block.size; k++) state.equals_to1_(i1 + k, 0) = HeapArea(i2, -1); equal = 1; i1 += heapinfo1->busy_block.size; } i2++; } if (!equal) { XBT_DEBUG("Block %zu not found (size_used = %zu, addr = %p)", i1, heapinfo1->busy_block.busy_size, addr_block1); i1 = state.heaplimit + 1; nb_diff1++; //i1++; } } else { /* Fragmented block */ for (j1 = 0; j1 < (size_t) (BLOCKSIZE >> heapinfo1->type); j1++) { if (heapinfo1->busy_frag.frag_size[j1] == -1) /* Free fragment */ continue; if (state.equals_to1_(i1, j1).valid) continue; addr_frag1 = (void *) ((char *) addr_block1 + (j1 << heapinfo1->type)); i2 = 1; equal = 0; /* Try first to associate to same fragment in the other heap */ if (heapinfo2->type == heapinfo1->type && !state.equals_to2_(i1, j1).valid) { addr_block2 = (ADDR2UINT(i1) - 1) * BLOCKSIZE + (char *) state.std_heap_copy.heapbase; addr_frag2 = (void *) ((char *) addr_block2 + (j1 << heapinfo2->type)); res_compare = compare_heap_area(state, simgrid::mc::ProcessIndexMissing, addr_frag1, addr_frag2, snapshot1, snapshot2, nullptr, nullptr, 0); if (res_compare != 1) equal = 1; } while (i2 < state.heaplimit && !equal) { const malloc_info* heapinfo2b = (const malloc_info*) MC_region_read( heap_region2, &heapinfo_temp2b, &heapinfos2[i2], sizeof(malloc_info)); if (heapinfo2b->type == MMALLOC_TYPE_FREE || heapinfo2b->type == MMALLOC_TYPE_HEAPINFO) { i2 ++; continue; } // We currently do not match fragments with unfragmented blocks (maybe we should). if (heapinfo2b->type == MMALLOC_TYPE_UNFRAGMENTED) { i2++; continue; } if (heapinfo2b->type < 0) { fprintf(stderr, "Unkown mmalloc block type.\n"); abort(); } for (j2 = 0; j2 < (size_t) (BLOCKSIZE >> heapinfo2b->type); j2++) { if (i2 == i1 && j2 == j1) continue; if (state.equals_to2_(i2, j2).valid) continue; addr_block2 = (ADDR2UINT(i2) - 1) * BLOCKSIZE + (char *) state.std_heap_copy.heapbase; addr_frag2 = (void *) ((char *) addr_block2 + (j2 << heapinfo2b->type)); res_compare = compare_heap_area( state, simgrid::mc::ProcessIndexMissing, addr_frag1, addr_frag2, snapshot2, snapshot2, nullptr, nullptr, 0); if (res_compare != 1) { equal = 1; break; } } i2++; } if (!equal) { XBT_DEBUG ("Block %zu, fragment %zu not found (size_used = %zd, address = %p)\n", i1, j1, heapinfo1->busy_frag.frag_size[j1], addr_frag1); i2 = state.heaplimit + 1; i1 = state.heaplimit + 1; nb_diff1++; break; } } i1++; } } /* All blocks/fragments are equal to another block/fragment ? */ size_t i = 1, j = 0; for(i = 1; i < state.heaplimit; i++) { const malloc_info* heapinfo1 = (const malloc_info*) MC_region_read( heap_region1, &heapinfo_temp1, &heapinfos1[i], sizeof(malloc_info)); if (heapinfo1->type == MMALLOC_TYPE_UNFRAGMENTED && i1 == state.heaplimit && heapinfo1->busy_block.busy_size > 0 && !state.equals_to1_(i, 0).valid) { XBT_DEBUG("Block %zu not found (size used = %zu)", i, heapinfo1->busy_block.busy_size); nb_diff1++; } if (heapinfo1->type <= 0) continue; for (j = 0; j < (size_t) (BLOCKSIZE >> heapinfo1->type); j++) if (i1 == state.heaplimit && heapinfo1->busy_frag.frag_size[j] > 0 && !state.equals_to1_(i, j).valid) { XBT_DEBUG("Block %zu, Fragment %zu not found (size used = %zd)", i, j, heapinfo1->busy_frag.frag_size[j]); nb_diff1++; } } if (i1 == state.heaplimit) XBT_DEBUG("Number of blocks/fragments not found in heap1 : %d", nb_diff1); for (i=1; i < state.heaplimit; i++) { const malloc_info* heapinfo2 = (const malloc_info*) MC_region_read( heap_region2, &heapinfo_temp2, &heapinfos2[i], sizeof(malloc_info)); if (heapinfo2->type == MMALLOC_TYPE_UNFRAGMENTED && i1 == state.heaplimit && heapinfo2->busy_block.busy_size > 0 && !state.equals_to2_(i, 0).valid) { XBT_DEBUG("Block %zu not found (size used = %zu)", i, heapinfo2->busy_block.busy_size); nb_diff2++; } if (heapinfo2->type <= 0) continue; for (j = 0; j < (size_t) (BLOCKSIZE >> heapinfo2->type); j++) if (i1 == state.heaplimit && heapinfo2->busy_frag.frag_size[j] > 0 && !state.equals_to2_(i, j).valid) { XBT_DEBUG("Block %zu, Fragment %zu not found (size used = %zd)", i, j, heapinfo2->busy_frag.frag_size[j]); nb_diff2++; } } if (i1 == state.heaplimit) XBT_DEBUG("Number of blocks/fragments not found in heap2 : %d", nb_diff2); return nb_diff1 > 0 || nb_diff2 > 0; } /** * * @param state * @param real_area1 Process address for state 1 * @param real_area2 Process address for state 2 * @param snapshot1 Snapshot of state 1 * @param snapshot2 Snapshot of state 2 * @param previous * @param size * @param check_ignore */ static int compare_heap_area_without_type( simgrid::mc::StateComparator& state, int process_index, const void *real_area1, const void *real_area2, simgrid::mc::Snapshot* snapshot1, simgrid::mc::Snapshot* snapshot2, HeapLocationPairs* previous, int size, int check_ignore) { simgrid::mc::Process* process = &mc_model_checker->process(); mc_mem_region_t heap_region1 = MC_get_heap_region(snapshot1); mc_mem_region_t heap_region2 = MC_get_heap_region(snapshot2); for (int i = 0; i < size; ) { if (check_ignore > 0) { ssize_t ignore1 = heap_comparison_ignore_size( state.processStates[0].to_ignore, (char *) real_area1 + i); if (ignore1 != -1) { ssize_t ignore2 = heap_comparison_ignore_size( state.processStates[1].to_ignore, (char *) real_area2 + i); if (ignore2 == ignore1) { if (ignore1 == 0) { check_ignore--; return 0; } else { i = i + ignore2; check_ignore--; continue; } } } } if (MC_snapshot_region_memcmp(((char *) real_area1) + i, heap_region1, ((char *) real_area2) + i, heap_region2, 1) != 0) { int pointer_align = (i / sizeof(void *)) * sizeof(void *); const void* addr_pointed1 = snapshot1->read( remote((void**)((char *) real_area1 + pointer_align)), process_index); const void* addr_pointed2 = snapshot2->read( remote((void**)((char *) real_area2 + pointer_align)), process_index); if (process->in_maestro_stack(remote(addr_pointed1)) && process->in_maestro_stack(remote(addr_pointed2))) { i = pointer_align + sizeof(void *); continue; } if (addr_pointed1 > state.std_heap_copy.heapbase && addr_pointed1 < mc_snapshot_get_heap_end(snapshot1) && addr_pointed2 > state.std_heap_copy.heapbase && addr_pointed2 < mc_snapshot_get_heap_end(snapshot2)) { // Both addreses are in the heap: int res_compare = compare_heap_area(state ,process_index, addr_pointed1, addr_pointed2, snapshot1, snapshot2, previous, nullptr, 0); if (res_compare == 1) return res_compare; i = pointer_align + sizeof(void *); continue; } return 1; } i++; } return 0; } /** * * @param state * @param real_area1 Process address for state 1 * @param real_area2 Process address for state 2 * @param snapshot1 Snapshot of state 1 * @param snapshot2 Snapshot of state 2 * @param previous * @param type_id * @param area_size either a byte_size or an elements_count (?) * @param check_ignore * @param pointer_level * @return 0 (same), 1 (different), -1 (unknown) */ static int compare_heap_area_with_type( simgrid::mc::StateComparator& state, int process_index, const void *real_area1, const void *real_area2, simgrid::mc::Snapshot* snapshot1, simgrid::mc::Snapshot* snapshot2, HeapLocationPairs* previous, simgrid::mc::Type* type, int area_size, int check_ignore, int pointer_level) { top: // HACK: This should not happen but in pratice, there are some // DW_TAG_typedef without an associated DW_AT_type: //<1><538832>: Abbrev Number: 111 (DW_TAG_typedef) // <538833> DW_AT_name : (indirect string, offset: 0x2292f3): gregset_t // <538837> DW_AT_decl_file : 98 // <538838> DW_AT_decl_line : 37 if (type == nullptr) return 0; if (is_stack(real_area1) && is_stack(real_area2)) return 0; if (check_ignore > 0) { ssize_t ignore1 = heap_comparison_ignore_size( state.processStates[0].to_ignore, real_area1); if (ignore1 > 0 && heap_comparison_ignore_size( state.processStates[1].to_ignore, real_area2) == ignore1) return 0; } simgrid::mc::Type *subtype, *subsubtype; int res, elm_size; const void *addr_pointed1, *addr_pointed2; mc_mem_region_t heap_region1 = MC_get_heap_region(snapshot1); mc_mem_region_t heap_region2 = MC_get_heap_region(snapshot2); switch (type->type) { case DW_TAG_unspecified_type: return 1; case DW_TAG_base_type: if (!type->name.empty() && type->name == "char") { /* String, hence random (arbitrary ?) size */ if (real_area1 == real_area2) return -1; else return MC_snapshot_region_memcmp(real_area1, heap_region1, real_area2, heap_region2, area_size) != 0; } else { if (area_size != -1 && type->byte_size != area_size) return -1; else return MC_snapshot_region_memcmp(real_area1, heap_region1, real_area2, heap_region2, type->byte_size) != 0; } break; case DW_TAG_enumeration_type: if (area_size != -1 && type->byte_size != area_size) return -1; return MC_snapshot_region_memcmp(real_area1, heap_region1, real_area2, heap_region2, type->byte_size) != 0; case DW_TAG_typedef: case DW_TAG_const_type: case DW_TAG_volatile_type: // Poor man's TCO: type = type->subtype; goto top; case DW_TAG_array_type: subtype = type->subtype; switch (subtype->type) { case DW_TAG_unspecified_type: return 1; case DW_TAG_base_type: case DW_TAG_enumeration_type: case DW_TAG_pointer_type: case DW_TAG_reference_type: case DW_TAG_rvalue_reference_type: case DW_TAG_structure_type: case DW_TAG_class_type: case DW_TAG_union_type: if (subtype->full_type) subtype = subtype->full_type; elm_size = subtype->byte_size; break; // TODO, just remove the type indirection? case DW_TAG_const_type: case DW_TAG_typedef: case DW_TAG_volatile_type: subsubtype = subtype->subtype; if (subsubtype->full_type) subsubtype = subsubtype->full_type; elm_size = subsubtype->byte_size; break; default: return 0; break; } for (int i = 0; i < type->element_count; i++) { // TODO, add support for variable stride (DW_AT_byte_stride) res = compare_heap_area_with_type(state, process_index, (char *) real_area1 + (i * elm_size), (char *) real_area2 + (i * elm_size), snapshot1, snapshot2, previous, type->subtype, subtype->byte_size, check_ignore, pointer_level); if (res == 1) return res; } return 0; case DW_TAG_reference_type: case DW_TAG_rvalue_reference_type: case DW_TAG_pointer_type: if (type->subtype && type->subtype->type == DW_TAG_subroutine_type) { addr_pointed1 = snapshot1->read(remote((void**)real_area1), process_index); addr_pointed2 = snapshot2->read(remote((void**)real_area2), process_index); return (addr_pointed1 != addr_pointed2); } pointer_level++; if (pointer_level <= 1) { addr_pointed1 = snapshot1->read(remote((void**)real_area1), process_index); addr_pointed2 = snapshot2->read(remote((void**)real_area2), process_index); if (addr_pointed1 > state.std_heap_copy.heapbase && addr_pointed1 < mc_snapshot_get_heap_end(snapshot1) && addr_pointed2 > state.std_heap_copy.heapbase && addr_pointed2 < mc_snapshot_get_heap_end(snapshot2)) return compare_heap_area(state, process_index, addr_pointed1, addr_pointed2, snapshot1, snapshot2, previous, type->subtype, pointer_level); else return (addr_pointed1 != addr_pointed2); } for (size_t i = 0; i < (area_size / sizeof(void *)); i++) { addr_pointed1 = snapshot1->read( remote((void**)((char*) real_area1 + i * sizeof(void *))), process_index); addr_pointed2 = snapshot2->read( remote((void**)((char*) real_area2 + i * sizeof(void *))), process_index); if (addr_pointed1 > state.std_heap_copy.heapbase && addr_pointed1 < mc_snapshot_get_heap_end(snapshot1) && addr_pointed2 > state.std_heap_copy.heapbase && addr_pointed2 < mc_snapshot_get_heap_end(snapshot2)) res = compare_heap_area(state, process_index, addr_pointed1, addr_pointed2, snapshot1, snapshot2, previous, type->subtype, pointer_level); else res = (addr_pointed1 != addr_pointed2); if (res == 1) return res; } return 0; case DW_TAG_structure_type: case DW_TAG_class_type: if (type->full_type) type = type->full_type; if (area_size != -1 && type->byte_size != area_size) { if (area_size <= type->byte_size || area_size % type->byte_size != 0) return -1; for (size_t i = 0; i < (size_t)(area_size / type->byte_size); i++) { int res = compare_heap_area_with_type(state, process_index, (char *) real_area1 + i * type->byte_size, (char *) real_area2 + i * type->byte_size, snapshot1, snapshot2, previous, type, -1, check_ignore, 0); if (res == 1) return res; } } else { for(simgrid::mc::Member& member : type->members) { // TODO, optimize this? (for the offset case) void *real_member1 = simgrid::dwarf::resolve_member( real_area1, type, &member, (simgrid::mc::AddressSpace*) snapshot1, process_index); void *real_member2 = simgrid::dwarf::resolve_member( real_area2, type, &member, (simgrid::mc::AddressSpace*) snapshot2, process_index); int res = compare_heap_area_with_type( state, process_index, real_member1, real_member2, snapshot1, snapshot2, previous, member.type, -1, check_ignore, 0); if (res == 1) return res; } } return 0; case DW_TAG_union_type: return compare_heap_area_without_type(state, process_index, real_area1, real_area2, snapshot1, snapshot2, previous, type->byte_size, check_ignore); return 0; default: return 0; } xbt_die("Unreachable"); } /** Infer the type of a part of the block from the type of the block * * TODO, handle DW_TAG_array_type as well as arrays of the object ((*p)[5], p[5]) * * TODO, handle subfields ((*p).bar.foo, (*p)[5].bar…) * * @param type_id DWARF type ID of the root address * @param area_size * @return DWARF type ID for given offset */ static simgrid::mc::Type* get_offset_type(void *real_base_address, simgrid::mc::Type* type, int offset, int area_size, simgrid::mc::Snapshot* snapshot, int process_index) { // Beginning of the block, the infered variable type if the type of the block: if (offset == 0) return type; switch (type->type) { case DW_TAG_structure_type: case DW_TAG_class_type: if (type->full_type) type = type->full_type; if (area_size != -1 && type->byte_size != area_size) { if (area_size > type->byte_size && area_size % type->byte_size == 0) return type; else return nullptr; } for(simgrid::mc::Member& member : type->members) { if (member.has_offset_location()) { // We have the offset, use it directly (shortcut): if (member.offset() == offset) return member.type; } else { void *real_member = simgrid::dwarf::resolve_member( real_base_address, type, &member, snapshot, process_index); if ((char*) real_member - (char *) real_base_address == offset) return member.type; } } return nullptr; default: /* FIXME : other cases ? */ return nullptr; } } /** * * @param area1 Process address for state 1 * @param area2 Process address for state 2 * @param snapshot1 Snapshot of state 1 * @param snapshot2 Snapshot of state 2 * @param previous Pairs of blocks already compared on the current path (or nullptr) * @param type_id Type of variable * @param pointer_level * @return 0 (same), 1 (different), -1 */ static int compare_heap_area(simgrid::mc::StateComparator& state, int process_index, const void *area1, const void *area2, simgrid::mc::Snapshot* snapshot1, simgrid::mc::Snapshot* snapshot2, HeapLocationPairs* previous, simgrid::mc::Type* type, int pointer_level) { simgrid::mc::Process* process = &mc_model_checker->process(); int res_compare; ssize_t block1, frag1, block2, frag2; ssize_t size; int check_ignore = 0; void *real_addr_block1, *real_addr_block2, *real_addr_frag1, *real_addr_frag2; int type_size = -1; int offset1 = 0, offset2 = 0; int new_size1 = -1, new_size2 = -1; simgrid::mc::Type *new_type1 = nullptr, *new_type2 = nullptr; bool match_pairs = false; // This is the address of std_heap->heapinfo in the application process: void* heapinfo_address = &((xbt_mheap_t) process->heap_address)->heapinfo; const malloc_info* heapinfos1 = snapshot1->read( remote((const malloc_info**)heapinfo_address), process_index); const malloc_info* heapinfos2 = snapshot2->read( remote((const malloc_info**)heapinfo_address), process_index); malloc_info heapinfo_temp1, heapinfo_temp2; simgrid::mc::HeapLocationPairs current; if (previous == nullptr) { previous = ¤t; match_pairs = true; } // Get block number: block1 = ((char *) area1 - (char *) state.std_heap_copy.heapbase) / BLOCKSIZE + 1; block2 = ((char *) area2 - (char *) state.std_heap_copy.heapbase) / BLOCKSIZE + 1; // If either block is a stack block: if (is_block_stack((int) block1) && is_block_stack((int) block2)) { previous->insert(simgrid::mc::makeHeapLocationPair( block1, -1, block2, -1)); if (match_pairs) state.match_equals(previous); return 0; } // If either block is not in the expected area of memory: if (((char *) area1 < (char *) state.std_heap_copy.heapbase) || (block1 > (ssize_t) state.processStates[0].heapsize) || (block1 < 1) || ((char *) area2 < (char *) state.std_heap_copy.heapbase) || (block2 > (ssize_t) state.processStates[1].heapsize) || (block2 < 1)) { return 1; } // Process address of the block: real_addr_block1 = (ADDR2UINT(block1) - 1) * BLOCKSIZE + (char *) state.std_heap_copy.heapbase; real_addr_block2 = (ADDR2UINT(block2) - 1) * BLOCKSIZE + (char *) state.std_heap_copy.heapbase; if (type) { if (type->full_type) type = type->full_type; // This assume that for "boring" types (volatile ...) byte_size is absent: while (type->byte_size == 0 && type->subtype != nullptr) type = type->subtype; // Find type_size: if (type->type == DW_TAG_pointer_type || (type->type == DW_TAG_base_type && !type->name.empty() && type->name == "char")) type_size = -1; else type_size = type->byte_size; } mc_mem_region_t heap_region1 = MC_get_heap_region(snapshot1); mc_mem_region_t heap_region2 = MC_get_heap_region(snapshot2); const malloc_info* heapinfo1 = (const malloc_info*) MC_region_read( heap_region1, &heapinfo_temp1, &heapinfos1[block1], sizeof(malloc_info)); const malloc_info* heapinfo2 = (const malloc_info*) MC_region_read( heap_region2, &heapinfo_temp2, &heapinfos2[block2], sizeof(malloc_info)); if ((heapinfo1->type == MMALLOC_TYPE_FREE || heapinfo1->type==MMALLOC_TYPE_HEAPINFO) && (heapinfo2->type == MMALLOC_TYPE_FREE || heapinfo2->type ==MMALLOC_TYPE_HEAPINFO)) { /* Free block */ if (match_pairs) state.match_equals(previous); return 0; } if (heapinfo1->type == MMALLOC_TYPE_UNFRAGMENTED && heapinfo2->type == MMALLOC_TYPE_UNFRAGMENTED) { /* Complete block */ // TODO, lookup variable type from block type as done for fragmented blocks offset1 = (char *) area1 - (char *) real_addr_block1; offset2 = (char *) area2 - (char *) real_addr_block2; if (state.equals_to1_(block1, 0).valid && state.equals_to2_(block2, 0).valid && state.blocksEqual(block1, block2)) { if (match_pairs) state.match_equals(previous); return 0; } if (type_size != -1) { if (type_size != (ssize_t) heapinfo1->busy_block.busy_size && type_size != (ssize_t) heapinfo2->busy_block.busy_size && (type->name.empty() || type->name == "struct s_smx_context")) { if (match_pairs) state.match_equals(previous); return -1; } } if (heapinfo1->busy_block.size != heapinfo2->busy_block.size) return 1; if (heapinfo1->busy_block.busy_size != heapinfo2->busy_block.busy_size) return 1; if (!previous->insert(simgrid::mc::makeHeapLocationPair( block1, -1, block2, -1)).second) { if (match_pairs) state.match_equals(previous); return 0; } size = heapinfo1->busy_block.busy_size; // Remember (basic) type inference. // The current data structure only allows us to do this for the whole block. if (type != nullptr && area1 == real_addr_block1) state.types1_(block1, 0) = type; if (type != nullptr && area2 == real_addr_block2) state.types2_(block2, 0) = type; if (size <= 0) { if (match_pairs) state.match_equals(previous); return 0; } frag1 = -1; frag2 = -1; if (heapinfo1->busy_block.ignore > 0 && heapinfo2->busy_block.ignore == heapinfo1->busy_block.ignore) check_ignore = heapinfo1->busy_block.ignore; } else if ((heapinfo1->type > 0) && (heapinfo2->type > 0)) { /* Fragmented block */ // Fragment number: frag1 = ((uintptr_t) (ADDR2UINT(area1) % (BLOCKSIZE))) >> heapinfo1->type; frag2 = ((uintptr_t) (ADDR2UINT(area2) % (BLOCKSIZE))) >> heapinfo2->type; // Process address of the fragment: real_addr_frag1 = (void *) ((char *) real_addr_block1 + (frag1 << heapinfo1->type)); real_addr_frag2 = (void *) ((char *) real_addr_block2 + (frag2 << heapinfo2->type)); // Check the size of the fragments against the size of the type: if (type_size != -1) { if (heapinfo1->busy_frag.frag_size[frag1] == -1 || heapinfo2->busy_frag.frag_size[frag2] == -1) { if (match_pairs) state.match_equals(previous); return -1; } // ? if (type_size != heapinfo1->busy_frag.frag_size[frag1] || type_size != heapinfo2->busy_frag.frag_size[frag2]) { if (match_pairs) state.match_equals(previous); return -1; } } // Check if the blocks are already matched together: if (state.equals_to1_(block1, frag1).valid && state.equals_to2_(block2, frag2).valid) { if (offset1==offset2 && state.fragmentsEqual(block1, frag1, block2, frag2)) { if (match_pairs) state.match_equals(previous); return 0; } } // Compare the size of both fragments: if (heapinfo1->busy_frag.frag_size[frag1] != heapinfo2->busy_frag.frag_size[frag2]) { if (type_size == -1) { if (match_pairs) state.match_equals(previous); return -1; } else return 1; } // Size of the fragment: size = heapinfo1->busy_frag.frag_size[frag1]; // Remember (basic) type inference. // The current data structure only allows us to do this for the whole fragment. if (type != nullptr && area1 == real_addr_frag1) state.types1_(block1, frag1) = type; if (type != nullptr && area2 == real_addr_frag2) state.types2_(block2, frag2) = type; // The type of the variable is already known: if (type) { new_type1 = type; new_type2 = type; } // Type inference from the block type. else if (state.types1_(block1, frag1) != nullptr || state.types2_(block2, frag2) != nullptr) { offset1 = (char *) area1 - (char *) real_addr_frag1; offset2 = (char *) area2 - (char *) real_addr_frag2; if (state.types1_(block1, frag1) != nullptr && state.types2_(block2, frag2) != nullptr) { new_type1 = get_offset_type(real_addr_frag1, state.types1_(block1, frag1), offset1, size, snapshot1, process_index); new_type2 = get_offset_type(real_addr_frag2, state.types2_(block2, frag2), offset1, size, snapshot2, process_index); } else if (state.types1_(block1, frag1) != nullptr) { new_type1 = get_offset_type(real_addr_frag1, state.types1_(block1, frag1), offset1, size, snapshot1, process_index); new_type2 = get_offset_type(real_addr_frag2, state.types1_(block1, frag1), offset2, size, snapshot2, process_index); } else if (state.types2_(block2, frag2) != nullptr) { new_type1 = get_offset_type(real_addr_frag1, state.types2_(block2, frag2), offset1, size, snapshot1, process_index); new_type2 = get_offset_type(real_addr_frag2, state.types2_(block2, frag2), offset2, size, snapshot2, process_index); } else { if (match_pairs) state.match_equals(previous); return -1; } if (new_type1 != nullptr && new_type2 != nullptr && new_type1 != new_type2) { type = new_type1; while (type->byte_size == 0 && type->subtype != nullptr) type = type->subtype; new_size1 = type->byte_size; type = new_type2; while (type->byte_size == 0 && type->subtype != nullptr) type = type->subtype; new_size2 = type->byte_size; } else { if (match_pairs) state.match_equals(previous); return -1; } } if (new_size1 > 0 && new_size1 == new_size2) { type = new_type1; size = new_size1; } if (offset1 == 0 && offset2 == 0 && !previous->insert(simgrid::mc::makeHeapLocationPair( block1, frag1, block2, frag2)).second) { if (match_pairs) state.match_equals(previous); return 0; } if (size <= 0) { if (match_pairs) state.match_equals(previous); return 0; } if ((heapinfo1->busy_frag.ignore[frag1] > 0) && (heapinfo2->busy_frag.ignore[frag2] == heapinfo1->busy_frag.ignore[frag1])) check_ignore = heapinfo1->busy_frag.ignore[frag1]; } else return 1; /* Start comparison */ if (type) res_compare = compare_heap_area_with_type(state, process_index, area1, area2, snapshot1, snapshot2, previous, type, size, check_ignore, pointer_level); else res_compare = compare_heap_area_without_type(state, process_index, area1, area2, snapshot1, snapshot2, previous, size, check_ignore); if (res_compare == 1) return res_compare; if (match_pairs) state.match_equals(previous); return 0; } } } /************************** Snapshot comparison *******************************/ /******************************************************************************/ static int compare_areas_with_type(simgrid::mc::StateComparator& state, int process_index, void* real_area1, simgrid::mc::Snapshot* snapshot1, mc_mem_region_t region1, void* real_area2, simgrid::mc::Snapshot* snapshot2, mc_mem_region_t region2, simgrid::mc::Type* type, int pointer_level) { simgrid::mc::Process* process = &mc_model_checker->process(); simgrid::mc::Type* subtype; simgrid::mc::Type* subsubtype; int elm_size, i, res; top: switch (type->type) { case DW_TAG_unspecified_type: return 1; case DW_TAG_base_type: case DW_TAG_enumeration_type: case DW_TAG_union_type: { return MC_snapshot_region_memcmp( real_area1, region1, real_area2, region2, type->byte_size) != 0; } case DW_TAG_typedef: case DW_TAG_volatile_type: case DW_TAG_const_type: // Poor man's TCO: type = type->subtype; goto top; case DW_TAG_array_type: subtype = type->subtype; switch (subtype->type) { case DW_TAG_unspecified_type: return 1; case DW_TAG_base_type: case DW_TAG_enumeration_type: case DW_TAG_pointer_type: case DW_TAG_reference_type: case DW_TAG_rvalue_reference_type: case DW_TAG_structure_type: case DW_TAG_class_type: case DW_TAG_union_type: if (subtype->full_type) subtype = subtype->full_type; elm_size = subtype->byte_size; break; case DW_TAG_const_type: case DW_TAG_typedef: case DW_TAG_volatile_type: subsubtype = subtype->subtype; if (subsubtype->full_type) subsubtype = subsubtype->full_type; elm_size = subsubtype->byte_size; break; default: return 0; break; } for (i = 0; i < type->element_count; i++) { size_t off = i * elm_size; res = compare_areas_with_type(state, process_index, (char*) real_area1 + off, snapshot1, region1, (char*) real_area2 + off, snapshot2, region2, type->subtype, pointer_level); if (res == 1) return res; } break; case DW_TAG_pointer_type: case DW_TAG_reference_type: case DW_TAG_rvalue_reference_type: { void* addr_pointed1 = MC_region_read_pointer(region1, real_area1); void* addr_pointed2 = MC_region_read_pointer(region2, real_area2); if (type->subtype && type->subtype->type == DW_TAG_subroutine_type) return (addr_pointed1 != addr_pointed2); if (addr_pointed1 == nullptr && addr_pointed2 == nullptr) return 0; if (addr_pointed1 == nullptr || addr_pointed2 == nullptr) return 1; if (!state.compared_pointers.insert( std::make_pair(addr_pointed1, addr_pointed2)).second) return 0; pointer_level++; // Some cases are not handled here: // * the pointers lead to different areas (one to the heap, the other to the RW segment ...); // * a pointer leads to the read-only segment of the current object; // * a pointer lead to a different ELF object. if (addr_pointed1 > process->heap_address && addr_pointed1 < mc_snapshot_get_heap_end(snapshot1)) { if (! (addr_pointed2 > process->heap_address && addr_pointed2 < mc_snapshot_get_heap_end(snapshot2))) return 1; // The pointers are both in the heap: return simgrid::mc::compare_heap_area(state, process_index, addr_pointed1, addr_pointed2, snapshot1, snapshot2, nullptr, type->subtype, pointer_level); } // The pointers are both in the current object R/W segment: else if (region1->contain(simgrid::mc::remote(addr_pointed1))) { if (!region2->contain(simgrid::mc::remote(addr_pointed2))) return 1; if (!type->type_id) return (addr_pointed1 != addr_pointed2); else return compare_areas_with_type(state, process_index, addr_pointed1, snapshot1, region1, addr_pointed2, snapshot2, region2, type->subtype, pointer_level); } // TODO, We do not handle very well the case where // it belongs to a different (non-heap) region from the current one. else return (addr_pointed1 != addr_pointed2); break; } case DW_TAG_structure_type: case DW_TAG_class_type: for(simgrid::mc::Member& member : type->members) { void *member1 = simgrid::dwarf::resolve_member( real_area1, type, &member, snapshot1, process_index); void *member2 = simgrid::dwarf::resolve_member( real_area2, type, &member, snapshot2, process_index); mc_mem_region_t subregion1 = mc_get_region_hinted(member1, snapshot1, process_index, region1); mc_mem_region_t subregion2 = mc_get_region_hinted(member2, snapshot2, process_index, region2); res = compare_areas_with_type(state, process_index, member1, snapshot1, subregion1, member2, snapshot2, subregion2, member.type, pointer_level); if (res == 1) return res; } break; case DW_TAG_subroutine_type: return -1; break; default: XBT_VERB("Unknown case : %d", type->type); break; } return 0; } static int compare_global_variables( simgrid::mc::StateComparator& state, simgrid::mc::ObjectInformation* object_info, int process_index, mc_mem_region_t r1, mc_mem_region_t r2, simgrid::mc::Snapshot* snapshot1, simgrid::mc::Snapshot* snapshot2) { xbt_assert(r1 && r2, "Missing region."); #if HAVE_SMPI if (r1->storage_type() == simgrid::mc::StorageType::Privatized) { xbt_assert(process_index >= 0); if (r2->storage_type() != simgrid::mc::StorageType::Privatized) return 1; size_t process_count = MC_smpi_process_count(); xbt_assert(process_count == r1->privatized_data().size() && process_count == r2->privatized_data().size()); // Compare the global variables separately for each simulates process: for (size_t process_index = 0; process_index < process_count; process_index++) { int is_diff = compare_global_variables(state, object_info, process_index, &r1->privatized_data()[process_index], &r2->privatized_data()[process_index], snapshot1, snapshot2); if (is_diff) return 1; } return 0; } #else xbt_assert(r1->storage_type() != simgrid::mc::StorageType::Privatized); #endif xbt_assert(r2->storage_type() != simgrid::mc::StorageType::Privatized); std::vector& variables = object_info->global_variables; for (simgrid::mc::Variable& current_var : variables) { // If the variable is not in this object, skip it: // We do not expect to find a pointer to something which is not reachable // by the global variables. if ((char *) current_var.address < (char *) object_info->start_rw || (char *) current_var.address > (char *) object_info->end_rw) continue; simgrid::mc::Type* bvariable_type = current_var.type; int res = compare_areas_with_type(state, process_index, (char *) current_var.address, snapshot1, r1, (char *) current_var.address, snapshot2, r2, bvariable_type, 0); if (res == 1) { XBT_VERB("Global variable %s (%p) is different between snapshots", current_var.name.c_str(), (char *) current_var.address); return 1; } } return 0; } static int compare_local_variables(simgrid::mc::StateComparator& state, int process_index, simgrid::mc::Snapshot* snapshot1, simgrid::mc::Snapshot* snapshot2, mc_snapshot_stack_t stack1, mc_snapshot_stack_t stack2) { if (stack1->local_variables.size() != stack2->local_variables.size()) { XBT_VERB("Different number of local variables"); return 1; } unsigned int cursor = 0; local_variable_t current_var1, current_var2; int res; while (cursor < stack1->local_variables.size()) { current_var1 = &stack1->local_variables[cursor]; current_var2 = &stack1->local_variables[cursor]; if (current_var1->name != current_var2->name || current_var1->subprogram != current_var2->subprogram || current_var1->ip != current_var2->ip) { // TODO, fix current_varX->subprogram->name to include name if DW_TAG_inlined_subprogram XBT_VERB ("Different name of variable (%s - %s) " "or frame (%s - %s) or ip (%lu - %lu)", current_var1->name.c_str(), current_var2->name.c_str(), current_var1->subprogram->name.c_str(), current_var2->subprogram->name.c_str(), current_var1->ip, current_var2->ip); return 1; } // TODO, fix current_varX->subprogram->name to include name if DW_TAG_inlined_subprogram simgrid::mc::Type* subtype = current_var1->type; res = compare_areas_with_type(state, process_index, current_var1->address, snapshot1, mc_get_snapshot_region(current_var1->address, snapshot1, process_index), current_var2->address, snapshot2, mc_get_snapshot_region(current_var2->address, snapshot2, process_index), subtype, 0); if (res == 1) { // TODO, fix current_varX->subprogram->name to include name if DW_TAG_inlined_subprogram XBT_VERB ("Local variable %s (%p - %p) in frame %s " "is different between snapshots", current_var1->name.c_str(), current_var1->address, current_var2->address, current_var1->subprogram->name.c_str()); return res; } cursor++; } return 0; } namespace simgrid { namespace mc { static std::unique_ptr state_comparator; int snapshot_compare(int num1, simgrid::mc::Snapshot* s1, int num2, simgrid::mc::Snapshot* s2) { // TODO, make this a field of ModelChecker or something similar if (state_comparator == nullptr) state_comparator = std::unique_ptr(new StateComparator()); else state_comparator->clear(); simgrid::mc::Process* process = &mc_model_checker->process(); int errors = 0; int res_init; int hash_result = 0; if (_sg_mc_hash) { hash_result = (s1->hash != s2->hash); if (hash_result) { XBT_VERB("(%d - %d) Different hash : 0x%" PRIx64 "--0x%" PRIx64, num1, num2, s1->hash, s2->hash); #ifndef MC_DEBUG return 1; #endif } else XBT_VERB("(%d - %d) Same hash : 0x%" PRIx64, num1, num2, s1->hash); } /* Compare enabled processes */ if (s1->enabled_processes != s2->enabled_processes) { XBT_VERB("(%d - %d) Different enabled processes", num1, num2); // return 1; ?? } unsigned long i = 0; size_t size_used1, size_used2; int is_diff = 0; /* Compare size of stacks */ while (i < s1->stacks.size()) { size_used1 = s1->stack_sizes[i]; size_used2 = s2->stack_sizes[i]; if (size_used1 != size_used2) { #ifdef MC_DEBUG XBT_DEBUG("(%d - %d) Different size used in stacks : %zu - %zu", num1, num2, size_used1, size_used2); errors++; is_diff = 1; #else #ifdef MC_VERBOSE XBT_VERB("(%d - %d) Different size used in stacks : %zu - %zu", num1, num2, size_used1, size_used2); #endif return 1; #endif } i++; } /* Init heap information used in heap comparison algorithm */ xbt_mheap_t heap1 = (xbt_mheap_t)s1->read_bytes( alloca(sizeof(struct mdesc)), sizeof(struct mdesc), remote(process->heap_address), simgrid::mc::ProcessIndexMissing, simgrid::mc::ReadOptions::lazy()); xbt_mheap_t heap2 = (xbt_mheap_t)s2->read_bytes( alloca(sizeof(struct mdesc)), sizeof(struct mdesc), remote(process->heap_address), simgrid::mc::ProcessIndexMissing, simgrid::mc::ReadOptions::lazy()); res_init = state_comparator->initHeapInformation( heap1, heap2, &s1->to_ignore, &s2->to_ignore); if (res_init == -1) { #ifdef MC_DEBUG XBT_DEBUG("(%d - %d) Different heap information", num1, num2); errors++; #else #ifdef MC_VERBOSE XBT_VERB("(%d - %d) Different heap information", num1, num2); #endif return 1; #endif } /* Stacks comparison */ unsigned cursor = 0; int diff_local = 0; #ifdef MC_DEBUG is_diff = 0; #endif mc_snapshot_stack_t stack1, stack2; while (cursor < s1->stacks.size()) { stack1 = &s1->stacks[cursor]; stack2 = &s2->stacks[cursor]; if (stack1->process_index != stack2->process_index) { diff_local = 1; XBT_DEBUG("(%d - %d) Stacks with different process index (%i vs %i)", num1, num2, stack1->process_index, stack2->process_index); } else diff_local = compare_local_variables(*state_comparator, stack1->process_index, s1, s2, stack1, stack2); if (diff_local > 0) { #ifdef MC_DEBUG XBT_DEBUG("(%d - %d) Different local variables between stacks %d", num1, num2, cursor + 1); errors++; is_diff = 1; #else #ifdef MC_VERBOSE XBT_VERB("(%d - %d) Different local variables between stacks %d", num1, num2, cursor + 1); #endif return 1; #endif } cursor++; } size_t regions_count = s1->snapshot_regions.size(); // TODO, raise a difference instead? xbt_assert(regions_count == s2->snapshot_regions.size()); for (size_t k = 0; k != regions_count; ++k) { mc_mem_region_t region1 = s1->snapshot_regions[k].get(); mc_mem_region_t region2 = s2->snapshot_regions[k].get(); // Preconditions: if (region1->region_type() != simgrid::mc::RegionType::Data) continue; xbt_assert(region1->region_type() == region2->region_type()); xbt_assert(region1->object_info() == region2->object_info()); xbt_assert(region1->object_info()); std::string const& name = region1->object_info()->file_name; /* Compare global variables */ is_diff = compare_global_variables(*state_comparator, region1->object_info(), simgrid::mc::ProcessIndexDisabled, region1, region2, s1, s2); if (is_diff != 0) { #ifdef MC_DEBUG XBT_DEBUG("(%d - %d) Different global variables in %s", num1, num2, name.c_str()); errors++; #else #ifdef MC_VERBOSE XBT_VERB("(%d - %d) Different global variables in %s", num1, num2, name.c_str()); #endif return 1; #endif } } /* Compare heap */ if (simgrid::mc::mmalloc_compare_heap(*state_comparator, s1, s2) > 0) { #ifdef MC_DEBUG XBT_DEBUG("(%d - %d) Different heap (mmalloc_compare)", num1, num2); errors++; #else #ifdef MC_VERBOSE XBT_VERB("(%d - %d) Different heap (mmalloc_compare)", num1, num2); #endif return 1; #endif } #ifdef MC_VERBOSE if (errors || hash_result) XBT_VERB("(%d - %d) Difference found", num1, num2); else XBT_VERB("(%d - %d) No difference found", num1, num2); #endif #if defined(MC_DEBUG) && defined(MC_VERBOSE) if (_sg_mc_hash) { // * false positive SHOULD be avoided. // * There MUST not be any false negative. XBT_VERB("(%d - %d) State equality hash test is %s %s", num1, num2, (hash_result != 0) == (errors != 0) ? "true" : "false", !hash_result ? "positive" : "negative"); } #endif return errors > 0 || hash_result; } } } SimGrid-3.14.159/src/mc/ObjectInformation.hpp000644 001750 001750 00000012467 13031031270 022413 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_OBJECT_INFORMATION_HPP #define SIMGRID_MC_OBJECT_INFORMATION_HPP #include #include #include #include #include #include "src/xbt/memory_map.hpp" #include "src/mc/mc_forward.hpp" #include "src/mc/Type.hpp" #include "src/mc/Frame.hpp" #include "src/smpi/private.h" namespace simgrid { namespace mc { /** An entry in the functions index * * See the code of ObjectInformation::find_function. */ struct FunctionIndexEntry { void* low_pc; simgrid::mc::Frame* function; }; /** Information about an ELF module (executable or shared object) * * This contains all the information we need about an executable or * shared-object in the model-checked process: * * - where it is located in the virtual address space; * * - where are located its different memory mappings in the the * virtual address space; * * - all the debugging (DWARF) information * - types, * - location of the functions and their local variables, * - global variables, * * - etc. */ class ObjectInformation { public: ObjectInformation(); // Not copyable: ObjectInformation(ObjectInformation const&) = delete; ObjectInformation& operator=(ObjectInformation const&) = delete; // Flag: static const int Executable = 1; /** Bitfield of flags */ int flags = 0; std::string file_name; const void* start = nullptr; const void *end = nullptr; // Location of its text segment: char *start_exec = nullptr; char *end_exec = nullptr; // Location of the read-only part of its data segment: char *start_rw = nullptr; char *end_rw = nullptr; // Location of the read/write part of its data segment: char *start_ro = nullptr; char *end_ro = nullptr; /** All of its subprograms indexed by their address */ std::unordered_map subprograms; /** Index of functions by instruction address * * We need to efficiently find the function from any given instruction * address inside its range. This index is sorted by low_pc * * The entries are sorted by low_pc and a binary search can be used to look * them up. In order to have a better cache locality, we only keep the * information we need for the lookup in this vector. We could probably * replace subprograms by an ordered vector of Frame and replace this one b * a parallel `std::vector`. */ std::vector functions_index; // TODO, remove the mutable (to remove it we'll have to add a lot of const everywhere) mutable std::vector global_variables; /** Types indexed by DWARF ID */ std::unordered_map types; /** Types indexed by name * * Different compilation units have their separate type definitions * (for the same type). When we find an opaque type in one compilation unit, * we use this in order to try to find its definition in another compilation * unit. */ std::unordered_map full_types_by_name; /** Whether this module is an executable * * More precisely we check if this is an ET_EXE ELF. These ELF files * use fixed addresses instead of base-addres relative addresses. * Position independant executables are in fact ET_DYN. */ bool executable() const { return this->flags & simgrid::mc::ObjectInformation::Executable; } /** Base address of the module * * All the location information in ELF and DWARF are expressed as an offsets * from this base address: * * - location of the functions and globale variables; * * - the DWARF instruction `OP_addr` pushes this on the DWARF stack. **/ void* base_address() const; /** Find a function by instruction address * * @param ip instruction address * @return corresponding function (if any) or nullptr */ simgrid::mc::Frame* find_function(const void *ip) const; /** Find a global variable by name * * This is used to ignore global variables and to find well-known variables * (`__mmalloc_default_mdp`). * * @param name scopes name of the global variable (`myproject::Foo::count`) * @return corresponding variable (if any) or nullptr */ simgrid::mc::Variable* find_variable(const char* name) const; /** Remove a global variable (in order to ignore it) * * This is used to ignore a global variable for the snapshot comparison. */ void remove_global_variable(const char* name); /** Remove a loval variables (in order to ignore it) * * @param name Name of the globale variable * @param scope Namespaceed name of the function (or null for all functions) */ void remove_local_variable( const char* name, const char* scope); }; XBT_PRIVATE std::shared_ptr createObjectInformation( std::vector const& maps, const char* name); /** Augment the current module with informations about the other ones */ XBT_PRIVATE void postProcessObjectInformation( simgrid::mc::Process* process, simgrid::mc::ObjectInformation* info); } } #endif SimGrid-3.14.159/src/mc/ChunkedData.hpp000644 001750 001750 00000005125 13031031270 021143 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_CHUNKED_DATA_HPP #define SIMGRID_MC_CHUNKED_DATA_HPP #include #include #include #include #include "src/mc/mc_forward.hpp" #include "src/mc/PageStore.hpp" namespace simgrid { namespace mc { /** A byte-string represented as a sequence of chunks from a PageStore * * In order to save memory when taking memory snapshots, a given byte-string * is split in fixed-size chunks. Identical chunks (either from the same * snapshot or more probably from different snpashots) share the same memory * storage. * * Thus a chunked is represented as a sequence of indices of each chunk. */ class ChunkedData { /** This is where we store the chunks */ PageStore* store_ = nullptr; /** Indices of the chunks in the `PageStore` */ std::vector pagenos_; public: ChunkedData() {} void clear() { for (std::size_t pageno : pagenos_) store_->unref_page(pageno); pagenos_.clear(); } ~ChunkedData() { clear(); } // Copy and move ChunkedData(ChunkedData const& that) { store_ = that.store_; pagenos_ = that.pagenos_; for (std::size_t pageno : pagenos_) store_->ref_page(pageno); } ChunkedData(ChunkedData&& that) { store_ = that.store_; that.store_ = nullptr; pagenos_ = std::move(that.pagenos_); that.pagenos_.clear(); } ChunkedData& operator=(ChunkedData const& that) { this->clear(); store_ = that.store_; pagenos_ = that.pagenos_; for (std::size_t pageno : pagenos_) store_->ref_page(pageno); return *this; } ChunkedData& operator=(ChunkedData && that) { this->clear(); store_ = that.store_; that.store_ = nullptr; pagenos_ = std::move(that.pagenos_); that.pagenos_.clear(); return *this; } /** How many pages are used */ std::size_t page_count() const { return pagenos_.size(); } /** Get a chunk index */ std::size_t pageno(std::size_t i) const { return pagenos_[i]; } /** Get a view of the chunk indices */ const std::size_t* pagenos() const { return pagenos_.data(); } /** Get a a pointer to a chunk */ const void* page(std::size_t i) const { return store_->get_page(pagenos_[i]); } ChunkedData(PageStore& store, AddressSpace& as, RemotePtr addr, std::size_t page_count); }; } } #endif SimGrid-3.14.159/src/mc/RegionSnapshot.hpp000644 001750 001750 00000016757 13031031270 021750 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_REGION_SNAPSHOT_HPP #define SIMGRID_MC_REGION_SNAPSHOT_HPP #include #include #include #include #include #include "src/mc/RemotePtr.hpp" #include "src/mc/PageStore.hpp" #include "src/mc/AddressSpace.hpp" #include "src/mc/ChunkedData.hpp" namespace simgrid { namespace mc { enum class RegionType { Unknown = 0, Heap = 1, Data = 2 }; enum class StorageType { NoData = 0, Flat = 1, Chunked = 2, Privatized = 3 }; class Buffer { private: enum class Type { Malloc, Mmap }; void* data_ = nullptr; std::size_t size_; Type type_ = Type::Malloc; private: Buffer(std::size_t size, Type type = Type::Malloc); Buffer(void* data, std::size_t size, Type type = Type::Malloc) : data_(data), size_(size), type_(type) {} public: Buffer() {} void clear() noexcept; ~Buffer() noexcept { clear(); } static Buffer malloc(std::size_t size) { return Buffer(size, Type::Malloc); } static Buffer mmap(std::size_t size) { return Buffer(size, Type::Mmap); } // No copy Buffer(Buffer const& buffer) = delete; Buffer& operator=(Buffer const& buffer) = delete; // Move Buffer(Buffer&& that) noexcept : data_(that.data_), size_(that.size_), type_(that.type_) { that.data_ = nullptr; that.size_ = 0; that.type_ = Type::Malloc; } Buffer& operator=(Buffer&& that) noexcept { clear(); data_ = that.data_; size_ = that.size_; type_ = that.type_; that.data_ = nullptr; that.size_ = 0; that.type_ = Type::Malloc; return *this; } void* get() { return data_; } const void* get() const { return data_; } std::size_t size() const { return size_; } }; /** A copy/snapshot of a given memory region * * Different types of region snapshot storage types exist: * * * flat/dense snapshots are a simple copy of the region; * * * sparse/per-page snapshots are snaapshots which shared * identical pages. * * * privatized (SMPI global variable privatisation). * * This is handled with a variant based approach: * * * `storage_type` identified the type of storage; * * * an anonymous enum is used to distinguish the relevant types for * each type. */ class RegionSnapshot { public: static const RegionType UnknownRegion = RegionType::Unknown; static const RegionType HeapRegion = RegionType::Heap; static const RegionType DataRegion = RegionType::Data; private: RegionType region_type_; StorageType storage_type_; simgrid::mc::ObjectInformation* object_info_; /** @brief Virtual address of the region in the simulated process */ void *start_addr_; /** @brief Size of the data region in bytes */ std::size_t size_; /** @brief Permanent virtual address of the region * * This is usually the same address as the simuilated process address. * However, when using SMPI privatization of global variables, * each SMPI process has its own set of global variables stored * at a different virtual address. The scheduler maps those region * on the region of the global variables. * * */ void *permanent_addr_; Buffer flat_data_; ChunkedData page_numbers_; std::vector privatized_regions_; public: RegionSnapshot() : region_type_(UnknownRegion), storage_type_(StorageType::NoData), object_info_(nullptr), start_addr_(nullptr), size_(0), permanent_addr_(nullptr) {} RegionSnapshot(RegionType type, void *start_addr, void* permanent_addr, size_t size) : region_type_(type), storage_type_(StorageType::NoData), object_info_(nullptr), start_addr_(start_addr), size_(size), permanent_addr_(permanent_addr) {} ~RegionSnapshot() {} RegionSnapshot(RegionSnapshot const&) = default; RegionSnapshot& operator=(RegionSnapshot const&) = default; RegionSnapshot(RegionSnapshot&& that) { region_type_ = that.region_type_; storage_type_ = that.storage_type_; object_info_ = that.object_info_; start_addr_ = that.start_addr_; size_ = that.size_; permanent_addr_ = that.permanent_addr_; flat_data_ = std::move(that.flat_data_); page_numbers_ = std::move(that.page_numbers_); privatized_regions_ = std::move(that.privatized_regions_); that.clear(); } RegionSnapshot& operator=(RegionSnapshot&& that) { region_type_ = that.region_type_; storage_type_ = that.storage_type_; object_info_ = that.object_info_; start_addr_ = that.start_addr_; size_ = that.size_; permanent_addr_ = that.permanent_addr_; flat_data_ = std::move(that.flat_data_); page_numbers_ = std::move(that.page_numbers_); privatized_regions_ = std::move(that.privatized_regions_); that.clear(); return *this; } // Data void clear() { region_type_ = UnknownRegion; storage_type_ = StorageType::NoData; privatized_regions_.clear(); page_numbers_.clear(); flat_data_.clear(); object_info_ = nullptr; start_addr_ = nullptr; size_ = 0; permanent_addr_ = nullptr; } void clear_data() { storage_type_ = StorageType::NoData; flat_data_.clear(); page_numbers_.clear(); privatized_regions_.clear(); } void flat_data(Buffer data) { storage_type_ = StorageType::Flat; flat_data_ = std::move(data); page_numbers_.clear(); privatized_regions_.clear(); } const Buffer& flat_data() const { return flat_data_; } Buffer& flat_data() { return flat_data_; } void page_data(ChunkedData page_data) { storage_type_ = StorageType::Chunked; flat_data_.clear(); page_numbers_ = std::move(page_data); privatized_regions_.clear(); } ChunkedData const& page_data() const { return page_numbers_; } void privatized_data(std::vector data) { storage_type_ = StorageType::Privatized; flat_data_.clear(); page_numbers_.clear(); privatized_regions_ = std::move(data); } std::vector const& privatized_data() const { return privatized_regions_; } std::vector& privatized_data() { return privatized_regions_; } simgrid::mc::ObjectInformation* object_info() const { return object_info_; } void object_info(simgrid::mc::ObjectInformation* info) { object_info_ = info; } // Other getters RemotePtr start() const { return remote(start_addr_); } RemotePtr end() const { return remote((char*)start_addr_ + size_); } RemotePtr permanent_address() const { return remote(permanent_addr_); } std::size_t size() const { return size_; } StorageType storage_type() const { return storage_type_; } RegionType region_type() const { return region_type_; } bool contain(RemotePtr p) const { return p >= start() && p < end(); } }; RegionSnapshot privatized_region( RegionType region_type, void *start_addr, void* permanent_addr, std::size_t size); RegionSnapshot dense_region( RegionType type, void *start_addr, void* data_addr, std::size_t size); simgrid::mc::RegionSnapshot sparse_region( RegionType type, void *start_addr, void* data_addr, std::size_t size); simgrid::mc::RegionSnapshot region( RegionType type, void *start_addr, void* data_addr, std::size_t size); } } typedef class simgrid::mc::RegionSnapshot s_mc_mem_region_t, *mc_mem_region_t; #endif SimGrid-3.14.159/src/mc/Variable.cpp000644 001750 001750 00000000000 13031031270 020473 0ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/src/mc/mc_dwarf_attrnames.cpp000644 001750 001750 00000015024 13031031270 022622 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* Warning: autogenerated, do not edit! */ #include #include #include namespace simgrid { namespace dwarf { /** \brief Get the name of an attribute (DW_AT_*) from its code * * \param attr attribute code (see the DWARF specification) * \return name of the attribute */ XBT_PRIVATE const char *attrname(int attr) { switch (attr) { case 0x01: return "DW_AT_sibling"; case 0x02: return "DW_AT_location"; case 0x03: return "DW_AT_name"; case 0x09: return "DW_AT_ordering"; case 0x0a: return "DW_AT_subscr_data"; case 0x0b: return "DW_AT_byte_size"; case 0x0c: return "DW_AT_bit_offset"; case 0x0d: return "DW_AT_bit_size"; case 0x0f: return "DW_AT_element_list"; case 0x10: return "DW_AT_stmt_list"; case 0x11: return "DW_AT_low_pc"; case 0x12: return "DW_AT_high_pc"; case 0x13: return "DW_AT_language"; case 0x14: return "DW_AT_member"; case 0x15: return "DW_AT_discr"; case 0x16: return "DW_AT_discr_value"; case 0x17: return "DW_AT_visibility"; case 0x18: return "DW_AT_import"; case 0x19: return "DW_AT_string_length"; case 0x1a: return "DW_AT_common_reference"; case 0x1b: return "DW_AT_comp_dir"; case 0x1c: return "DW_AT_const_value"; case 0x1d: return "DW_AT_containing_type"; case 0x1e: return "DW_AT_default_value"; case 0x20: return "DW_AT_inline"; case 0x21: return "DW_AT_is_optional"; case 0x22: return "DW_AT_lower_bound"; case 0x25: return "DW_AT_producer"; case 0x27: return "DW_AT_prototyped"; case 0x2a: return "DW_AT_return_addr"; case 0x2c: return "DW_AT_start_scope"; case 0x2e: return "DW_AT_bit_stride"; case 0x2f: return "DW_AT_upper_bound"; case 0x31: return "DW_AT_abstract_origin"; case 0x32: return "DW_AT_accessibility"; case 0x33: return "DW_AT_address_class"; case 0x34: return "DW_AT_artificial"; case 0x35: return "DW_AT_base_types"; case 0x36: return "DW_AT_calling_convention"; case 0x37: return "DW_AT_count"; case 0x38: return "DW_AT_data_member_location"; case 0x39: return "DW_AT_decl_column"; case 0x3a: return "DW_AT_decl_file"; case 0x3b: return "DW_AT_decl_line"; case 0x3c: return "DW_AT_declaration"; case 0x3d: return "DW_AT_discr_list"; case 0x3e: return "DW_AT_encoding"; case 0x3f: return "DW_AT_external"; case 0x40: return "DW_AT_frame_base"; case 0x41: return "DW_AT_friend"; case 0x42: return "DW_AT_identifier_case"; case 0x43: return "DW_AT_macro_info"; case 0x44: return "DW_AT_namelist_item"; case 0x45: return "DW_AT_priority"; case 0x46: return "DW_AT_segment"; case 0x47: return "DW_AT_specification"; case 0x48: return "DW_AT_static_link"; case 0x49: return "DW_AT_type"; case 0x4a: return "DW_AT_use_location"; case 0x4b: return "DW_AT_variable_parameter"; case 0x4c: return "DW_AT_virtuality"; case 0x4d: return "DW_AT_vtable_elem_location"; case 0x4e: return "DW_AT_allocated"; case 0x4f: return "DW_AT_associated"; case 0x50: return "DW_AT_data_location"; case 0x51: return "DW_AT_byte_stride"; case 0x52: return "DW_AT_entry_pc"; case 0x53: return "DW_AT_use_UTF8"; case 0x54: return "DW_AT_extension"; case 0x55: return "DW_AT_ranges"; case 0x56: return "DW_AT_trampoline"; case 0x57: return "DW_AT_call_column"; case 0x58: return "DW_AT_call_file"; case 0x59: return "DW_AT_call_line"; case 0x5a: return "DW_AT_description"; case 0x5b: return "DW_AT_binary_scale"; case 0x5c: return "DW_AT_decimal_scale"; case 0x5d: return "DW_AT_small"; case 0x5e: return "DW_AT_decimal_sign"; case 0x5f: return "DW_AT_digit_count"; case 0x60: return "DW_AT_picture_string"; case 0x61: return "DW_AT_mutable"; case 0x62: return "DW_AT_threads_scaled"; case 0x63: return "DW_AT_explicit"; case 0x64: return "DW_AT_object_pointer"; case 0x65: return "DW_AT_endianity"; case 0x66: return "DW_AT_elemental"; case 0x67: return "DW_AT_pure"; case 0x68: return "DW_AT_recursive"; case 0x69: return "DW_AT_signature"; case 0x6a: return "DW_AT_main_subprogram"; case 0x6b: return "DW_AT_data_bit_offset"; case 0x6c: return "DW_AT_const_expr"; case 0x6d: return "DW_AT_enum_class"; case 0x6e: return "DW_AT_linkage_name"; case 0x2000: return "DW_AT_lo_user"; case 0x2001: return "DW_AT_MIPS_fde"; case 0x2002: return "DW_AT_MIPS_loop_begin"; case 0x2003: return "DW_AT_MIPS_tail_loop_begin"; case 0x2004: return "DW_AT_MIPS_epilog_begin"; case 0x2005: return "DW_AT_MIPS_loop_unroll_factor"; case 0x2006: return "DW_AT_MIPS_software_pipeline_depth"; case 0x2007: return "DW_AT_MIPS_linkage_name"; case 0x2008: return "DW_AT_MIPS_stride"; case 0x2009: return "DW_AT_MIPS_abstract_name"; case 0x200a: return "DW_AT_MIPS_clone_origin"; case 0x200b: return "DW_AT_MIPS_has_inlines"; case 0x200c: return "DW_AT_MIPS_stride_byte"; case 0x200d: return "DW_AT_MIPS_stride_elem"; case 0x200e: return "DW_AT_MIPS_ptr_dopetype"; case 0x200f: return "DW_AT_MIPS_allocatable_dopetype"; case 0x2010: return "DW_AT_MIPS_assumed_shape_dopetype"; case 0x2011: return "DW_AT_MIPS_assumed_size"; case 0x2101: return "DW_AT_sf_names"; case 0x2102: return "DW_AT_src_info"; case 0x2103: return "DW_AT_mac_info"; case 0x2104: return "DW_AT_src_coords"; case 0x2105: return "DW_AT_body_begin"; case 0x2106: return "DW_AT_body_end"; case 0x2107: return "DW_AT_GNU_vector"; case 0x2108: return "DW_AT_GNU_guarded_by"; case 0x2109: return "DW_AT_GNU_pt_guarded_by"; case 0x210a: return "DW_AT_GNU_guarded"; case 0x210b: return "DW_AT_GNU_pt_guarded"; case 0x210c: return "DW_AT_GNU_locks_excluded"; case 0x210d: return "DW_AT_GNU_exclusive_locks_required"; case 0x210e: return "DW_AT_GNU_shared_locks_required"; case 0x210f: return "DW_AT_GNU_odr_signature"; case 0x2110: return "DW_AT_GNU_template_name"; case 0x2111: return "DW_AT_GNU_call_site_value"; case 0x2112: return "DW_AT_GNU_call_site_data_value"; case 0x2113: return "DW_AT_GNU_call_site_target"; case 0x2114: return "DW_AT_GNU_call_site_target_clobbered"; case 0x2115: return "DW_AT_GNU_tail_call"; case 0x2116: return "DW_AT_GNU_all_tail_call_sites"; case 0x2117: return "DW_AT_GNU_all_call_sites"; case 0x2118: return "DW_AT_GNU_all_source_call_sites"; case 0x2119: return "DW_AT_GNU_macros"; case 0x3fff: return "DW_AT_hi_user"; default: return "DW_AT_unknown"; } } } } SimGrid-3.14.159/src/mc/mc_ignore.h000644 001750 001750 00000000773 13031031270 020376 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_IGNORE_H #define SIMGRID_MC_IGNORE_H #include /* SG_BEGIN_DECL */ #include SG_BEGIN_DECL(); XBT_PRIVATE xbt_dynar_t MC_checkpoint_ignore_new(void); SG_END_DECL(); #endif SimGrid-3.14.159/src/mc/PageStore.hpp000644 001750 001750 00000014563 13031031273 020672 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_PAGESTORE_HPP #define SIMGRID_MC_PAGESTORE_HPP #include #include #include #include #include #include "src/mc/mc_mmu.h" #include "src/mc/mc_forward.hpp" namespace simgrid { namespace mc { /** @brief Storage for snapshot memory pages * * The first (lower) layer of the per-page snapshot mechanism is a page * store: its responsibility is to store immutable shareable * reference-counted memory pages independently of the snapshotting * logic. Snapshot management and representation is * handled to an higher layer. READMORE * * Data structure: * * * A pointer (`memory_`) to a (currently anonymous) `mmap()`ed memory * region holding the memory pages (the address of the first page). * * We want to keep this memory region aligned on the memory pages (so * that we might be able to create non-linear memory mappings on those * pages in the future) and be able to expand it without coyping the * data (there will be a lot of pages here): we will be able to * efficiently expand the memory mapping using `mremap()`, moving it * to another virtual address if necessary. * * Because we will move this memory mapping on the virtual address * space, only the index of the page will be stored in the snapshots * and the page will always be looked up by going through `memory`: * * void* page = (char*) page_store->memory + page_index << pagebits; * * * The number of pages mapped in virtual memory (`capacity_`). Once all * those pages are used, we need to expand the page store with * `mremap()`. * * * A reference count for each memory page `page_counts_`. Each time a * snapshot references a page, the counter is incremented. If a * snapshot is freed, the reference count is decremented. When the * reference count, of a page reaches 0 it is added to a list of available * pages (`free_pages_`). * * * A list of free pages `free_pages_` which can be reused. This avoids having * to scan the reference count list to find a free page. * * * When we are expanding the memory map we do not want to add thousand of page * to the `free_pages_` list and remove them just afterwards. The `top_index_` * field is an index after which all pages are free and are not in the `free_pages_` * list. * * * When we are adding a page, we need to check if a page with the same * content is already in the page store in order to reuse it. For this * reason, we maintain an index (`hash_index_`) mapping the hash of a * page to the list of page indices with this hash. * We use a fast (non cryptographic) hash so there may be conflicts: * we must be able to store multiple indices for the same hash. * */ class PageStore { public: // Types typedef std::uint64_t hash_type; private: // Types // We are using a cheap hash to index a page. // We should expect collision and we need to associate multiple page indices // to the same hash. typedef std::unordered_set page_set_type; typedef std::unordered_map pages_map_type; private: // Fields: /** First page */ void* memory_; /** Number of available pages in virtual memory */ std::size_t capacity_; /** Top of the used pages (index of the next available page) */ std::size_t top_index_; /** Page reference count */ std::vector page_counts_; /** Index of available pages before the top */ std::vector free_pages_; /** Index from page hash to page index */ pages_map_type hash_index_; private: // Methods void resize(std::size_t size); std::size_t alloc_page(); void remove_page(std::size_t pageno); public: // Constructors PageStore(PageStore const&) = delete; PageStore& operator=(PageStore const&) = delete; explicit PageStore(std::size_t size); ~PageStore(); public: // Methods /** @brief Decrement the reference count for a given page * * Decrement the reference count of this page. Used when a snapshot is * destroyed. * * If the reference count reaches zero, the page is recycled: * it is added to the `free_pages_` list and removed from the `hash_index_`. * * */ void unref_page(std::size_t pageno); /** @brief Increment the refcount for a given page * * This method used to increase a reference count of a page if we know * that the content of a page is the same as a page already in the page * store. * * This will be the case if a page if soft clean: we know that is has not * changed since the previous cnapshot/restoration and we can avoid * hashing the page, comparing byte-per-byte to candidates. * */ void ref_page(size_t pageno); /** @brief Store a page in the page store */ std::size_t store_page(void* page); /** @brief Get a page from its page number * * @param Number of the memory page in the store * @return Start of the page */ const void* get_page(std::size_t pageno) const; public: // Debug/test methods /** @brief Get the number of references for a page */ std::size_t get_ref(std::size_t pageno); /** @brief Get the number of used pages */ std::size_t size(); /** @brief Get the capacity of the page store * * The capacity is expanded by a system call (mremap). * */ std::size_t capacity(); }; inline __attribute__((always_inline)) void PageStore::unref_page(std::size_t pageno) { if ((--this->page_counts_[pageno]) == 0) this->remove_page(pageno); } inline __attribute__((always_inline)) void PageStore::ref_page(size_t pageno) { ++this->page_counts_[pageno]; } inline __attribute__((always_inline)) const void* PageStore::get_page(std::size_t pageno) const { return (void*) simgrid::mc::mmu::join(pageno, (std::uintptr_t) this->memory_); } inline __attribute__((always_inline)) std::size_t PageStore::get_ref(std::size_t pageno) { return this->page_counts_[pageno]; } inline __attribute__((always_inline)) std::size_t PageStore::size() { return this->top_index_ - this->free_pages_.size(); } inline __attribute__((always_inline)) std::size_t PageStore::capacity() { return this->capacity_; } } } #endif SimGrid-3.14.159/src/mc/mc_record.h000644 001750 001750 00000003171 13031031273 020367 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /** \file mc_record.h * * This file contains the MC replay/record functionnality. * A MC path may be recorded by using ``-cfg=model-check/record:1`'`. * The path is written in the log output and an be replayed with MC disabled * (even with an non-LC build) with `--cfg=model-check/replay:$replayPath`. * * The same version of Simgrid should be used and the same arguments should be * passed to the application (without the MC specific arguments). */ #ifndef SIMGRID_MC_RECORD_H #define SIMGRID_MC_RECORD_H #include #include #include #include "src/mc/Transition.hpp" namespace simgrid { namespace mc { typedef std::vector RecordTrace; /** Convert a string representation of the path into a array of `simgrid::mc::Transition` */ XBT_PRIVATE RecordTrace parseRecordTrace(const char* data); XBT_PRIVATE std::string traceToString(simgrid::mc::RecordTrace const& trace); XBT_PRIVATE void dumpRecordPath(); XBT_PRIVATE void replay(RecordTrace const& trace); XBT_PRIVATE void replay(const char* trace); } } SG_BEGIN_DECL() /** Whether the MC record mode is enabled * * The behaviour is not changed. The only real difference is that * the path is writtent in the log when an interesting path is found. */ #define MC_record_is_active() _sg_do_model_check_record // **** Data conversion SG_END_DECL() #endif SimGrid-3.14.159/src/mc/ModelChecker.cpp000644 001750 001750 00000030334 13031031270 021310 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include #include #include #include #include #include "simgrid/sg_config.h" #include "src/mc/ModelChecker.hpp" #include "src/mc/PageStore.hpp" #include "src/mc/ModelChecker.hpp" #include "src/mc/mc_protocol.h" #include "src/mc/mc_private.h" #include "src/mc/mc_ignore.h" #include "src/mc/mc_exit.h" #include "src/mc/mc_record.h" #include "src/mc/Transition.hpp" #include "src/mc/Checker.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_ModelChecker, mc, "ModelChecker"); ::simgrid::mc::ModelChecker* mc_model_checker = nullptr; using simgrid::mc::remote; #ifdef __linux__ # define WAITPID_CHECKED_FLAGS __WALL #else # define WAITPID_CHECKED_FLAGS 0 #endif namespace simgrid { namespace mc { ModelChecker::ModelChecker(std::unique_ptr process) : base_(nullptr), socket_event_(nullptr), signal_event_(nullptr), page_store_(500), process_(std::move(process)), parent_snapshot_(nullptr) { } ModelChecker::~ModelChecker() { if (socket_event_ != nullptr) event_free(socket_event_); if (signal_event_ != nullptr) event_free(signal_event_); if (base_ != nullptr) event_base_free(base_); } void ModelChecker::start() { const pid_t pid = process_->pid(); base_ = event_base_new(); event_callback_fn event_callback = [](evutil_socket_t fd, short events, void *arg) { ((ModelChecker *)arg)->handle_events(fd, events); }; socket_event_ = event_new(base_, process_->getChannel().getSocket(), EV_READ|EV_PERSIST, event_callback, this); event_add(socket_event_, NULL); signal_event_ = event_new(base_, SIGCHLD, EV_SIGNAL|EV_PERSIST, event_callback, this); event_add(signal_event_, NULL); XBT_DEBUG("Waiting for the model-checked process"); int status; // The model-checked process SIGSTOP itself to signal it's ready: pid_t res = waitpid(pid, &status, WAITPID_CHECKED_FLAGS); if (res < 0 || !WIFSTOPPED(status) || WSTOPSIG(status) != SIGSTOP) xbt_die("Could not wait model-checked process"); process_->init(); if ((_sg_mc_dot_output_file != nullptr) && (_sg_mc_dot_output_file[0] != '\0')) MC_init_dot_output(); setup_ignore(); #ifdef __linux__ ptrace(PTRACE_SETOPTIONS, pid, nullptr, PTRACE_O_TRACEEXIT); ptrace(PTRACE_CONT, pid, 0, 0); #elif defined BSD ptrace(PT_CONTINUE, pid, (caddr_t)1, 0); #else # error "no ptrace equivalent coded for this platform" #endif } static const std::pair ignored_local_variables[] = { std::pair{ "e", "*" }, std::pair{ "__ex_cleanup", "*" }, std::pair{ "__ex_mctx_en", "*" }, std::pair{ "__ex_mctx_me", "*" }, std::pair{ "__xbt_ex_ctx_ptr", "*" }, std::pair{ "_log_ev", "*" }, std::pair{ "_throw_ctx", "*" }, std::pair{ "ctx", "*" }, std::pair{ "self", "simcall_BODY_mc_snapshot" }, std::pair{ "next_context", "smx_ctx_sysv_suspend_serial" }, std::pair{ "i", "smx_ctx_sysv_suspend_serial" }, /* Ignore local variable about time used for tracing */ std::pair{ "start_time", "*" }, }; void ModelChecker::setup_ignore() { Process& process = this->process(); for (std::pair const& var : ignored_local_variables) process.ignore_local_variable(var.first, var.second); /* Static variable used for tracing */ process.ignore_global_variable("counter"); } void ModelChecker::shutdown() { XBT_DEBUG("Shuting down model-checker"); simgrid::mc::Process* process = &this->process(); if (process->running()) { XBT_DEBUG("Killing process"); kill(process->pid(), SIGTERM); process->terminate(); } } void ModelChecker::resume(simgrid::mc::Process& process) { int res = process.getChannel().send(MC_MESSAGE_CONTINUE); if (res) throw simgrid::xbt::errno_error(); process.clear_cache(); } static void throw_socket_error(int fd) { int error = 0; socklen_t errlen = sizeof(error); if (getsockopt(fd, SOL_SOCKET, SO_ERROR, (void *)&error, &errlen) == -1) error = errno; throw simgrid::xbt::errno_error(); } static void MC_report_crash(int status) { XBT_INFO("**************************"); XBT_INFO("** CRASH IN THE PROGRAM **"); XBT_INFO("**************************"); if (WIFSIGNALED(status)) XBT_INFO("From signal: %s", strsignal(WTERMSIG(status))); else if (WIFEXITED(status)) XBT_INFO("From exit: %i", WEXITSTATUS(status)); if (WCOREDUMP(status)) XBT_INFO("A core dump was generated by the system."); else XBT_INFO("No core dump was generated by the system."); XBT_INFO("Counter-example execution trace:"); simgrid::mc::dumpRecordPath(); for (auto& s : mc_model_checker->getChecker()->getTextualTrace()) XBT_INFO("%s", s.c_str()); simgrid::mc::session->logState(); XBT_INFO("Stack trace:"); mc_model_checker->process().dumpStack(); } static void MC_report_assertion_error(void) { XBT_INFO("**************************"); XBT_INFO("*** PROPERTY NOT VALID ***"); XBT_INFO("**************************"); XBT_INFO("Counter-example execution trace:"); simgrid::mc::dumpRecordPath(); for (auto& s : mc_model_checker->getChecker()->getTextualTrace()) XBT_INFO("%s", s.c_str()); simgrid::mc::session->logState(); } bool ModelChecker::handle_message(char* buffer, ssize_t size) { s_mc_message_t base_message; if (size < (ssize_t) sizeof(base_message)) xbt_die("Broken message"); memcpy(&base_message, buffer, sizeof(base_message)); switch(base_message.type) { case MC_MESSAGE_IGNORE_HEAP: { s_mc_ignore_heap_message_t message; if (size != sizeof(message)) xbt_die("Broken messsage"); memcpy(&message, buffer, sizeof(message)); IgnoredHeapRegion region; region.block = message.block; region.fragment = message.fragment; region.address = message.address; region.size = message.size; process().ignore_heap(region); break; } case MC_MESSAGE_UNIGNORE_HEAP: { s_mc_ignore_memory_message_t message; if (size != sizeof(message)) xbt_die("Broken messsage"); memcpy(&message, buffer, sizeof(message)); process().unignore_heap( (void *)(std::uintptr_t) message.addr, message.size); break; } case MC_MESSAGE_IGNORE_MEMORY: { s_mc_ignore_memory_message_t message; if (size != sizeof(message)) xbt_die("Broken messsage"); memcpy(&message, buffer, sizeof(message)); this->process().ignore_region(message.addr, message.size); break; } case MC_MESSAGE_STACK_REGION: { s_mc_stack_region_message_t message; if (size != sizeof(message)) xbt_die("Broken messsage"); memcpy(&message, buffer, sizeof(message)); this->process().stack_areas().push_back(message.stack_region); } break; case MC_MESSAGE_REGISTER_SYMBOL: { s_mc_register_symbol_message_t message; if (size != sizeof(message)) xbt_die("Broken message"); memcpy(&message, buffer, sizeof(message)); if (message.callback) xbt_die("Support for client-side function proposition is not implemented."); XBT_DEBUG("Received symbol: %s", message.name); if (simgrid::mc::property_automaton == nullptr) simgrid::mc::property_automaton = xbt_automaton_new(); simgrid::mc::Process* process = &this->process(); simgrid::mc::RemotePtr address = simgrid::mc::remote((int*) message.data); simgrid::xbt::add_proposition(simgrid::mc::property_automaton, message.name, [process, address]() { return process->read(address); } ); break; } case MC_MESSAGE_WAITING: return false; case MC_MESSAGE_ASSERTION_FAILED: MC_report_assertion_error(); this->exit(SIMGRID_MC_EXIT_SAFETY); break; default: xbt_die("Unexpected message from model-checked application"); } return true; } /** Terminate the model-checker application */ void ModelChecker::exit(int status) { // TODO, terminate the model checker politely instead of exiting rudely if (process().running()) kill(process().pid(), SIGKILL); ::exit(status); } void ModelChecker::handle_events(int fd, short events) { if (events == EV_READ) { char buffer[MC_MESSAGE_LENGTH]; ssize_t size = process_->getChannel().receive(buffer, sizeof(buffer), false); if (size == -1 && errno != EAGAIN) throw simgrid::xbt::errno_error(); if (!handle_message(buffer, size)) { event_base_loopbreak(base_); } } else if (events == EV_SIGNAL) { on_signal(fd); } else { xbt_die("Unexpected event"); } } void ModelChecker::loop() { if (this->process().running()) event_base_dispatch(base_); } void ModelChecker::handle_waitpid() { XBT_DEBUG("Check for wait event"); int status; pid_t pid; while ((pid = waitpid(-1, &status, WNOHANG)) != 0) { if (pid == -1) { if (errno == ECHILD) { // No more children: if (this->process().running()) xbt_die("Inconsistent state"); else break; } else { XBT_ERROR("Could not wait for pid"); throw simgrid::xbt::errno_error(); } } if (pid == this->process().pid()) { // From PTRACE_O_TRACEEXIT: #ifdef __linux__ if (status>>8 == (SIGTRAP | (PTRACE_EVENT_EXIT<<8))) { if (ptrace(PTRACE_GETEVENTMSG, this->process().pid(), 0, &status) == -1) xbt_die("Could not get exit status"); if (WIFSIGNALED(status)) { MC_report_crash(status); mc_model_checker->exit(SIMGRID_MC_EXIT_PROGRAM_CRASH); } } #endif // We don't care about signals, just reinject them: if (WIFSTOPPED(status)) { XBT_DEBUG("Stopped with signal %i", (int) WSTOPSIG(status)); errno = 0; #ifdef __linux__ ptrace(PTRACE_CONT, this->process().pid(), 0, WSTOPSIG(status)); #elif defined BSD ptrace(PT_CONTINUE, this->process().pid(), (caddr_t)1, WSTOPSIG(status)); #endif if (errno != 0) xbt_die("Could not PTRACE_CONT"); } else if (WIFEXITED(status) || WIFSIGNALED(status)) { XBT_DEBUG("Child process is over"); this->process().terminate(); } } } } void ModelChecker::on_signal(int signo) { switch(signo) { case SIGCHLD: this->handle_waitpid(); break; default: break; } } void ModelChecker::wait_client(simgrid::mc::Process& process) { this->resume(process); if (this->process().running()) event_base_dispatch(base_); } void ModelChecker::handle_simcall(Transition const& transition) { s_mc_simcall_handle_message m; memset(&m, 0, sizeof(m)); m.type = MC_MESSAGE_SIMCALL_HANDLE; m.pid = transition.pid; m.value = transition.argument; this->process_->getChannel().send(m); this->process_->clear_cache(); if (this->process_->running()) event_base_dispatch(base_); } bool ModelChecker::checkDeadlock() { int res; if ((res = this->process().getChannel().send(MC_MESSAGE_DEADLOCK_CHECK))) xbt_die("Could not check deadlock state"); s_mc_int_message_t message; ssize_t s = mc_model_checker->process().getChannel().receive(message); if (s == -1) xbt_die("Could not receive message"); if (s != sizeof(message) || message.type != MC_MESSAGE_DEADLOCK_CHECK_REPLY) xbt_die("Received unexpected message %s (%i, size=%i) " "expected MC_MESSAGE_DEADLOCK_CHECK_REPLY (%i, size=%i)", MC_message_type_name(message.type), (int) message.type, (int) s, (int) MC_MESSAGE_DEADLOCK_CHECK_REPLY, (int) sizeof(message) ); return message.value != 0; } } } SimGrid-3.14.159/src/mc/SafetyChecker.cpp000644 001750 001750 00000026715 13031031270 021513 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include #include "src/mc/mc_state.h" #include "src/mc/mc_request.h" #include "src/mc/mc_safety.h" #include "src/mc/mc_private.h" #include "src/mc/mc_record.h" #include "src/mc/mc_smx.h" #include "src/mc/Client.hpp" #include "src/mc/mc_exit.h" #include "src/mc/Checker.hpp" #include "src/mc/SafetyChecker.hpp" #include "src/mc/VisitedState.hpp" #include "src/mc/Transition.hpp" #include "src/mc/Session.hpp" #include "src/xbt/mmalloc/mmprivate.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_safety, mc, "Logging specific to MC safety verification "); namespace simgrid { namespace mc { static void MC_show_non_termination(void) { XBT_INFO("******************************************"); XBT_INFO("*** NON-PROGRESSIVE CYCLE DETECTED ***"); XBT_INFO("******************************************"); XBT_INFO("Counter-example execution trace:"); for (auto& s : mc_model_checker->getChecker()->getTextualTrace()) XBT_INFO("%s", s.c_str()); simgrid::mc::session->logState(); } static int snapshot_compare(simgrid::mc::State* state1, simgrid::mc::State* state2) { simgrid::mc::Snapshot* s1 = state1->system_state.get(); simgrid::mc::Snapshot* s2 = state2->system_state.get(); int num1 = state1->num; int num2 = state2->num; return snapshot_compare(num1, s1, num2, s2); } bool SafetyChecker::checkNonTermination(simgrid::mc::State* current_state) { for (auto i = stack_.rbegin(); i != stack_.rend(); ++i) if (snapshot_compare(i->get(), current_state) == 0){ XBT_INFO("Non-progressive cycle : state %d -> state %d", (*i)->num, current_state->num); return true; } return false; } RecordTrace SafetyChecker::getRecordTrace() // override { RecordTrace res; for (auto const& state : stack_) res.push_back(state->getTransition()); return res; } std::vector SafetyChecker::getTextualTrace() // override { std::vector trace; for (auto const& state : stack_) { int value = state->transition.argument; smx_simcall_t req = &state->executed_req; if (req) trace.push_back(simgrid::mc::request_to_string( req, value, simgrid::mc::RequestType::executed)); } return trace; } void SafetyChecker::logState() // override { Checker::logState(); XBT_INFO("Expanded states = %lu", expandedStatesCount_); XBT_INFO("Visited states = %lu", mc_model_checker->visited_states); XBT_INFO("Executed transitions = %lu", mc_model_checker->executed_transitions); } int SafetyChecker::run() { this->init(); while (!stack_.empty()) { /* Get current state */ simgrid::mc::State* state = stack_.back().get(); XBT_DEBUG("**************************************************"); XBT_DEBUG( "Exploration depth=%zi (state=%p, num %d)(%zu interleave)", stack_.size(), state, state->num, state->interleaveSize()); mc_model_checker->visited_states++; // The interleave set is empty or the maximum depth is reached, // let's back-track. smx_simcall_t req = nullptr; if (stack_.size() > (std::size_t) _sg_mc_max_depth || (req = MC_state_get_request(state)) == nullptr || visitedState_ != nullptr) { int res = this->backtrack(); if (res) return res; else continue; } // If there are processes to interleave and the maximum depth has not been // reached then perform one step of the exploration algorithm. XBT_DEBUG("Execute: %s", simgrid::mc::request_to_string( req, state->transition.argument, simgrid::mc::RequestType::simix).c_str()); std::string req_str; if (dot_output != nullptr) req_str = simgrid::mc::request_get_dot_output(req, state->transition.argument); mc_model_checker->executed_transitions++; /* Answer the request */ this->getSession().execute(state->transition); /* Create the new expanded state */ std::unique_ptr next_state = std::unique_ptr(MC_state_new(++expandedStatesCount_)); if (_sg_mc_termination && this->checkNonTermination(next_state.get())) { MC_show_non_termination(); return SIMGRID_MC_EXIT_NON_TERMINATION; } if (_sg_mc_visited == 0 || (visitedState_ = visitedStates_.addVisitedState(expandedStatesCount_, next_state.get(), true)) == nullptr) { /* Get an enabled process and insert it in the interleave set of the next state */ for (auto& p : mc_model_checker->process().simix_processes()) if (simgrid::mc::process_is_enabled(p.copy.getBuffer())) { next_state->interleave(p.copy.getBuffer()); if (reductionMode_ != simgrid::mc::ReductionMode::none) break; } if (dot_output != nullptr) std::fprintf(dot_output, "\"%d\" -> \"%d\" [%s];\n", state->num, next_state->num, req_str.c_str()); } else if (dot_output != nullptr) std::fprintf(dot_output, "\"%d\" -> \"%d\" [%s];\n", state->num, visitedState_->other_num == -1 ? visitedState_->num : visitedState_->other_num, req_str.c_str()); stack_.push_back(std::move(next_state)); } XBT_INFO("No property violation found."); simgrid::mc::session->logState(); return SIMGRID_MC_EXIT_SUCCESS; } int SafetyChecker::backtrack() { if (stack_.size() > (std::size_t) _sg_mc_max_depth || visitedState_ != nullptr) { if (visitedState_ == nullptr) XBT_WARN("/!\\ Max depth reached ! /!\\ "); else XBT_DEBUG("State already visited (equal to state %d)," " exploration stopped on this path.", visitedState_->other_num == -1 ? visitedState_->num : visitedState_->other_num); } else XBT_DEBUG("There are no more processes to interleave. (depth %zi)", stack_.size() + 1); stack_.pop_back(); visitedState_ = nullptr; /* Check for deadlocks */ if (mc_model_checker->checkDeadlock()) { MC_show_deadlock(); return SIMGRID_MC_EXIT_DEADLOCK; } /* Traverse the stack backwards until a state with a non empty interleave set is found, deleting all the states that have it empty in the way. For each deleted state, check if the request that has generated it (from it's predecesor state), depends on any other previous request executed before it. If it does then add it to the interleave set of the state that executed that previous request. */ while (!stack_.empty()) { std::unique_ptr state = std::move(stack_.back()); stack_.pop_back(); if (reductionMode_ == simgrid::mc::ReductionMode::dpor) { smx_simcall_t req = &state->internal_req; if (req->call == SIMCALL_MUTEX_LOCK || req->call == SIMCALL_MUTEX_TRYLOCK) xbt_die("Mutex is currently not supported with DPOR, " "use --cfg=model-check/reduction:none"); const smx_actor_t issuer = MC_smx_simcall_get_issuer(req); for (auto i = stack_.rbegin(); i != stack_.rend(); ++i) { simgrid::mc::State* prev_state = i->get(); if (reductionMode_ != simgrid::mc::ReductionMode::none && simgrid::mc::request_depend(req, &prev_state->internal_req)) { if (XBT_LOG_ISENABLED(mc_safety, xbt_log_priority_debug)) { XBT_DEBUG("Dependent Transitions:"); int value = prev_state->transition.argument; smx_simcall_t prev_req = &prev_state->executed_req; XBT_DEBUG("%s (state=%d)", simgrid::mc::request_to_string( prev_req, value, simgrid::mc::RequestType::internal).c_str(), prev_state->num); value = state->transition.argument; prev_req = &state->executed_req; XBT_DEBUG("%s (state=%d)", simgrid::mc::request_to_string( prev_req, value, simgrid::mc::RequestType::executed).c_str(), state->num); } if (!prev_state->processStates[issuer->pid].isDone()) prev_state->interleave(issuer); else XBT_DEBUG("Process %p is in done set", req->issuer); break; } else if (req->issuer == prev_state->internal_req.issuer) { XBT_DEBUG("Simcall %d and %d with same issuer", req->call, prev_state->internal_req.call); break; } else { const smx_actor_t previous_issuer = MC_smx_simcall_get_issuer(&prev_state->internal_req); XBT_DEBUG("Simcall %d, process %lu (state %d) and simcall %d, process %lu (state %d) are independant", req->call, issuer->pid, state->num, prev_state->internal_req.call, previous_issuer->pid, prev_state->num); } } } if (state->interleaveSize() && stack_.size() < (std::size_t) _sg_mc_max_depth) { /* We found a back-tracking point, let's loop */ XBT_DEBUG("Back-tracking to state %d at depth %zi", state->num, stack_.size() + 1); stack_.push_back(std::move(state)); this->restoreState(); XBT_DEBUG("Back-tracking to state %d at depth %zi done", stack_.back()->num, stack_.size()); break; } else { XBT_DEBUG("Delete state %d at depth %zi", state->num, stack_.size() + 1); } } return SIMGRID_MC_EXIT_SUCCESS; } void SafetyChecker::restoreState() { /* Intermediate backtracking */ { simgrid::mc::State* state = stack_.back().get(); if (state->system_state) { simgrid::mc::restore_snapshot(state->system_state); return; } } /* Restore the initial state */ simgrid::mc::session->restoreInitialState(); /* Traverse the stack from the state at position start and re-execute the transitions */ for (std::unique_ptr const& state : stack_) { if (state == stack_.back()) break; session->execute(state->transition); /* Update statistics */ mc_model_checker->visited_states++; mc_model_checker->executed_transitions++; } } void SafetyChecker::init() { reductionMode_ = simgrid::mc::reduction_mode; if( _sg_mc_termination) reductionMode_ = simgrid::mc::ReductionMode::none; else if (reductionMode_ == simgrid::mc::ReductionMode::unset) reductionMode_ = simgrid::mc::ReductionMode::dpor; if (_sg_mc_termination) XBT_INFO("Check non progressive cycles"); else XBT_INFO("Check a safety property"); simgrid::mc::session->initialize(); XBT_DEBUG("Starting the safety algorithm"); std::unique_ptr initial_state = std::unique_ptr(MC_state_new(++expandedStatesCount_)); XBT_DEBUG("**************************************************"); XBT_DEBUG("Initial state"); /* Get an enabled process and insert it in the interleave set of the initial state */ for (auto& p : mc_model_checker->process().simix_processes()) if (simgrid::mc::process_is_enabled(p.copy.getBuffer())) { initial_state->interleave(p.copy.getBuffer()); if (reductionMode_ != simgrid::mc::ReductionMode::none) break; } stack_.push_back(std::move(initial_state)); } SafetyChecker::SafetyChecker(Session& session) : Checker(session) { } SafetyChecker::~SafetyChecker() { } Checker* createSafetyChecker(Session& session) { return new SafetyChecker(session); } } } SimGrid-3.14.159/src/mc/mc_base.h000644 001750 001750 00000002703 13031031270 020020 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_BASE_H #define SIMGRID_MC_BASE_H #ifdef __cplusplus #include #endif #include #include #ifdef __cplusplus namespace simgrid { namespace mc { /** Can this requests can be executed? * * Most requests are always enabled but WAIT and WAITANY * are not always enabled: a WAIT where the communication does not * have both a source and a destination yet is not enabled * (unless timeout is enabled in the wait and enabeld in SimGridMC). */ XBT_PRIVATE bool request_is_enabled(smx_simcall_t req); /** Execute everything which is invisible * * Execute all the processes that are ready to run and all invisible simcalls * iteratively until there doesn't remain any. At this point, the function * returns to the caller which can handle the visible (and ready) simcalls. */ XBT_PRIVATE void wait_for_requests(); XBT_PRIVATE extern std::vector processes_time; /** Execute a given simcall */ XBT_PRIVATE void handle_simcall(smx_simcall_t req, int req_num); /** Check if the given simcall is visible * * \return `TRUE` or `FALSE` */ XBT_PRIVATE bool request_is_visible(smx_simcall_t req); } } #endif #endif SimGrid-3.14.159/src/mc/mc_member.cpp000644 001750 001750 00000002500 13031031270 020703 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "src/mc/mc_private.h" #include "src/mc/Type.hpp" #include "src/mc/mc_dwarf.hpp" namespace simgrid { namespace dwarf { /** Resolve snapshot in the process address space * * @param object Process address of the struct/class * @param type Type of the struct/class * @param member Member description * @param snapshot Snapshot (or nullptr) * @return Process address of the given member of the 'object' struct/class */ void *resolve_member( const void *base, simgrid::mc::Type* type, simgrid::mc::Member* member, simgrid::mc::AddressSpace* address_space, int process_index) { // TODO, get rid of this? if (!member->has_offset_location()) return ((char *) base) + member->offset(); ExpressionContext state; state.frame_base = nullptr; state.cursor = nullptr; state.address_space = address_space; state.process_index = process_index; ExpressionStack stack; stack.push((ExpressionStack::value_type) base); simgrid::dwarf::execute(member->location_expression, state, stack); return (void*) stack.top(); } } } SimGrid-3.14.159/src/mc/mc_config.cpp000644 001750 001750 00000017015 13031031270 020710 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include "src/mc/mc_replay.h" #include #if HAVE_MC #include "src/mc/mc_safety.h" #include "src/mc/mc_private.h" #endif #include "src/mc/mc_record.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_config, mc, "Configuration of MC"); #if HAVE_MC namespace simgrid { namespace mc { /* Configuration support */ simgrid::mc::ReductionMode reduction_mode = simgrid::mc::ReductionMode::unset; } } #endif #if !HAVE_MC #define _sg_do_model_check 0 #endif int _sg_mc_timeout = 0; void _mc_cfg_cb_timeout(const char *name) { if (_sg_cfg_init_status && !(_sg_do_model_check || MC_record_path)) xbt_die("You are specifying a value to enable/disable timeout for wait requests after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_timeout = xbt_cfg_get_boolean(name); } #if HAVE_MC int _sg_do_model_check = 0; int _sg_do_model_check_record = 0; int _sg_mc_checkpoint = 0; int _sg_mc_sparse_checkpoint = 0; int _sg_mc_ksm = 0; char *_sg_mc_property_file = nullptr; int _sg_mc_hash = 0; int _sg_mc_max_depth = 1000; int _sg_mc_visited = 0; char *_sg_mc_dot_output_file = nullptr; int _sg_mc_comms_determinism = 0; int _sg_mc_send_determinism = 0; int _sg_mc_snapshot_fds = 0; int _sg_mc_termination = 0; void _mc_cfg_cb_reduce(const char *name) { if (_sg_cfg_init_status && !_sg_do_model_check) xbt_die ("You are specifying a reduction strategy after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); char *val = xbt_cfg_get_string(name); if (!strcasecmp(val, "none")) simgrid::mc::reduction_mode = simgrid::mc::ReductionMode::none; else if (!strcasecmp(val, "dpor")) simgrid::mc::reduction_mode = simgrid::mc::ReductionMode::dpor; else xbt_die("configuration option %s can only take 'none' or 'dpor' as a value", name); } void _mc_cfg_cb_checkpoint(const char *name) { if (_sg_cfg_init_status && !_sg_do_model_check) xbt_die ("You are specifying a checkpointing value after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_checkpoint = xbt_cfg_get_int(name); } void _mc_cfg_cb_sparse_checkpoint(const char *name) { if (_sg_cfg_init_status && !_sg_do_model_check) xbt_die("You are specifying a checkpointing value after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_sparse_checkpoint = xbt_cfg_get_boolean(name); } void _mc_cfg_cb_ksm(const char *name) { if (_sg_cfg_init_status && !_sg_do_model_check) xbt_die("You are specifying a KSM value after the initialization (through MSG_config?), but model-checking was not activated at config time (through --cfg=model-check:1). This won't work, sorry."); _sg_mc_ksm = xbt_cfg_get_boolean(name); } void _mc_cfg_cb_property(const char *name) { if (_sg_cfg_init_status && !_sg_do_model_check) xbt_die ("You are specifying a property after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_property_file = xbt_cfg_get_string(name); } void _mc_cfg_cb_hash(const char *name) { if (_sg_cfg_init_status && !_sg_do_model_check) xbt_die ("You are specifying a value to enable/disable the use of global hash to speedup state comparaison, but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_hash = xbt_cfg_get_boolean(name); } void _mc_cfg_cb_snapshot_fds(const char *name) { if (_sg_cfg_init_status && !_sg_do_model_check) xbt_die ("You are specifying a value to enable/disable the use of FD snapshotting, but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_snapshot_fds = xbt_cfg_get_boolean(name); } void _mc_cfg_cb_max_depth(const char *name) { if (_sg_cfg_init_status && !_sg_do_model_check) xbt_die ("You are specifying a max depth value after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_max_depth = xbt_cfg_get_int(name); } void _mc_cfg_cb_visited(const char *name) { if (_sg_cfg_init_status && !_sg_do_model_check) xbt_die ("You are specifying a number of stored visited states after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_visited = xbt_cfg_get_int(name); } void _mc_cfg_cb_dot_output(const char *name) { if (_sg_cfg_init_status && !_sg_do_model_check) xbt_die ("You are specifying a file name for a dot output of graph state after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_dot_output_file = xbt_cfg_get_string(name); } void _mc_cfg_cb_comms_determinism(const char *name) { if (_sg_cfg_init_status && !_sg_do_model_check) xbt_die ("You are specifying a value to enable/disable the detection of determinism in the communications schemes after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_comms_determinism = xbt_cfg_get_boolean(name); } void _mc_cfg_cb_send_determinism(const char *name) { if (_sg_cfg_init_status && !_sg_do_model_check) xbt_die ("You are specifying a value to enable/disable the detection of send-determinism in the communications schemes after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_send_determinism = xbt_cfg_get_boolean(name); } void _mc_cfg_cb_termination(const char *name) { if (_sg_cfg_init_status && !_sg_do_model_check) xbt_die ("You are specifying a value to enable/disable the detection of non progressive cycles after the initialization (through MSG_config?), but model-checking was not activated at config time (through bu the program was not runned under the model-checker (with simgrid-mc)). This won't work, sorry."); _sg_mc_termination = xbt_cfg_get_boolean(name); } #endif SimGrid-3.14.159/src/mc/mc_protocol.cpp000644 001750 001750 00000002606 13031031271 021305 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include // perror #include // std::size_t #include #include #include #include "src/mc/mc_protocol.h" #include "src/mc/Client.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_protocol, mc, "Generic MC protocol logic"); const char* MC_message_type_name(e_mc_message_type type) { switch(type) { case MC_MESSAGE_NONE: return "NONE"; case MC_MESSAGE_CONTINUE: return "CONTINUE"; case MC_MESSAGE_IGNORE_HEAP: return "IGNORE_HEAP"; case MC_MESSAGE_UNIGNORE_HEAP: return "UNIGNORE_HEAP"; case MC_MESSAGE_IGNORE_MEMORY: return "IGNORE_MEMORY"; case MC_MESSAGE_STACK_REGION: return "STACK_REGION"; case MC_MESSAGE_REGISTER_SYMBOL: return "REGISTER_SYMBOL"; case MC_MESSAGE_DEADLOCK_CHECK: return "DEADLOCK_CHECK"; case MC_MESSAGE_DEADLOCK_CHECK_REPLY: return "DEADLOCK_CHECK_REPLY"; case MC_MESSAGE_WAITING: return "WAITING"; case MC_MESSAGE_SIMCALL_HANDLE: return "SIMCALL_HANDLE"; case MC_MESSAGE_ASSERTION_FAILED: return "ASSERTION_FAILED"; default: return "?"; } } SimGrid-3.14.159/src/mc/mc_hash.hpp000644 001750 001750 00000001065 13031031270 020371 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_HASH_HPP #define SIMGRID_MC_HASH_HPP #include #include #include "src/mc/mc_forward.hpp" namespace simgrid { namespace mc { typedef std::uint64_t hash_type; XBT_PRIVATE hash_type hash(simgrid::mc::Snapshot const& snapshot); } } #endif SimGrid-3.14.159/src/mc/Client.hpp000644 001750 001750 00000003222 13031031270 020202 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_CLIENT_H #define SIMGRID_MC_CLIENT_H #include "src/internal_config.h" #include #include #include #include #include "src/mc/mc_protocol.h" #include "src/mc/Channel.hpp" namespace simgrid { namespace mc { /** Model-checked-side of the communication protocol * * Send messages to the model-checker and handles message from it. */ class XBT_PUBLIC() Client { private: bool active_ = false; Channel channel_; static std::unique_ptr client_; public: Client(); explicit Client(int fd) : active_(true), channel_(fd) {} void handleMessages(); Channel const& getChannel() const { return channel_; } Channel& getChannel() { return channel_; } void mainLoop(); void reportAssertionFailure(const char* description = nullptr); void ignoreMemory(void* addr, std::size_t size); void ignoreHeap(void* addr, std::size_t size); void unignoreHeap(void* addr, std::size_t size); void declareSymbol(const char *name, int* value); #if HAVE_UCONTEXT_H void declareStack(void *stack, size_t size, smx_actor_t process, ucontext_t* context); #endif // Singleton :/ // TODO, remove the singleton antipattern. static Client* initialize(); static Client* get() { return client_.get(); } }; } } SG_BEGIN_DECL() #if HAVE_MC void MC_ignore(void* addr, std::size_t size); #endif SG_END_DECL() #endif SimGrid-3.14.159/src/mc/AddressSpace.cpp000644 001750 001750 00000000610 13031031270 021316 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "src/mc/AddressSpace.hpp" namespace simgrid { namespace mc { AddressSpace::~AddressSpace() { } } } SimGrid-3.14.159/src/mc/Client.cpp000644 001750 001750 00000016714 13031031270 020207 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include #include #include #include #include "src/internal_config.h" #include "src/mc/mc_protocol.h" #include "src/mc/Client.hpp" #include "src/mc/mc_request.h" // We won't need those once the separation MCer/MCed is complete: #include "src/mc/mc_ignore.h" #include "src/mc/mc_smx.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_client, mc, "MC client logic"); namespace simgrid { namespace mc { std::unique_ptr Client::client_; Client* Client::initialize() { // We are not in MC mode: // TODO, handle this more gracefully. if (!getenv(MC_ENV_SOCKET_FD)) return nullptr; // Do not break if we are called multiple times: if (client_) return client_.get(); _sg_do_model_check = 1; // Fetch socket from MC_ENV_SOCKET_FD: char* fd_env = std::getenv(MC_ENV_SOCKET_FD); if (!fd_env) xbt_die("No MC socket passed in the environment"); int fd = xbt_str_parse_int(fd_env, bprintf("Variable %s should contain a number but contains '%%s'", MC_ENV_SOCKET_FD)); XBT_DEBUG("Model-checked application found socket FD %i", fd); // Check the socket type/validity: int type; socklen_t socklen = sizeof(type); if (getsockopt(fd, SOL_SOCKET, SO_TYPE, &type, &socklen) != 0) xbt_die("Could not check socket type"); if (type != SOCK_DGRAM) xbt_die("Unexpected socket type %i", type); XBT_DEBUG("Model-checked application found expected socket type"); client_ = std::unique_ptr(new simgrid::mc::Client(fd)); // Wait for the model-checker: errno = 0; #if defined __linux__ ptrace(PTRACE_TRACEME, 0, nullptr, nullptr); #elif defined BSD ptrace(PT_TRACE_ME, 0, nullptr, 0); #else # error "no ptrace equivalent coded for this platform" #endif if(errno != 0 || raise(SIGSTOP) != 0) xbt_die("Could not wait for the model-checker"); client_->handleMessages(); return client_.get(); } void Client::handleMessages() { while (1) { XBT_DEBUG("Waiting messages from model-checker"); char message_buffer[MC_MESSAGE_LENGTH]; ssize_t s; if ((s = channel_.receive(&message_buffer, sizeof(message_buffer))) < 0) xbt_die("Could not receive commands from the model-checker"); s_mc_message_t message; if ((size_t) s < sizeof(message)) xbt_die("Received message is too small"); memcpy(&message, message_buffer, sizeof(message)); switch (message.type) { case MC_MESSAGE_DEADLOCK_CHECK: { // Check deadlock: bool deadlock = false; smx_actor_t process; if (xbt_swag_size(simix_global->process_list)) { deadlock = true; xbt_swag_foreach(process, simix_global->process_list) if (simgrid::mc::process_is_enabled(process)) { deadlock = false; break; } } // Send result: s_mc_int_message_t answer; answer.type = MC_MESSAGE_DEADLOCK_CHECK_REPLY; answer.value = deadlock; if (channel_.send(answer)) xbt_die("Could not send response"); } break; case MC_MESSAGE_CONTINUE: return; case MC_MESSAGE_SIMCALL_HANDLE: { s_mc_simcall_handle_message_t message; if (s != sizeof(message)) xbt_die("Unexpected size for SIMCALL_HANDLE"); memcpy(&message, message_buffer, sizeof(message)); smx_actor_t process = SIMIX_process_from_PID(message.pid); if (!process) xbt_die("Invalid pid %lu", (unsigned long) message.pid); SIMIX_simcall_handle(&process->simcall, message.value); if (channel_.send(MC_MESSAGE_WAITING)) xbt_die("Could not send MESSAGE_WAITING to model-checker"); } break; case MC_MESSAGE_RESTORE: { s_mc_restore_message_t message; if (s != sizeof(message)) xbt_die("Unexpected size for SIMCALL_HANDLE"); memcpy(&message, message_buffer, sizeof(message)); #if HAVE_SMPI smpi_really_switch_data_segment(message.index); #endif } break; default: xbt_die("Received unexpected message %s (%i)", MC_message_type_name(message.type), message.type ); } } } void Client::mainLoop(void) { while (1) { simgrid::mc::wait_for_requests(); if (channel_.send(MC_MESSAGE_WAITING)) xbt_die("Could not send WAITING mesage to model-checker"); this->handleMessages(); } } void Client::reportAssertionFailure(const char* description) { if (channel_.send(MC_MESSAGE_ASSERTION_FAILED)) xbt_die("Could not send assertion to model-checker"); this->handleMessages(); } void Client::ignoreMemory(void* addr, std::size_t size) { s_mc_ignore_memory_message_t message; message.type = MC_MESSAGE_IGNORE_MEMORY; message.addr = (std::uintptr_t) addr; message.size = size; if (channel_.send(message)) xbt_die("Could not send IGNORE_MEMORY mesage to model-checker"); } void Client::ignoreHeap(void* address, std::size_t size) { xbt_mheap_t heap = mmalloc_get_current_heap(); s_mc_ignore_heap_message_t message; message.type = MC_MESSAGE_IGNORE_HEAP; message.address = address; message.size = size; message.block = ((char *) address - (char *) heap->heapbase) / BLOCKSIZE + 1; if (heap->heapinfo[message.block].type == 0) { message.fragment = -1; heap->heapinfo[message.block].busy_block.ignore++; } else { message.fragment = ((uintptr_t) (ADDR2UINT(address) % (BLOCKSIZE))) >> heap->heapinfo[message.block].type; heap->heapinfo[message.block].busy_frag.ignore[message.fragment]++; } if (channel_.send(message)) xbt_die("Could not send ignored region to MCer"); } void Client::unignoreHeap(void* address, std::size_t size) { s_mc_ignore_memory_message_t message; message.type = MC_MESSAGE_UNIGNORE_HEAP; message.addr = (std::uintptr_t) address; message.size = size; if (channel_.send(message)) xbt_die("Could not send IGNORE_HEAP mesasge to model-checker"); } void Client::declareSymbol(const char *name, int* value) { s_mc_register_symbol_message_t message; message.type = MC_MESSAGE_REGISTER_SYMBOL; if (strlen(name) + 1 > sizeof(message.name)) xbt_die("Symbol is too long"); strncpy(message.name, name, sizeof(message.name)); message.callback = nullptr; message.data = value; if (channel_.send(message)) xbt_die("Could send REGISTER_SYMBOL message to model-checker"); } void Client::declareStack(void *stack, size_t size, smx_actor_t process, ucontext_t* context) { xbt_mheap_t heap = mmalloc_get_current_heap(); s_stack_region_t region; memset(®ion, 0, sizeof(region)); region.address = stack; region.context = context; region.size = size; region.block = ((char *) stack - (char *) heap->heapbase) / BLOCKSIZE + 1; #if HAVE_SMPI if (smpi_privatize_global_variables && process) region.process_index = smpi_process_index_of_smx_process(process); else #endif region.process_index = -1; s_mc_stack_region_message_t message; message.type = MC_MESSAGE_STACK_REGION; message.stack_region = region; if (channel_.send(message)) xbt_die("Coule not send STACK_REGION to model-checker"); } } } SimGrid-3.14.159/src/mc/VisitedState.cpp000644 001750 001750 00000010514 13031031270 021371 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2011-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include "src/mc/mc_comm_pattern.h" #include "src/mc/mc_safety.h" #include "src/mc/mc_private.h" #include "src/mc/Process.hpp" #include "src/mc/mc_smx.h" #include "src/mc/VisitedState.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_VisitedState, mc, "Logging specific to state equaity detection mechanisms"); namespace simgrid { namespace mc { static int snapshot_compare(simgrid::mc::VisitedState* state1, simgrid::mc::VisitedState* state2) { simgrid::mc::Snapshot* s1 = state1->system_state.get(); simgrid::mc::Snapshot* s2 = state2->system_state.get(); int num1 = state1->num; int num2 = state2->num; return snapshot_compare(num1, s1, num2, s2); } /** * \brief Save the current state * \return Snapshot of the current state. */ VisitedState::VisitedState(unsigned long state_number) { simgrid::mc::Process* process = &(mc_model_checker->process()); this->heap_bytes_used = mmalloc_get_bytes_used_remote( process->get_heap()->heaplimit, process->get_malloc_info()); this->nb_processes = mc_model_checker->process().simix_processes().size(); this->system_state = simgrid::mc::take_snapshot(state_number); this->num = state_number; this->other_num = -1; } VisitedState::~VisitedState() { } void VisitedStates::prune() { while (states_.size() > (std::size_t) _sg_mc_visited) { XBT_DEBUG("Try to remove visited state (maximum number of stored states reached)"); auto min_element = boost::range::min_element(states_, [](std::unique_ptr& a, std::unique_ptr& b) { return a->num < b->num; }); xbt_assert(min_element != states_.end()); // and drop it: states_.erase(min_element); XBT_DEBUG("Remove visited state (maximum number of stored states reached)"); } } /** * \brief Checks whether a given state has already been visited by the algorithm. */ std::unique_ptr VisitedStates::addVisitedState( unsigned long state_number, simgrid::mc::State* graph_state, bool compare_snpashots) { std::unique_ptr new_state = std::unique_ptr(new VisitedState(state_number)); graph_state->system_state = new_state->system_state; XBT_DEBUG("Snapshot %p of visited state %d (exploration stack state %d)", new_state->system_state.get(), new_state->num, graph_state->num); auto range = boost::range::equal_range(states_, new_state.get(), simgrid::mc::DerefAndCompareByNbProcessesAndUsedHeap()); if (compare_snpashots) for (auto i = range.first; i != range.second; ++i) { auto& visited_state = *i; if (snapshot_compare(visited_state.get(), new_state.get()) == 0) { // The state has been visited: std::unique_ptr old_state = std::move(visited_state); if (old_state->other_num == -1) new_state->other_num = old_state->num; else new_state->other_num = old_state->other_num; if (dot_output == nullptr) XBT_DEBUG("State %d already visited ! (equal to state %d)", new_state->num, old_state->num); else XBT_DEBUG( "State %d already visited ! (equal to state %d (state %d in dot_output))", new_state->num, old_state->num, new_state->other_num); /* Replace the old state with the new one (with a bigger num) (when the max number of visited states is reached, the oldest one is removed according to its number (= with the min number) */ XBT_DEBUG("Replace visited state %d with the new visited state %d", old_state->num, new_state->num); visited_state = std::move(new_state); return old_state; } } XBT_DEBUG("Insert new visited state %d (total : %lu)", new_state->num, (unsigned long) states_.size()); states_.insert(range.first, std::move(new_state)); this->prune(); return nullptr; } } } SimGrid-3.14.159/src/mc/Variable.hpp000644 001750 001750 00000002304 13031031270 020511 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_VARIABLE_HPP #define SIMGRID_MC_VARIABLE_HPP #include #include #include "src/mc/mc_forward.hpp" #include "src/mc/LocationList.hpp" namespace simgrid { namespace mc { /** A variable (global or local) in the model-checked program */ class Variable { public: Variable() {} std::uint32_t id = 0; bool global = false; std::string name; unsigned type_id = 0; simgrid::mc::Type* type = nullptr; /** Address of the variable (if it is fixed) */ void* address = nullptr; /** Description of the location of the variable (if it's not fixed) */ simgrid::dwarf::LocationList location_list; /** Offset of validity of the variable (DW_AT_start_scope) * * This is an offset from the variable scope beginning. This variable * is only valid starting from this offset. */ std::size_t start_scope = 0; simgrid::mc::ObjectInformation* object_info = nullptr; }; } } #endif SimGrid-3.14.159/src/mc/mc_hash.cpp000644 001750 001750 00000002206 13031031270 020362 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include "src/mc/mc_private.h" #include "src/mc/mc_snapshot.h" #include "mc/datatypes.h" #include "src/mc/mc_hash.hpp" #include XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_hash, mc, "Logging specific to mc_hash"); namespace simgrid { namespace mc { namespace { class djb_hash { hash_type state_ = 5381ll; public: template void update(T& x) { state_ = (state_ << 5) + state_ + state_; } hash_type value() { return state_; } }; } hash_type hash(Snapshot const& snapshot) { XBT_DEBUG("START hash %i", snapshot.num_state); djb_hash hash; // TODO, nb_processes // TODO, heap_bytes_used // TODO, root variables // TODO, basic stack frame information // TODO, stack frame local variables XBT_DEBUG("END hash %i", snapshot.num_state); return hash.value(); } } } SimGrid-3.14.159/src/mc/Session.cpp000644 001750 001750 00000011121 13031031270 020377 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2015-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include #include #include "src/mc/Session.hpp" #include "src/mc/mc_state.h" #include "src/mc/mc_private.h" #include "src/mc/Checker.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_Session, mc, "Model-checker session"); namespace simgrid { namespace mc { static void setup_child_environment(int socket) { #ifdef __linux__ // Make sure we do not outlive our parent: sigset_t mask; sigemptyset (&mask); if (sigprocmask(SIG_SETMASK, &mask, nullptr) < 0) throw simgrid::xbt::errno_error("Could not unblock signals"); if (prctl(PR_SET_PDEATHSIG, SIGHUP) != 0) throw simgrid::xbt::errno_error("Could not PR_SET_PDEATHSIG"); #endif int res; // Remove CLOEXEC in order to pass the socket to the exec-ed program: int fdflags = fcntl(socket, F_GETFD, 0); if (fdflags == -1 || fcntl(socket, F_SETFD, fdflags & ~FD_CLOEXEC) == -1) throw simgrid::xbt::errno_error("Could not remove CLOEXEC for socket"); // Set environment: setenv(MC_ENV_VARIABLE, "1", 1); // Disable lazy relocation in the model-checked process. // We don't want the model-checked process to modify its .got.plt during // snapshot. setenv("LC_BIND_NOW", "1", 1); char buffer[64]; res = std::snprintf(buffer, sizeof(buffer), "%i", socket); if ((size_t) res >= sizeof(buffer) || res == -1) std::abort(); setenv(MC_ENV_SOCKET_FD, buffer, 1); } /** Execute some code in a forked process */ template static inline pid_t do_fork(F code) { pid_t pid = fork(); if (pid < 0) throw simgrid::xbt::errno_error("Could not fork model-checked process"); if (pid != 0) return pid; // Child-process: try { code(); _exit(EXIT_SUCCESS); } catch(...) { // The callback should catch exceptions: std::terminate(); } } Session::Session(pid_t pid, int socket) { std::unique_ptr process(new simgrid::mc::Process(pid, socket)); // TODO, automatic detection of the config from the process process->privatized( xbt_cfg_get_boolean("smpi/privatize-global-variables")); modelChecker_ = std::unique_ptr( new simgrid::mc::ModelChecker(std::move(process))); xbt_assert(mc_model_checker == nullptr); mc_model_checker = modelChecker_.get(); mc_model_checker->start(); } Session::~Session() { this->close(); } void Session::initialize() { xbt_assert(initialSnapshot_ == nullptr); mc_model_checker->wait_for_requests(); initialSnapshot_ = simgrid::mc::take_snapshot(0); } void Session::execute(Transition const& transition) { modelChecker_->handle_simcall(transition); modelChecker_->wait_for_requests(); } void Session::restoreInitialState() { simgrid::mc::restore_snapshot(this->initialSnapshot_); } void Session::logState() { mc_model_checker->getChecker()->logState(); if ((_sg_mc_dot_output_file != nullptr) && (_sg_mc_dot_output_file[0] != '\0')) { fprintf(dot_output, "}\n"); fclose(dot_output); } if (getenv("SIMGRID_MC_SYSTEM_STATISTICS")){ int ret=system("free"); if(ret!=0)XBT_WARN("system call did not return 0, but %d",ret); } } // static Session* Session::fork(std::function code) { // Create a AF_LOCAL socketpair used for exchanging messages // bewteen the model-checker process (ourselves) and the model-checked // process: int res; int sockets[2]; res = socketpair(AF_LOCAL, SOCK_DGRAM | SOCK_CLOEXEC, 0, sockets); if (res == -1) throw simgrid::xbt::errno_error("Could not create socketpair"); pid_t pid = do_fork([&] { ::close(sockets[1]); setup_child_environment(sockets[0]); code(); xbt_die("The model-checked process failed to exec()"); }); // Parent (model-checker): ::close(sockets[0]); return new Session(pid, sockets[1]); } // static Session* Session::spawnv(const char *path, char *const argv[]) { return Session::fork([&] { execv(path, argv); }); } // static Session* Session::spawnvp(const char *file, char *const argv[]) { return Session::fork([&] { execvp(file, argv); }); } void Session::close() { initialSnapshot_ = nullptr; if (modelChecker_) { modelChecker_->shutdown(); modelChecker_ = nullptr; mc_model_checker = nullptr; } } simgrid::mc::Session* session; } } SimGrid-3.14.159/src/mc/mc_mmu.h000644 001750 001750 00000003445 13031031273 017713 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_MMU_H #define SIMGRID_MC_MMU_H #include #include #include #include // xbt_pagesize... #include #include namespace simgrid { namespace mc { // TODO, do not depend on xbt_pagesize/xbt_pagebits but our own chunk size namespace mmu { static int chunkSize() { return xbt_pagesize; } /** @brief How many memory pages are necessary to store size bytes? * * @param size Byte size * @return Number of memory pages */ static inline __attribute__ ((always_inline)) std::size_t chunkCount(std::size_t size) { size_t page_count = size >> xbt_pagebits; if (size & (xbt_pagesize-1)) page_count ++; return page_count; } /** @brief Split into chunk number and remaining offset */ static inline __attribute__ ((always_inline)) std::pair split(std::uintptr_t offset) { return { offset >> xbt_pagebits, offset & (xbt_pagesize-1) }; } /** Merge chunk number and remaining offset info a global offset */ static inline __attribute__ ((always_inline)) std::uintptr_t join(std::size_t page, std::uintptr_t offset) { return ((std::uintptr_t) page << xbt_pagebits) + offset; } static inline __attribute__ ((always_inline)) std::uintptr_t join(std::pair value) { return join(value.first, value.second); } static inline __attribute__ ((always_inline)) bool sameChunk(std::uintptr_t a, std::uintptr_t b) { return (a >> xbt_pagebits) == (b >> xbt_pagebits); } } } } #endif SimGrid-3.14.159/src/mc/simgrid_mc.cpp000644 001750 001750 00000005047 13031031271 021104 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include #include "simgrid/sg_config.h" #include "src/xbt_modinter.h" #include "src/mc/mc_base.h" #include "src/mc/mc_private.h" #include "src/mc/mc_protocol.h" #include "src/mc/mc_safety.h" #include "src/mc/mc_comm_pattern.h" #include "src/mc/mc_exit.h" #include "src/mc/Session.hpp" #include "src/mc/Checker.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_main, mc, "Entry point for simgrid-mc"); static inline char** argvdup(int argc, char** argv) { char** argv_copy = xbt_new(char*, argc+1); std::memcpy(argv_copy, argv, sizeof(char*) * argc); argv_copy[argc] = nullptr; return argv_copy; } static std::unique_ptr createChecker(simgrid::mc::Session& session) { if (_sg_mc_comms_determinism || _sg_mc_send_determinism) return std::unique_ptr( simgrid::mc::createCommunicationDeterminismChecker(session)); else if (!_sg_mc_property_file || _sg_mc_property_file[0] == '\0') return std::unique_ptr( simgrid::mc::createSafetyChecker(session)); else return std::unique_ptr( simgrid::mc::createLivenessChecker(session)); } int main(int argc, char** argv) { using simgrid::mc::Session; try { if (argc < 2) xbt_die("Missing arguments.\n"); // Currently, we need this before sg_config_init: _sg_do_model_check = 1; // The initialisation function can touch argv. // We make a copy of argv before modifying it in order to pass the original // value to the model-checked: char** argv_copy = argvdup(argc, argv); xbt_log_init(&argc, argv); sg_config_init(&argc, argv); std::unique_ptr session = std::unique_ptr(Session::spawnvp(argv_copy[1], argv_copy+1)); free(argv_copy); simgrid::mc::session = session.get(); std::unique_ptr checker = createChecker(*session); int res = checker->run(); checker = nullptr; session->close(); return res; } catch(std::exception& e) { XBT_ERROR("Exception: %s", e.what()); return SIMGRID_MC_EXIT_ERROR; } catch(...) { XBT_ERROR("Unknown exception"); return SIMGRID_MC_EXIT_ERROR; } } SimGrid-3.14.159/src/mc/DwarfExpression.hpp000644 001750 001750 00000010336 13031031270 022113 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_DWARF_EXPRESSION_HPP #define SIMGRID_MC_DWARF_EXPRESSION_HPP #include #include #include // runtime_error #include #include #include #include "src/mc/mc_forward.hpp" #include "src/mc/AddressSpace.hpp" /** @file DwarfExpression.hpp * * Evaluation of DWARF location expressions. */ namespace simgrid { namespace dwarf { /** A DWARF expression * * DWARF defines a simple stack-based VM for evaluating expressions * (such as locations of variables, etc.): a DWARF expression is * just a sequence of dwarf instructions. We currently directly use * `Dwarf_Op` from `dwarf.h` for dwarf instructions. */ typedef std::vector DwarfExpression; /** Context of evaluation of a DWARF expression * * Some DWARF instructions need to read the CPU registers, * the process memory, etc. All those informations are gathered in * the evaluation context. */ struct ExpressionContext { ExpressionContext() : cursor(nullptr), frame_base(nullptr), address_space(nullptr), object_info(nullptr), process_index(simgrid::mc::ProcessIndexMissing) {} /** CPU state (registers) */ unw_cursor_t* cursor; void* frame_base; /** Address space used to read memory */ simgrid::mc::AddressSpace* address_space; simgrid::mc::ObjectInformation* object_info; int process_index; }; /** When an error happens in the execution of a DWARF expression */ class evaluation_error : std::runtime_error { public: evaluation_error(const char* what): std::runtime_error(what) {} ~evaluation_error() noexcept(true); }; /** A stack for evaluating a DWARF expression * * DWARF expressions work by manipulating a stack of integer values. */ class ExpressionStack { public: typedef std::uintptr_t value_type; static const std::size_t max_size = 64; private: // Values of the stack (the top is stack_[size_ - 1]): uintptr_t stack_[max_size]; size_t size_; public: ExpressionStack() : size_(0) {} // Access: std::size_t size() const { return size_; } bool empty() const { return size_ == 0; } void clear() { size_ = 0; } uintptr_t& operator[](int i) { return stack_[i]; } uintptr_t const& operator[](int i) const { return stack_[i]; } /** Top of the stack */ value_type& top() { if (size_ == 0) throw evaluation_error("Empty stack"); return stack_[size_ - 1]; } /** Access the i-th element from the top of the stack */ value_type& top(unsigned i) { if (size_ < i) throw evaluation_error("Invalid element"); return stack_[size_ - 1 - i]; } /** Push a value on the top of the stack */ void push(value_type value) { if (size_ == max_size) throw evaluation_error("DWARF stack overflow"); stack_[size_++] = value; } /* Pop a value from the top of the stack */ value_type pop() { if (size_ == 0) throw evaluation_error("DWARF stack underflow"); return stack_[--size_]; } // These are DWARF operations (DW_OP_foo): /* Push a copy of the top-value (DW_OP_dup) */ void dup() { push(top()); } /* Swap the two top-most values */ void swap() { std::swap(top(), top(1)); } }; /** Executes a DWARF expression * * @param ops DWARF expression instructions * @param n number of instructions * @param context evaluation context (registers, memory, etc.) * @param stack DWARf stack where the operations are executed */ void execute(const Dwarf_Op* ops, std::size_t n, ExpressionContext const& context, ExpressionStack& stack); /** Executes/evaluates a DWARF expression * * @param expression DWARF expression to execute * @param context evaluation context (registers, memory, etc.) * @param stack DWARf stack where the operations are executed */ inline void execute(simgrid::dwarf::DwarfExpression const& expression, ExpressionContext const& context, ExpressionStack& stack) { execute(expression.data(), expression.size(), context, stack); } } } #endif SimGrid-3.14.159/src/mc/mc_replay.h000644 001750 001750 00000001250 13031031270 020376 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_REPLAY_H #define SIMGRID_MC_REPLAY_H #include SG_BEGIN_DECL() /** Replay path (if any) in string representation * * This is using the format generated by traceToString(). */ XBT_PUBLIC_DATA(char*) MC_record_path; /** Whether the replay mode is enabled */ static inline int MC_record_replay_is_active() { return MC_record_path != 0; } SG_END_DECL() #endif SimGrid-3.14.159/src/mc/Type.hpp000644 001750 001750 00000006023 13031031270 017707 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_TYPE_HPP #define SIMGRID_MC_TYPE_HPP #include #include #include #include #include #include #include "src/mc/mc_forward.hpp" #include "src/mc/LocationList.hpp" namespace simgrid { namespace mc { /** A member of a structure, union * * Inheritance is seen as a special member as well. */ class Member { public: typedef int flags_type; static constexpr flags_type INHERITANCE_FLAG = 1; static constexpr flags_type VIRTUAL_POINTER_FLAG = 2; Member() {} /** Whether this member represent some inherited part of the object */ flags_type flags = 0; /** Name of the member (if any) */ std::string name; /** DWARF location expression for locating the location of the member */ simgrid::dwarf::DwarfExpression location_expression; std::size_t byte_size = 0; // Do we really need this? unsigned type_id = 0; simgrid::mc::Type* type = nullptr; bool isInheritance() const { return this->flags & INHERITANCE_FLAG; } bool isVirtualPointer() const { return this->flags & VIRTUAL_POINTER_FLAG; } /** Whether the member is at a fixed offset from the base address */ bool has_offset_location() const { // Recognize the expression `DW_OP_plus_uconst(offset)`: return location_expression.size() == 1 && location_expression[0].atom == DW_OP_plus_uconst; } /** Get the offset of the member * * This is only valid is the member is at a fixed offset from the base. * This is often the case (for C types, C++ type without virtual * inheritance). * * If the location is more complex, the location expression has * to be evaluated (which might need accessing the memory). */ int offset() const { xbt_assert(this->has_offset_location()); return this->location_expression[0].number; } /** Set the location of the member as a fixed offset */ void offset(int new_offset) { // Set the expression to be `DW_OP_plus_uconst(offset)`: Dwarf_Op op; op.atom = DW_OP_plus_uconst; op.number = new_offset; this->location_expression = { op }; } }; /** A type in the model-checked program */ class Type { public: Type() {} /** The DWARF TAG of the type (e.g. DW_TAG_array_type) */ int type = 0; unsigned id = 0; /* Offset in the section (in hexadecimal form) */ std::string name; /* Name of the type */ int byte_size = 0; /* Size in bytes */ int element_count = 0; /* Number of elements for array type */ unsigned type_id = 0; /* DW_AT_type id */ std::vector members; /* if DW_TAG_structure_type, DW_TAG_class_type, DW_TAG_union_type*/ simgrid::mc::Type* subtype = nullptr; // DW_AT_type simgrid::mc::Type* full_type = nullptr; // The same (but more complete) type }; } } #endif SimGrid-3.14.159/src/mc/ObjectInformation.cpp000644 001750 001750 00000014775 13031031270 022412 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include "src/mc/Frame.hpp" #include "src/mc/ObjectInformation.hpp" #include "src/mc/Variable.hpp" namespace simgrid { namespace mc { ObjectInformation::ObjectInformation() {} /* For an executable object, addresses are virtual address * (there is no offset) i.e. * \f$\text{virtual address} = \{dwarf address}\f$; * * For a shared object, the addreses are offset from the begining * of the shared object (the base address of the mapped shared * object must be used as offset * i.e. \f$\text{virtual address} = \text{shared object base address} * + \text{dwarf address}\f$. */ void *ObjectInformation::base_address() const { // For an executable (more precisely for a ET_EXEC) the base it 0: if (this->executable()) return nullptr; // For an a shared-object (ET_DYN, including position-independant executables) // the base address is its lowest address: void *result = this->start_exec; if (this->start_rw != nullptr && result > (void *) this->start_rw) result = this->start_rw; if (this->start_ro != nullptr && result > (void *) this->start_ro) result = this->start_ro; return result; } simgrid::mc::Frame* ObjectInformation::find_function(const void *ip) const { /* This is implemented by binary search on a sorted array. * * We do quite a lot of those so we want this to be cache efficient. * We pack the only information we need in the index entries in order * to successfully do the binary search. We do not need the high_pc * during the binary search (only at the end) so it is not included * in the index entry. We could use parallel arrays as well. * * We cannot really use the std:: algorithm for this. * We could use std::binary_search by including the high_pc inside * the FunctionIndexEntry. */ const simgrid::mc::FunctionIndexEntry* base = this->functions_index.data(); int i = 0; int j = this->functions_index.size() - 1; while (j >= i) { int k = i + ((j - i) / 2); /* In most of the search, we do not dereference the base[k].function. * This way the memory accesses are located in the base[k] array. */ if (ip < base[k].low_pc) j = k - 1; else if (k < j && ip >= base[k + 1].low_pc) i = k + 1; /* At this point, the search is over. * Either we have found the correct function or we do not know * any function corresponding to this instruction address. * Only at the point do we dereference the function pointer. */ else if ((std::uint64_t) ip < base[k].function->range.end()) return base[k].function; else return nullptr; } return nullptr; } simgrid::mc::Variable* ObjectInformation::find_variable(const char* name) const { for (simgrid::mc::Variable& variable : this->global_variables) if(variable.name == name) return &variable; return nullptr; } void ObjectInformation::remove_global_variable(const char* name) { typedef std::vector::size_type size_type; if (this->global_variables.empty()) return; // Binary search: size_type first = 0; size_type last = this->global_variables.size() - 1; while (first <= last) { size_type cursor = first + (last - first) / 2; simgrid::mc::Variable& current_var = this->global_variables[cursor]; int cmp = current_var.name.compare(name); if (cmp == 0) { // Find the whole range: size_type first = cursor; while (first != 0 && this->global_variables[first - 1].name == name) first--; size_type size = this->global_variables.size(); size_type last = cursor; while (last != size - 1 && this->global_variables[last + 1].name == name) last++; // Remove the whole range: this->global_variables.erase( this->global_variables.begin() + first, this->global_variables.begin() + last + 1); return; } else if (cmp < 0) first = cursor + 1; else if (cursor != 0) last = cursor - 1; else break; } } /** Ignore a local variable in a scope * * Ignore all instances of variables with a given name in * any (possibly inlined) subprogram with a given namespaced * name. * * @param var_name Name of the local variable to ignore * @param subprogram_name Name of the subprogram to ignore (nullptr for any) * @param subprogram (possibly inlined) Subprogram of the scope current scope * @param scope Current scope */ static void remove_local_variable(simgrid::mc::Frame& scope, const char *var_name, const char *subprogram_name, simgrid::mc::Frame const& subprogram) { typedef std::vector::size_type size_type; // If the current subprogram matches the given name: if ((subprogram_name == nullptr || (!subprogram.name.empty() && subprogram.name == subprogram_name)) && !scope.variables.empty()) { // Try to find the variable and remove it: size_type start = 0; size_type end = scope.variables.size() - 1; // Binary search: while (start <= end) { size_type cursor = start + (end - start) / 2; simgrid::mc::Variable& current_var = scope.variables[cursor]; int compare = current_var.name.compare(var_name); if (compare == 0) { // Variable found, remove it: scope.variables.erase(scope.variables.begin() + cursor); break; } else if (compare < 0) start = cursor + 1; else if (cursor != 0) end = cursor - 1; else break; } } // And recursive processing in nested scopes: for (simgrid::mc::Frame& nested_scope : scope.scopes) { // The new scope may be an inlined subroutine, in this case we want to use its // namespaced name in recursive calls: simgrid::mc::Frame const& nested_subprogram = nested_scope.tag == DW_TAG_inlined_subroutine ? nested_scope : subprogram; remove_local_variable(nested_scope, var_name, subprogram_name, nested_subprogram); } } void ObjectInformation::remove_local_variable( const char* var_name, const char* subprogram_name) { for (auto& entry : this->subprograms) simgrid::mc::remove_local_variable(entry.second, var_name, subprogram_name, entry.second); } } }SimGrid-3.14.159/src/mc/mc_xbt.cpp000644 001750 001750 00000002132 13031031271 020233 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include "src/mc/RemotePtr.hpp" #include "src/mc/AddressSpace.hpp" #include "src/mc/mc_xbt.hpp" #include #include namespace simgrid { namespace mc { void read_element(AddressSpace const& as, void* local, RemotePtr addr, std::size_t i, std::size_t len) { s_xbt_dynar_t d; as.read_bytes(&d, sizeof(d), addr); if (i >= d.used) xbt_die("Out of bound index %zi/%lu", i, d.used); if (len != d.elmsize) xbt_die("Bad size in simgrid::mc::read_element"); as.read_bytes(local, len, remote(xbt_dynar_get_ptr(&d, i))); } std::size_t read_length(AddressSpace const& as, RemotePtr addr) { if (!addr) return 0; unsigned long res; as.read_bytes(&res, sizeof(res), remote(&((xbt_dynar_t)addr.address())->used)); return res; } } } SimGrid-3.14.159/src/mc/mc_request.cpp000644 001750 001750 00000047567 13031031270 021152 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include "src/mc/mc_request.h" #include "src/mc/mc_safety.h" #include "src/mc/mc_private.h" #include "src/mc/mc_smx.h" #include "src/mc/mc_xbt.hpp" using simgrid::mc::remote; XBT_LOG_NEW_DEFAULT_SUBCATEGORY(mc_request, mc, "Logging specific to MC (request)"); static char *pointer_to_string(void *pointer); static char *buff_size_to_string(size_t size); static inline simgrid::kernel::activity::Comm* MC_get_comm(smx_simcall_t r) { switch (r->call ) { case SIMCALL_COMM_WAIT: return static_cast(simcall_comm_wait__get__comm(r)); case SIMCALL_COMM_TEST: return static_cast(simcall_comm_test__get__comm(r)); default: return nullptr; } } static inline smx_mailbox_t MC_get_mbox(smx_simcall_t r) { switch(r->call) { case SIMCALL_COMM_ISEND: return simcall_comm_isend__get__mbox(r); case SIMCALL_COMM_IRECV: return simcall_comm_irecv__get__mbox(r); default: return nullptr; } } namespace simgrid { namespace mc { // Does half the job static inline bool request_depend_asymmetric(smx_simcall_t r1, smx_simcall_t r2) { if (r1->call == SIMCALL_COMM_ISEND && r2->call == SIMCALL_COMM_IRECV) return false; if (r1->call == SIMCALL_COMM_IRECV && r2->call == SIMCALL_COMM_ISEND) return false; // Those are internal requests, we do not need indirection // because those objects are copies: simgrid::kernel::activity::Comm* synchro1 = MC_get_comm(r1); simgrid::kernel::activity::Comm* synchro2 = MC_get_comm(r2); if ((r1->call == SIMCALL_COMM_ISEND || r1->call == SIMCALL_COMM_IRECV) && r2->call == SIMCALL_COMM_WAIT) { smx_mailbox_t mbox = MC_get_mbox(r1); if (mbox != synchro2->mbox_cpy && simcall_comm_wait__get__timeout(r2) <= 0) return false; if ((r1->issuer != synchro2->src_proc) && (r1->issuer != synchro2->dst_proc) && simcall_comm_wait__get__timeout(r2) <= 0) return false; if ((r1->call == SIMCALL_COMM_ISEND) && (synchro2->type == SIMIX_COMM_SEND) && (synchro2->src_buff != simcall_comm_isend__get__src_buff(r1)) && simcall_comm_wait__get__timeout(r2) <= 0) return false; if ((r1->call == SIMCALL_COMM_IRECV) && (synchro2->type == SIMIX_COMM_RECEIVE) && (synchro2->dst_buff != simcall_comm_irecv__get__dst_buff(r1)) && simcall_comm_wait__get__timeout(r2) <= 0) return false; } /* FIXME: the following rule assumes that the result of the * isend/irecv call is not stored in a buffer used in the * test call. */ #if 0 if((r1->call == SIMCALL_COMM_ISEND || r1->call == SIMCALL_COMM_IRECV) && r2->call == SIMCALL_COMM_TEST) return FALSE; #endif if (r1->call == SIMCALL_COMM_WAIT && (r2->call == SIMCALL_COMM_WAIT || r2->call == SIMCALL_COMM_TEST) && (synchro1->src_proc == nullptr || synchro1->dst_proc == nullptr)) return false; if (r1->call == SIMCALL_COMM_TEST && (simcall_comm_test__get__comm(r1) == nullptr || synchro1->src_buff == nullptr || synchro1->dst_buff == nullptr)) return false; if (r1->call == SIMCALL_COMM_TEST && r2->call == SIMCALL_COMM_WAIT && synchro1->src_buff == synchro2->src_buff && synchro1->dst_buff == synchro2->dst_buff) return false; if (r1->call == SIMCALL_COMM_WAIT && r2->call == SIMCALL_COMM_TEST && synchro1->src_buff != nullptr && synchro1->dst_buff != nullptr && synchro2->src_buff != nullptr && synchro2->dst_buff != nullptr && synchro1->dst_buff != synchro2->src_buff && synchro1->dst_buff != synchro2->dst_buff && synchro2->dst_buff != synchro1->src_buff) return false; return true; } // Those are internal_req bool request_depend(smx_simcall_t r1, smx_simcall_t r2) { if (r1->issuer == r2->issuer) return false; /* Wait with timeout transitions are not considered by the independence theorem, thus we consider them as dependant with all other transitions */ if ((r1->call == SIMCALL_COMM_WAIT && simcall_comm_wait__get__timeout(r1) > 0) || (r2->call == SIMCALL_COMM_WAIT && simcall_comm_wait__get__timeout(r2) > 0)) return TRUE; if (r1->call != r2->call) return request_depend_asymmetric(r1, r2) && request_depend_asymmetric(r2, r1); // Those are internal requests, we do not need indirection // because those objects are copies: simgrid::kernel::activity::Comm* synchro1 = MC_get_comm(r1); simgrid::kernel::activity::Comm* synchro2 = MC_get_comm(r2); switch(r1->call) { case SIMCALL_COMM_ISEND: return simcall_comm_isend__get__mbox(r1) == simcall_comm_isend__get__mbox(r2); case SIMCALL_COMM_IRECV: return simcall_comm_irecv__get__mbox(r1) == simcall_comm_irecv__get__mbox(r2); case SIMCALL_COMM_WAIT: if (synchro1->src_buff == synchro2->src_buff && synchro1->dst_buff == synchro2->dst_buff) return false; else if (synchro1->src_buff != nullptr && synchro1->dst_buff != nullptr && synchro2->src_buff != nullptr && synchro2->dst_buff != nullptr && synchro1->dst_buff != synchro2->src_buff && synchro1->dst_buff != synchro2->dst_buff && synchro2->dst_buff != synchro1->src_buff) return false; else return true; default: return true; } } } } static char *pointer_to_string(void *pointer) { if (XBT_LOG_ISENABLED(mc_request, xbt_log_priority_verbose)) return bprintf("%p", pointer); return xbt_strdup("(verbose only)"); } static char *buff_size_to_string(size_t buff_size) { if (XBT_LOG_ISENABLED(mc_request, xbt_log_priority_verbose)) return bprintf("%zu", buff_size); return xbt_strdup("(verbose only)"); } std::string simgrid::mc::request_to_string(smx_simcall_t req, int value, simgrid::mc::RequestType request_type) { xbt_assert(mc_model_checker != nullptr); bool use_remote_comm = true; switch(request_type) { case simgrid::mc::RequestType::simix: use_remote_comm = true; break; case simgrid::mc::RequestType::executed: case simgrid::mc::RequestType::internal: use_remote_comm = false; break; } const char* type = nullptr; char *args = nullptr; smx_actor_t issuer = MC_smx_simcall_get_issuer(req); switch (req->call) { case SIMCALL_COMM_ISEND: { type = "iSend"; char* p = pointer_to_string(simcall_comm_isend__get__src_buff(req)); char* bs = buff_size_to_string(simcall_comm_isend__get__src_buff_size(req)); if (issuer->host) args = bprintf("src=(%lu)%s (%s), buff=%s, size=%s", issuer->pid, MC_smx_process_get_host_name(issuer), MC_smx_process_get_name(issuer), p, bs); else args = bprintf("src=(%lu)%s, buff=%s, size=%s", issuer->pid, MC_smx_process_get_name(issuer), p, bs); xbt_free(bs); xbt_free(p); break; } case SIMCALL_COMM_IRECV: { size_t* remote_size = simcall_comm_irecv__get__dst_buff_size(req); // size_t size = size_pointer ? *size_pointer : 0; size_t size = 0; if (remote_size) mc_model_checker->process().read_bytes(&size, sizeof(size), remote(remote_size)); type = "iRecv"; char* p = pointer_to_string(simcall_comm_irecv__get__dst_buff(req)); char* bs = buff_size_to_string(size); if (issuer->host) args = bprintf("dst=(%lu)%s (%s), buff=%s, size=%s", issuer->pid, MC_smx_process_get_host_name(issuer), MC_smx_process_get_name(issuer), p, bs); else args = bprintf("dst=(%lu)%s, buff=%s, size=%s", issuer->pid, MC_smx_process_get_name(issuer), p, bs); xbt_free(bs); xbt_free(p); break; } case SIMCALL_COMM_WAIT: { simgrid::kernel::activity::Comm* remote_act = static_cast(simcall_comm_wait__get__comm(req)); char* p; if (value == -1) { type = "WaitTimeout"; p = pointer_to_string(remote_act); args = bprintf("comm=%s", p); } else { type = "Wait"; p = pointer_to_string(remote_act); simgrid::mc::Remote temp_synchro; simgrid::kernel::activity::Comm* act; if (use_remote_comm) { mc_model_checker->process().read(temp_synchro, remote( static_cast(remote_act))); act = temp_synchro.getBuffer(); } else act = remote_act; smx_actor_t src_proc = mc_model_checker->process().resolveProcess( simgrid::mc::remote(act->src_proc)); smx_actor_t dst_proc = mc_model_checker->process().resolveProcess( simgrid::mc::remote(act->dst_proc)); args = bprintf("comm=%s [(%lu)%s (%s)-> (%lu)%s (%s)]", p, src_proc ? src_proc->pid : 0, src_proc ? MC_smx_process_get_host_name(src_proc) : "", src_proc ? MC_smx_process_get_name(src_proc) : "", dst_proc ? dst_proc->pid : 0, dst_proc ? MC_smx_process_get_host_name(dst_proc) : "", dst_proc ? MC_smx_process_get_name(dst_proc) : ""); } xbt_free(p); break; } case SIMCALL_COMM_TEST: { simgrid::kernel::activity::Comm* remote_act = static_cast( simcall_comm_test__get__comm(req)); simgrid::mc::Remote temp_synchro; simgrid::kernel::activity::Comm* act; if (use_remote_comm) { mc_model_checker->process().read(temp_synchro, remote( static_cast(remote_act))); act = temp_synchro.getBuffer(); } else act = remote_act; char* p; if (act->src_proc == nullptr || act->dst_proc == nullptr) { type = "Test FALSE"; p = pointer_to_string(remote_act); args = bprintf("comm=%s", p); } else { type = "Test TRUE"; p = pointer_to_string(remote_act); smx_actor_t src_proc = mc_model_checker->process().resolveProcess( simgrid::mc::remote(act->src_proc)); smx_actor_t dst_proc = mc_model_checker->process().resolveProcess( simgrid::mc::remote(act->dst_proc)); args = bprintf("comm=%s [(%lu)%s (%s) -> (%lu)%s (%s)]", p, src_proc->pid, MC_smx_process_get_name(src_proc), MC_smx_process_get_host_name(src_proc), dst_proc->pid, MC_smx_process_get_name(dst_proc), MC_smx_process_get_host_name(dst_proc)); } xbt_free(p); break; } case SIMCALL_COMM_WAITANY: { type = "WaitAny"; s_xbt_dynar_t comms; mc_model_checker->process().read_bytes( &comms, sizeof(comms), remote(simcall_comm_waitany__get__comms(req))); if (!xbt_dynar_is_empty(&comms)) { smx_activity_t remote_sync; read_element(mc_model_checker->process(), &remote_sync, remote(simcall_comm_waitany__get__comms(req)), value, sizeof(remote_sync)); char* p = pointer_to_string(remote_sync); args = bprintf("comm=%s (%d of %lu)", p, value + 1, xbt_dynar_length(&comms)); xbt_free(p); } else args = bprintf("comm at idx %d", value); break; } case SIMCALL_COMM_TESTANY: if (value == -1) { type = "TestAny FALSE"; args = xbt_strdup("-"); } else { type = "TestAny"; args = bprintf("(%d of %zu)", value + 1, simcall_comm_testany__get__count(req)); } break; case SIMCALL_MUTEX_TRYLOCK: case SIMCALL_MUTEX_LOCK: { if (req->call == SIMCALL_MUTEX_LOCK) type = "Mutex LOCK"; else type = "Mutex TRYLOCK"; simgrid::mc::Remote mutex; mc_model_checker->process().read_bytes(mutex.getBuffer(), sizeof(mutex), remote( req->call == SIMCALL_MUTEX_LOCK ? simcall_mutex_lock__get__mutex(req) : simcall_mutex_trylock__get__mutex(req) )); s_xbt_swag_t mutex_sleeping; mc_model_checker->process().read_bytes(&mutex_sleeping, sizeof(mutex_sleeping), remote(mutex.getBuffer()->sleeping)); args = bprintf("locked = %d, owner = %d, sleeping = %d", mutex.getBuffer()->locked, mutex.getBuffer()->owner != nullptr ? (int) mc_model_checker->process().resolveProcess( simgrid::mc::remote(mutex.getBuffer()->owner))->pid : -1, mutex_sleeping.count); break; } case SIMCALL_MC_RANDOM: type = "MC_RANDOM"; args = bprintf("%d", value); break; default: THROW_UNIMPLEMENTED; } std::string str; if (args != nullptr) str = simgrid::xbt::string_printf("[(%lu)%s (%s)] %s(%s)", issuer->pid, MC_smx_process_get_host_name(issuer), MC_smx_process_get_name(issuer), type, args); else str = simgrid::xbt::string_printf("[(%lu)%s (%s)] %s ", issuer->pid, MC_smx_process_get_host_name(issuer), MC_smx_process_get_name(issuer), type); xbt_free(args); return str; } namespace simgrid { namespace mc { bool request_is_enabled_by_idx(smx_simcall_t req, unsigned int idx) { smx_activity_t remote_act = nullptr; switch (req->call) { case SIMCALL_COMM_WAIT: /* FIXME: check also that src and dst processes are not suspended */ remote_act = simcall_comm_wait__get__comm(req); break; case SIMCALL_COMM_WAITANY: { read_element( mc_model_checker->process(), &remote_act, remote(simcall_comm_waitany__get__comms(req)), idx, sizeof(remote_act)); } break; case SIMCALL_COMM_TESTANY: remote_act = mc_model_checker->process().read(remote( simcall_comm_testany__get__comms(req) + idx)); break; default: return true; } simgrid::mc::Remote temp_comm; mc_model_checker->process().read(temp_comm, remote( static_cast(remote_act))); simgrid::kernel::activity::Comm* comm = temp_comm.getBuffer(); return comm->src_proc && comm->dst_proc; } bool process_is_enabled(smx_actor_t process) { return simgrid::mc::request_is_enabled(&process->simcall); } static const char* colors[] = { "blue", "red", "green3", "goldenrod", "brown", "purple", "magenta", "turquoise4", "gray25", "forestgreen", "hotpink", "lightblue", "tan", }; static inline const char* get_color(int id) { return colors[id % (sizeof(colors) / sizeof(colors[0])) ]; } std::string request_get_dot_output(smx_simcall_t req, int value) { std::string label; const smx_actor_t issuer = MC_smx_simcall_get_issuer(req); switch (req->call) { case SIMCALL_COMM_ISEND: if (issuer->host) label = simgrid::xbt::string_printf("[(%lu)%s] iSend", issuer->pid, MC_smx_process_get_host_name(issuer)); else label = bprintf("[(%lu)] iSend", issuer->pid); break; case SIMCALL_COMM_IRECV: if (issuer->host) label = simgrid::xbt::string_printf("[(%lu)%s] iRecv", issuer->pid, MC_smx_process_get_host_name(issuer)); else label = simgrid::xbt::string_printf("[(%lu)] iRecv", issuer->pid); break; case SIMCALL_COMM_WAIT: { if (value == -1) { if (issuer->host) label = simgrid::xbt::string_printf("[(%lu)%s] WaitTimeout", issuer->pid, MC_smx_process_get_host_name(issuer)); else label = simgrid::xbt::string_printf("[(%lu)] WaitTimeout", issuer->pid); } else { smx_activity_t remote_act = simcall_comm_wait__get__comm(req); simgrid::mc::Remote temp_comm; mc_model_checker->process().read(temp_comm, remote( static_cast(remote_act))); simgrid::kernel::activity::Comm* comm = temp_comm.getBuffer(); smx_actor_t src_proc = mc_model_checker->process().resolveProcess( simgrid::mc::remote(comm->src_proc)); smx_actor_t dst_proc = mc_model_checker->process().resolveProcess( simgrid::mc::remote(comm->dst_proc)); if (issuer->host) label = simgrid::xbt::string_printf("[(%lu)%s] Wait [(%lu)->(%lu)]", issuer->pid, MC_smx_process_get_host_name(issuer), src_proc ? src_proc->pid : 0, dst_proc ? dst_proc->pid : 0); else label = simgrid::xbt::string_printf("[(%lu)] Wait [(%lu)->(%lu)]", issuer->pid, src_proc ? src_proc->pid : 0, dst_proc ? dst_proc->pid : 0); } break; } case SIMCALL_COMM_TEST: { smx_activity_t remote_act = simcall_comm_test__get__comm(req); simgrid::mc::Remote temp_comm; mc_model_checker->process().read(temp_comm, remote( static_cast(remote_act))); simgrid::kernel::activity::Comm* comm = temp_comm.getBuffer(); if (comm->src_proc == nullptr || comm->dst_proc == nullptr) { if (issuer->host) label = simgrid::xbt::string_printf("[(%lu)%s] Test FALSE", issuer->pid, MC_smx_process_get_host_name(issuer)); else label = bprintf("[(%lu)] Test FALSE", issuer->pid); } else { if (issuer->host) label = simgrid::xbt::string_printf("[(%lu)%s] Test TRUE", issuer->pid, MC_smx_process_get_host_name(issuer)); else label = simgrid::xbt::string_printf("[(%lu)] Test TRUE", issuer->pid); } break; } case SIMCALL_COMM_WAITANY: { unsigned long comms_size = read_length( mc_model_checker->process(), remote(simcall_comm_waitany__get__comms(req))); if (issuer->host) label = simgrid::xbt::string_printf("[(%lu)%s] WaitAny [%d of %lu]", issuer->pid, MC_smx_process_get_host_name(issuer), value + 1, comms_size); else label = simgrid::xbt::string_printf("[(%lu)] WaitAny [%d of %lu]", issuer->pid, value + 1, comms_size); break; } case SIMCALL_COMM_TESTANY: if (value == -1) { if (issuer->host) label = simgrid::xbt::string_printf("[(%lu)%s] TestAny FALSE", issuer->pid, MC_smx_process_get_host_name(issuer)); else label = simgrid::xbt::string_printf("[(%lu)] TestAny FALSE", issuer->pid); } else { if (issuer->host) label = simgrid::xbt::string_printf("[(%lu)%s] TestAny TRUE [%d of %lu]", issuer->pid, MC_smx_process_get_host_name(issuer), value + 1, simcall_comm_testany__get__count(req)); else label = simgrid::xbt::string_printf("[(%lu)] TestAny TRUE [%d of %lu]", issuer->pid, value + 1, simcall_comm_testany__get__count(req)); } break; case SIMCALL_MUTEX_TRYLOCK: label = simgrid::xbt::string_printf("[(%lu)] Mutex TRYLOCK", issuer->pid); break; case SIMCALL_MUTEX_LOCK: label = simgrid::xbt::string_printf("[(%lu)] Mutex LOCK", issuer->pid); break; case SIMCALL_MC_RANDOM: if (issuer->host) label = simgrid::xbt::string_printf("[(%lu)%s] MC_RANDOM (%d)", issuer->pid, MC_smx_process_get_host_name(issuer), value); else label = simgrid::xbt::string_printf("[(%lu)] MC_RANDOM (%d)", issuer->pid, value); break; default: THROW_UNIMPLEMENTED; } const char* color = get_color(issuer->pid - 1); return simgrid::xbt::string_printf( "label = \"%s\", color = %s, fontcolor = %s", label.c_str(), color, color); } } } SimGrid-3.14.159/src/mc/AddressSpace.hpp000644 001750 001750 00000011324 13031031270 021327 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2008-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_ADDRESS_SPACE_H #define SIMGRID_MC_ADDRESS_SPACE_H #include #include #include #include #include #include #include #include "src/mc/mc_forward.hpp" #include "src/mc/RemotePtr.hpp" namespace simgrid { namespace mc { /** Process index used when no process is available (SMPI privatization) * * The expected behavior is that if a process index is needed it will fail. * */ const int ProcessIndexMissing = -1; /** Process index used when we don't care about the process index (SMPI privatization) * */ const int ProcessIndexDisabled = -2; /** Constant used when any process will do (SMPI privatization) * * Note: This is is index of the first process. */ const int ProcessIndexAny = 0; /** Options for read operations * * This is a set of flags managed with bitwise operators. Only the * meaningful operations are defined: addition, conversions to/from * integers are not allowed. */ class ReadOptions { std::uint32_t value_; constexpr explicit ReadOptions(std::uint32_t value) : value_(value) {} public: constexpr ReadOptions() : value_(0) {} constexpr operator bool() const { return value_ != 0; } constexpr bool operator!() const { return value_ == 0; } constexpr ReadOptions operator|(ReadOptions const& that) const { return ReadOptions(value_ | that.value_); } constexpr ReadOptions operator&(ReadOptions const& that) const { return ReadOptions(value_ & that.value_); } constexpr ReadOptions operator^(ReadOptions const& that) const { return ReadOptions(value_ ^ that.value_); } constexpr ReadOptions operator~() const { return ReadOptions(~value_); } ReadOptions& operator|=(ReadOptions const& that) { value_ |= that.value_; return *this; } ReadOptions& operator&=(ReadOptions const& that) { value_ &= that.value_; return *this; } ReadOptions& operator^=(ReadOptions const& that) { value_ &= that.value_; return *this; } /** Copy the data to the given buffer */ static constexpr ReadOptions none() { return ReadOptions(0); } /** Allows to return a pointer to another buffer where the data is * available instead of copying the data into the buffer */ static constexpr ReadOptions lazy() { return ReadOptions(1); } }; /** A given state of a given process (abstract base class) * * Currently, this might either be: * * * the current state of an existing process; * * * a snapshot. * * In order to support SMPI privatization, the can read the memory from the * context of a given SMPI process: if specified, the code reads data from the * correct SMPI privatization VMA. */ class AddressSpace { private: Process* process_; public: AddressSpace(Process* process) : process_(process) {} virtual ~AddressSpace(); /** The process of this addres space * * This is where we can get debug informations, memory layout, etc. */ simgrid::mc::Process* process() const { return process_; } /** Read data from the address space * * @param buffer target buffer for the data * @param size number of bytes to read * @param address remote source address of the data * @param process_index which process (used for SMPI privatization) * @param options */ virtual const void* read_bytes(void* buffer, std::size_t size, RemotePtr address, int process_index = ProcessIndexAny, ReadOptions options = ReadOptions::none()) const = 0; /** Read a given data structure from the address space */ template inline void read(T *buffer, RemotePtr ptr, int process_index = ProcessIndexAny) const { this->read_bytes(buffer, sizeof(T), ptr, process_index); } template inline void read(Remote& buffer, RemotePtr ptr, int process_index = ProcessIndexAny) const { this->read_bytes(buffer.getBuffer(), sizeof(T), ptr, process_index); } /** Read a given data structure from the addres space * * This version returns by value. */ template inline Remote read(RemotePtr ptr, int process_index = ProcessIndexMissing) const { Remote res; this->read_bytes(&res, sizeof(T), ptr, process_index); return res; } /** Read a string of known size */ std::string read_string(RemotePtr address, std::size_t len) const { std::string res; res.resize(len); this->read_bytes(&res[0], len, address); return res; } }; } } #endif SimGrid-3.14.159/src/mc/mc_safety.h000644 001750 001750 00000001224 13031031270 020376 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_SAFETY_H #define SIMGRID_MC_SAFETY_H #include #include #include #include #include "src/mc/mc_forward.hpp" #include "src/mc/mc_state.h" namespace simgrid { namespace mc { enum class ReductionMode { unset, none, dpor, }; extern XBT_PRIVATE simgrid::mc::ReductionMode reduction_mode; } } #endif SimGrid-3.14.159/src/mc/mc_private.h000644 001750 001750 00000003632 13031031270 020562 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SIMGRID_MC_PRIVATE_H #define SIMGRID_MC_PRIVATE_H #include "simgrid_config.h" #include #include #include #include #include #include #include "mc/mc.h" #include "mc/datatypes.h" #include "src/mc/mc_base.h" #include "src/simix/smx_private.h" #ifdef __cplusplus #include "src/mc/mc_forward.hpp" #include "src/xbt/memory_map.hpp" #endif #ifdef __cplusplus namespace simgrid { namespace mc { struct DerefAndCompareByNbProcessesAndUsedHeap { template bool operator()(X const& a, Y const& b) { return std::make_pair(a->nb_processes, a->heap_bytes_used) < std::make_pair(b->nb_processes, b->heap_bytes_used); } }; } } #endif SG_BEGIN_DECL() /********************************* MC Global **********************************/ XBT_PRIVATE void MC_init_dot_output(); XBT_PRIVATE extern FILE *dot_output; XBT_PRIVATE void MC_show_deadlock(void); /********************************** Snapshot comparison **********************************/ //#define MC_DEBUG 1 #define MC_VERBOSE 1 /********************************** Miscellaneous **********************************/ SG_END_DECL() #ifdef __cplusplus namespace simgrid { namespace mc { XBT_PRIVATE void find_object_address( std::vector const& maps, simgrid::mc::ObjectInformation* result); XBT_PRIVATE int snapshot_compare(int num1, simgrid::mc::Snapshot* s1, int num2, simgrid::mc::Snapshot* s2); // Move is somewhere else (in the LivenessChecker class, in the Session class?): extern XBT_PRIVATE xbt_automaton_t property_automaton; } } #endif #endif SimGrid-3.14.159/src/simdag/000755 001750 001750 00000000000 13031031273 017124 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/src/simdag/sd_dotloader.cpp000644 001750 001750 00000024211 13031031271 022271 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2009-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "src/internal_config.h" #include "simgrid/simdag.h" #include "xbt/file.h" #include #include "simdag_private.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(sd_dotparse, sd, "Parsing DOT files"); #if HAVE_GRAPHVIZ #include typedef enum { sequential =0, parallel } seq_par_t; xbt_dynar_t SD_dotload_generic(const char * filename, seq_par_t seq_or_par, bool schedule); static void dot_task_p_free(void *task) { SD_task_destroy(*(SD_task_t *)task); } /** @brief loads a DOT file describing a DAG * * See http://www.graphviz.org/doc/info/lang.html for more details. * The size attribute of a node describes: * - for a compute task: the amount of flops to execute * - for a communication task : the amount of bytes to transfer * If this attribute is ommited, the default value is zero. */ xbt_dynar_t SD_dotload(const char *filename) { return SD_dotload_generic(filename, sequential, false); } xbt_dynar_t SD_PTG_dotload(const char * filename) { return SD_dotload_generic(filename, parallel, false); } xbt_dynar_t SD_dotload_with_sched(const char *filename) { return SD_dotload_generic(filename, sequential, true); } static int edge_compare(const void *a, const void *b) { unsigned va = AGSEQ(*(Agedge_t **)a); unsigned vb = AGSEQ(*(Agedge_t **)b); return va == vb ? 0 : (va < vb ? -1 : 1); } xbt_dynar_t SD_dotload_generic(const char * filename, seq_par_t seq_or_par, bool schedule){ xbt_assert(filename, "Unable to use a null file descriptor\n"); FILE *in_file = fopen(filename, "r"); xbt_assert(in_file != nullptr, "Failed to open file: %s", filename); unsigned int i; SD_task_t root; SD_task_t end; SD_task_t task; xbt_dict_t computers; xbt_dynar_t computer = nullptr; xbt_dict_cursor_t dict_cursor; bool schedule_success = true; xbt_dict_t jobs = xbt_dict_new_homogeneous(nullptr); xbt_dynar_t result = xbt_dynar_new(sizeof(SD_task_t), dot_task_p_free); Agraph_t * dag_dot = agread(in_file, NIL(Agdisc_t *)); if (schedule) computers = xbt_dict_new_homogeneous(nullptr); /* Create all the nodes */ Agnode_t *node = nullptr; for (node = agfstnode(dag_dot); node; node = agnxtnode(dag_dot, node)) { char *name = agnameof(node); double amount = atof(agget(node, (char*)"size")); task = static_cast(xbt_dict_get_or_null(jobs, name)); if (task == nullptr) { if (seq_or_par == sequential){ XBT_DEBUG("See ", name, amount); task = SD_task_create_comp_seq(name, nullptr , amount); } else { double alpha = atof(agget(node, (char *) "alpha")); XBT_DEBUG("See ", name, amount, alpha); task = SD_task_create_comp_par_amdahl(name, nullptr , amount, alpha); } xbt_dict_set(jobs, name, task, nullptr); if (strcmp(name,"root") && strcmp(name,"end")) xbt_dynar_push(result, &task); if((seq_or_par == sequential) && ((schedule && schedule_success) || XBT_LOG_ISENABLED(sd_dotparse, xbt_log_priority_verbose))){ /* try to take the information to schedule the task only if all is right*/ char *char_performer = agget(node, (char *) "performer"); char *char_order = agget(node, (char *) "order"); /* Tasks will execute on in a given "order" on a given set of "performer" hosts */ int performer = ((!char_performer || !strcmp(char_performer,"")) ? -1:atoi(char_performer)); int order = ((!char_order || !strcmp(char_order, ""))? -1:atoi(char_order)); if((performer != -1 && order != -1) && performer < (int) sg_host_count()){ /* required parameters are given and less performers than hosts are required */ XBT_DEBUG ("Task '%s' is scheduled on workstation '%d' in position '%d'", task->name, performer, order); computer = static_cast (xbt_dict_get_or_null(computers, char_performer)); if(computer == nullptr){ computer = xbt_dynar_new(sizeof(SD_task_t), nullptr); xbt_dict_set(computers, char_performer, computer, nullptr); } if(static_cast(order) < xbt_dynar_length(computer)){ SD_task_t *task_test = (SD_task_t *)xbt_dynar_get_ptr(computer,order); if(*task_test && *task_test != task){ /* the user gave the same order to several tasks */ schedule_success = false; XBT_VERB("Task '%s' wants to start on performer '%s' at the same position '%s' as task '%s'", (*task_test)->name, char_performer, char_order, task->name); continue; } } /* the parameter seems to be ok */ xbt_dynar_set_as(computer, order, SD_task_t, task); } else { /* one of required parameters is not given */ schedule_success = false; XBT_VERB("The schedule is ignored, task '%s' can not be scheduled on %d hosts", task->name, performer); } } } else { XBT_WARN("Task '%s' is defined more than once", name); } } /*Check if 'root' and 'end' nodes have been explicitly declared. If not, create them. */ root = static_cast(xbt_dict_get_or_null(jobs, "root")); if (root == nullptr) root = (seq_or_par == sequential?SD_task_create_comp_seq("root", nullptr, 0): SD_task_create_comp_par_amdahl("root", nullptr, 0, 0)); SD_task_set_state(root, SD_SCHEDULABLE); /* by design the root task is always SCHEDULABLE */ xbt_dynar_insert_at(result, 0, &root); /* Put it at the beginning of the dynar */ end = static_cast(xbt_dict_get_or_null(jobs, "end")); if (end == nullptr) end = (seq_or_par == sequential?SD_task_create_comp_seq("end", nullptr, 0): SD_task_create_comp_par_amdahl("end", nullptr, 0, 0)); /* Create edges */ xbt_dynar_t edges = xbt_dynar_new(sizeof(Agedge_t*), nullptr); for (node = agfstnode(dag_dot); node; node = agnxtnode(dag_dot, node)) { Agedge_t * edge; xbt_dynar_reset(edges); for (edge = agfstout(dag_dot, node); edge; edge = agnxtout(dag_dot, edge)) xbt_dynar_push_as(edges, Agedge_t *, edge); /* Be sure edges are sorted */ xbt_dynar_sort(edges, edge_compare); xbt_dynar_foreach(edges, i, edge) { char *src_name=agnameof(agtail(edge)); char *dst_name=agnameof(aghead(edge)); double size = atof(agget(edge, (char *) "size")); SD_task_t src = static_cast(xbt_dict_get_or_null(jobs, src_name)); SD_task_t dst = static_cast(xbt_dict_get_or_null(jobs, dst_name)); if (size > 0) { char *name = bprintf("%s->%s", src_name, dst_name); XBT_DEBUG("See ", name, size); task = static_cast(xbt_dict_get_or_null(jobs, name)); if (task == nullptr) { if (seq_or_par == sequential) task = SD_task_create_comm_e2e(name, nullptr , size); else task = SD_task_create_comm_par_mxn_1d_block(name, nullptr , size); SD_task_dependency_add(nullptr, nullptr, src, task); SD_task_dependency_add(nullptr, nullptr, task, dst); xbt_dict_set(jobs, name, task, nullptr); xbt_dynar_push(result, &task); } else { XBT_WARN("Task '%s' is defined more than once", name); } xbt_free(name); } else { SD_task_dependency_add(nullptr, nullptr, src, dst); } } } xbt_dynar_free(&edges); XBT_DEBUG("All tasks have been created, put %s at the end of the dynar", end->name); xbt_dynar_push(result, &end); /* Connect entry tasks to 'root', and exit tasks to 'end'*/ xbt_dynar_foreach (result, i, task){ if (task->predecessors->empty() && task->inputs->empty() && task != root) { XBT_DEBUG("Task '%s' has no source. Add dependency from 'root'", task->name); SD_task_dependency_add(nullptr, nullptr, root, task); } if (task->successors->empty() && task->outputs->empty() && task != end) { XBT_DEBUG("Task '%s' has no destination. Add dependency to 'end'", task->name); SD_task_dependency_add(nullptr, nullptr, task, end); } } agclose(dag_dot); xbt_dict_free(&jobs); fclose(in_file); if(schedule){ char *computer_name; if (schedule_success) { const sg_host_t *workstations = sg_host_list (); xbt_dict_foreach(computers,dict_cursor,computer_name,computer){ SD_task_t previous_task = nullptr; xbt_dynar_foreach(computer, i, task){ /* add dependency between the previous and the task to avoid parallel execution */ if(task){ if(previous_task && !SD_task_dependency_exists(previous_task, task)) SD_task_dependency_add(nullptr, nullptr, previous_task, task); SD_task_schedulel(task, 1, workstations[atoi(computer_name)]); previous_task = task; } } xbt_dynar_free(&computer); } xbt_dict_free(&computers); } else { XBT_WARN("The scheduling is ignored"); xbt_dict_foreach(computers,dict_cursor,computer_name,computer) xbt_dynar_free(&computer); xbt_dict_free(&computers); xbt_dynar_free(&result); result = nullptr; } } if (result && !acyclic_graph_detail(result)) { char* base = xbt_basename(filename); XBT_ERROR("The DOT described in %s is not a DAG. It contains a cycle.", base); free(base); xbt_dynar_free(&result); result = nullptr; } return result; } #else xbt_dynar_t SD_dotload(const char *filename) { xbt_die("SD_dotload_generic() is not usable because graphviz was not found.\n" "Please install graphviz, graphviz-dev, and libgraphviz-dev (and erase CMakeCache.txt) before recompiling."); } xbt_dynar_t SD_dotload_with_sched(const char *filename) { return SD_dotload(filename); } xbt_dynar_t SD_PTG_dotload(const char * filename) { return SD_dotload(filename); } #endif SimGrid-3.14.159/src/simdag/dax_dtd.c000644 001750 001750 00000400645 13031031273 020710 0ustar00mquinsonmquinson000000 000000 #define YY_INT_ALIGNED short int /* A lexical scanner generated by flex */ #define yy_create_buffer dax__create_buffer #define yy_delete_buffer dax__delete_buffer #define yy_flex_debug dax__flex_debug #define yy_init_buffer dax__init_buffer #define yy_flush_buffer dax__flush_buffer #define yy_load_buffer_state dax__load_buffer_state #define yy_switch_to_buffer dax__switch_to_buffer #define yyin dax_in #define yyleng dax_leng #define yylex dax_lex #define yylineno dax_lineno #define yyout dax_out #define yyrestart dax_restart #define yytext dax_text #define yywrap dax_wrap #define yyalloc dax_alloc #define yyrealloc dax_realloc #define yyfree dax_free #define FLEX_SCANNER #define YY_FLEX_MAJOR_VERSION 2 #define YY_FLEX_MINOR_VERSION 6 #define YY_FLEX_SUBMINOR_VERSION 0 #if YY_FLEX_SUBMINOR_VERSION > 0 #define FLEX_BETA #endif /* First, we deal with platform-specific or compiler-specific issues. */ /* begin standard C headers. */ #include #include #include #include /* end standard C headers. */ /* flex integer type definitions */ #ifndef FLEXINT_H #define FLEXINT_H /* C99 systems have . Non-C99 systems may or may not. */ #if defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 says to define __STDC_LIMIT_MACROS before including stdint.h, * if you want the limit (max/min) macros for int types. */ #ifndef __STDC_LIMIT_MACROS #define __STDC_LIMIT_MACROS 1 #endif #include typedef int8_t flex_int8_t; typedef uint8_t flex_uint8_t; typedef int16_t flex_int16_t; typedef uint16_t flex_uint16_t; typedef int32_t flex_int32_t; typedef uint32_t flex_uint32_t; #else typedef signed char flex_int8_t; typedef short int flex_int16_t; typedef int flex_int32_t; typedef unsigned char flex_uint8_t; typedef unsigned short int flex_uint16_t; typedef unsigned int flex_uint32_t; /* Limits of integral types. */ #ifndef INT8_MIN #define INT8_MIN (-128) #endif #ifndef INT16_MIN #define INT16_MIN (-32767-1) #endif #ifndef INT32_MIN #define INT32_MIN (-2147483647-1) #endif #ifndef INT8_MAX #define INT8_MAX (127) #endif #ifndef INT16_MAX #define INT16_MAX (32767) #endif #ifndef INT32_MAX #define INT32_MAX (2147483647) #endif #ifndef UINT8_MAX #define UINT8_MAX (255U) #endif #ifndef UINT16_MAX #define UINT16_MAX (65535U) #endif #ifndef UINT32_MAX #define UINT32_MAX (4294967295U) #endif #endif /* ! C99 */ #endif /* ! FLEXINT_H */ #ifdef __cplusplus /* The "const" storage-class-modifier is valid. */ #define YY_USE_CONST #else /* ! __cplusplus */ /* C99 requires __STDC__ to be defined as 1. */ #if defined (__STDC__) #define YY_USE_CONST #endif /* defined (__STDC__) */ #endif /* ! __cplusplus */ #ifdef YY_USE_CONST #define yyconst const #else #define yyconst #endif /* Returned upon end-of-file. */ #define YY_NULL 0 /* Promotes a possibly negative, possibly signed char to an unsigned * integer for use as an array index. If the signed char is negative, * we want to instead treat it as an 8-bit unsigned char, hence the * double cast. */ #define YY_SC_TO_UI(c) ((unsigned int) (unsigned char) c) /* Enter a start condition. This macro really ought to take a parameter, * but we do it the disgusting crufty way forced on us by the ()-less * definition of BEGIN. */ #define BEGIN (yy_start) = 1 + 2 * /* Translate the current start state into a value that can be later handed * to BEGIN to return to the state. The YYSTATE alias is for lex * compatibility. */ #define YY_START (((yy_start) - 1) / 2) #define YYSTATE YY_START /* Action number for EOF rule of a given start state. */ #define YY_STATE_EOF(state) (YY_END_OF_BUFFER + state + 1) /* Special action meaning "start processing a new file". */ #define YY_NEW_FILE dax_restart(dax_in ) #define YY_END_OF_BUFFER_CHAR 0 /* Size of default input buffer. */ #ifndef YY_BUF_SIZE #ifdef __ia64__ /* On IA-64, the buffer size is 16k, not 8k. * Moreover, YY_BUF_SIZE is 2*YY_READ_BUF_SIZE in the general case. * Ditto for the __ia64__ case accordingly. */ #define YY_BUF_SIZE 32768 #else #define YY_BUF_SIZE 16384 #endif /* __ia64__ */ #endif /* The state buf must be large enough to hold one state per character in the main buffer. */ #define YY_STATE_BUF_SIZE ((YY_BUF_SIZE + 2) * sizeof(yy_state_type)) #ifndef YY_TYPEDEF_YY_BUFFER_STATE #define YY_TYPEDEF_YY_BUFFER_STATE typedef struct yy_buffer_state *YY_BUFFER_STATE; #endif #ifndef YY_TYPEDEF_YY_SIZE_T #define YY_TYPEDEF_YY_SIZE_T typedef size_t yy_size_t; #endif extern yy_size_t dax_leng; extern FILE *dax_in, *dax_out; #define EOB_ACT_CONTINUE_SCAN 0 #define EOB_ACT_END_OF_FILE 1 #define EOB_ACT_LAST_MATCH 2 /* Note: We specifically omit the test for yy_rule_can_match_eol because it requires * access to the local variable yy_act. Since yyless() is a macro, it would break * existing scanners that call yyless() from OUTSIDE dax_lex. * One obvious solution it to make yy_act a global. I tried that, and saw * a 5% performance hit in a non-dax_lineno scanner, because yy_act is * normally declared as a register variable-- so it is not worth it. */ #define YY_LESS_LINENO(n) \ do { \ unsigned int yyl;\ for ( yyl = n; yyl < dax_leng; ++yyl )\ if ( dax_text[yyl] == '\n' )\ --dax_lineno;\ }while(0) #define YY_LINENO_REWIND_TO(dst) \ do {\ const char *p;\ for ( p = yy_cp-1; p >= (dst); --p)\ if ( *p == '\n' )\ --dax_lineno;\ }while(0) /* Return all but the first "n" matched characters back to the input stream. */ #define yyless(n) \ do \ { \ /* Undo effects of setting up dax_text. */ \ int yyless_macro_arg = (n); \ YY_LESS_LINENO(yyless_macro_arg);\ *yy_cp = (yy_hold_char); \ YY_RESTORE_YY_MORE_OFFSET \ (yy_c_buf_p) = yy_cp = yy_bp + yyless_macro_arg - YY_MORE_ADJ; \ YY_DO_BEFORE_ACTION; /* set up dax_text again */ \ } \ while ( 0 ) #define unput(c) yyunput( c, (yytext_ptr) ) #ifndef YY_STRUCT_YY_BUFFER_STATE #define YY_STRUCT_YY_BUFFER_STATE struct yy_buffer_state { FILE *yy_input_file; char *yy_ch_buf; /* input buffer */ char *yy_buf_pos; /* current position in input buffer */ /* Size of input buffer in bytes, not including room for EOB * characters. */ yy_size_t yy_buf_size; /* Number of characters read into yy_ch_buf, not including EOB * characters. */ int yy_n_chars; /* Whether we "own" the buffer - i.e., we know we created it, * and can realloc() it to grow it, and should free() it to * delete it. */ int yy_is_our_buffer; /* Whether this is an "interactive" input source; if so, and * if we're using stdio for input, then we want to use getc() * instead of fread(), to make sure we stop fetching input after * each newline. */ int yy_is_interactive; /* Whether we're considered to be at the beginning of a line. * If so, '^' rules will be active on the next match, otherwise * not. */ int yy_at_bol; int yy_bs_lineno; /**< The line count. */ int yy_bs_column; /**< The column count. */ /* Whether to try to fill the input buffer when we reach the * end of it. */ int yy_fill_buffer; int yy_buffer_status; #define YY_BUFFER_NEW 0 #define YY_BUFFER_NORMAL 1 /* When an EOF's been seen but there's still some text to process * then we mark the buffer as YY_EOF_PENDING, to indicate that we * shouldn't try reading from the input source any more. We might * still have a bunch of tokens to match, though, because of * possible backing-up. * * When we actually see the EOF, we change the status to "new" * (via dax_restart()), so that the user can continue scanning by * just pointing dax_in at a new input file. */ #define YY_BUFFER_EOF_PENDING 2 }; #endif /* !YY_STRUCT_YY_BUFFER_STATE */ /* Stack of input buffers. */ static size_t yy_buffer_stack_top = 0; /**< index of top of stack. */ static size_t yy_buffer_stack_max = 0; /**< capacity of stack. */ static YY_BUFFER_STATE * yy_buffer_stack = 0; /**< Stack as an array. */ /* We provide macros for accessing buffer states in case in the * future we want to put the buffer states in a more general * "scanner state". * * Returns the top of the stack, or NULL. */ #define YY_CURRENT_BUFFER ( (yy_buffer_stack) \ ? (yy_buffer_stack)[(yy_buffer_stack_top)] \ : NULL) /* Same as previous macro, but useful when we know that the buffer stack is not * NULL or when we need an lvalue. For internal use only. */ #define YY_CURRENT_BUFFER_LVALUE (yy_buffer_stack)[(yy_buffer_stack_top)] /* yy_hold_char holds the character lost when dax_text is formed. */ static char yy_hold_char; static int yy_n_chars; /* number of characters read into yy_ch_buf */ yy_size_t dax_leng; /* Points to current character in buffer. */ static char *yy_c_buf_p = (char *) 0; static int yy_init = 0; /* whether we need to initialize */ static int yy_start = 0; /* start state number */ /* Flag which is used to allow dax_wrap()'s to do buffer switches * instead of setting up a fresh dax_in. A bit of a hack ... */ static int yy_did_buffer_switch_on_eof; void dax_restart (FILE *input_file ); void dax__switch_to_buffer (YY_BUFFER_STATE new_buffer ); YY_BUFFER_STATE dax__create_buffer (FILE *file,int size ); void dax__delete_buffer (YY_BUFFER_STATE b ); void dax__flush_buffer (YY_BUFFER_STATE b ); void dax_push_buffer_state (YY_BUFFER_STATE new_buffer ); void dax_pop_buffer_state (void ); static void dax_ensure_buffer_stack (void ); static void dax__load_buffer_state (void ); static void dax__init_buffer (YY_BUFFER_STATE b,FILE *file ); #define YY_FLUSH_BUFFER dax__flush_buffer(YY_CURRENT_BUFFER ) YY_BUFFER_STATE dax__scan_buffer (char *base,yy_size_t size ); YY_BUFFER_STATE dax__scan_string (yyconst char *yy_str ); YY_BUFFER_STATE dax__scan_bytes (yyconst char *bytes,yy_size_t len ); void *dax_alloc (yy_size_t ); void *dax_realloc (void *,yy_size_t ); void dax_free (void * ); #define yy_new_buffer dax__create_buffer #define yy_set_interactive(is_interactive) \ { \ if ( ! YY_CURRENT_BUFFER ){ \ dax_ensure_buffer_stack (); \ YY_CURRENT_BUFFER_LVALUE = \ dax__create_buffer(dax_in,YY_BUF_SIZE ); \ } \ YY_CURRENT_BUFFER_LVALUE->yy_is_interactive = is_interactive; \ } #define yy_set_bol(at_bol) \ { \ if ( ! YY_CURRENT_BUFFER ){\ dax_ensure_buffer_stack (); \ YY_CURRENT_BUFFER_LVALUE = \ dax__create_buffer(dax_in,YY_BUF_SIZE ); \ } \ YY_CURRENT_BUFFER_LVALUE->yy_at_bol = at_bol; \ } #define YY_AT_BOL() (YY_CURRENT_BUFFER_LVALUE->yy_at_bol) /* Begin user sect3 */ #define dax_wrap() (/*CONSTCOND*/1) #define YY_SKIP_YYWRAP typedef unsigned char YY_CHAR; FILE *dax_in = (FILE *) 0, *dax_out = (FILE *) 0; typedef int yy_state_type; extern int dax_lineno; int dax_lineno = 1; extern char *dax_text; #ifdef yytext_ptr #undef yytext_ptr #endif #define yytext_ptr dax_text static yy_state_type yy_get_previous_state (void ); static yy_state_type yy_try_NUL_trans (yy_state_type current_state ); static int yy_get_next_buffer (void ); #if defined(__GNUC__) && __GNUC__ >= 3 __attribute__((__noreturn__)) #endif static void yy_fatal_error (yyconst char msg[] ); /* Done after the current pattern has been matched and before the * corresponding action - sets up dax_text. */ #define YY_DO_BEFORE_ACTION \ (yytext_ptr) = yy_bp; \ dax_leng = (size_t) (yy_cp - yy_bp); \ (yy_hold_char) = *yy_cp; \ *yy_cp = '\0'; \ (yy_c_buf_p) = yy_cp; #define YY_NUM_RULES 140 #define YY_END_OF_BUFFER 141 /* This struct is not used in this scanner, but its presence is necessary. */ struct yy_trans_info { flex_int32_t yy_verify; flex_int32_t yy_nxt; }; static yyconst flex_int16_t yy_accept[775] = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 141, 139, 15, 10, 10, 15, 15, 119, 10, 119, 5, 6, 5, 8, 9, 8, 135, 127, 128, 136, 133, 136, 134, 138, 127, 128, 138, 139, 40, 10, 40, 40, 40, 38, 40, 40, 40, 40, 40, 40, 40, 44, 10, 44, 44, 139, 139, 44, 51, 10, 51, 51, 51, 49, 51, 55, 10, 55, 139, 55, 72, 10, 72, 72, 72, 70, 72, 72, 72, 72, 72, 76, 10, 76, 139, 76, 83, 10, 83, 83, 83, 81, 83, 87, 10, 87, 114, 10, 114, 114, 114, 112, 114, 114, 114, 114, 114, 114, 118, 10, 118, 136, 135, 10, 0, 2, 2, 0, 4, 7, 130, 129, 0, 0, 0, 0, 0, 0, 0, 39, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 0, 0, 0, 0, 0, 0, 0, 0, 50, 52, 52, 0, 0, 71, 73, 73, 73, 73, 73, 73, 0, 0, 82, 84, 84, 0, 113, 115, 115, 115, 115, 115, 115, 115, 115, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 137, 0, 41, 41, 41, 41, 41, 41, 41, 41, 41, 0, 0, 0, 0, 0, 0, 0, 0, 0, 52, 0, 0, 0, 0, 0, 73, 73, 73, 73, 0, 0, 0, 84, 0, 0, 115, 115, 115, 115, 115, 115, 115, 0, 0, 0, 132, 0, 14, 1, 0, 0, 125, 0, 0, 0, 122, 121, 0, 0, 41, 41, 41, 41, 41, 41, 41, 41, 41, 0, 0, 43, 0, 0, 0, 57, 0, 0, 56, 0, 0, 0, 0, 0, 54, 0, 0, 0, 59, 58, 73, 73, 73, 73, 0, 0, 75, 0, 0, 0, 0, 0, 0, 86, 0, 115, 115, 115, 115, 115, 115, 115, 0, 0, 117, 0, 0, 0, 0, 126, 120, 0, 0, 17, 41, 41, 41, 41, 41, 0, 0, 41, 41, 41, 0, 16, 0, 57, 0, 88, 56, 0, 0, 48, 47, 0, 0, 73, 0, 0, 73, 73, 73, 0, 89, 0, 80, 79, 0, 0, 0, 0, 0, 115, 115, 0, 0, 115, 0, 0, 0, 0, 0, 0, 0, 12, 0, 123, 124, 17, 41, 0, 0, 41, 0, 0, 41, 0, 29, 28, 41, 0, 41, 0, 41, 0, 16, 46, 0, 88, 45, 0, 0, 0, 0, 0, 63, 62, 73, 73, 73, 0, 74, 89, 0, 0, 91, 90, 0, 0, 0, 115, 115, 0, 105, 104, 115, 0, 111, 110, 0, 0, 0, 0, 0, 11, 41, 0, 21, 20, 41, 0, 25, 24, 41, 41, 41, 0, 33, 32, 41, 0, 42, 46, 77, 45, 0, 78, 0, 61, 60, 73, 73, 73, 0, 0, 0, 0, 0, 115, 115, 115, 0, 116, 0, 0, 0, 0, 41, 41, 41, 0, 0, 41, 41, 77, 0, 53, 78, 73, 0, 0, 0, 0, 0, 0, 0, 0, 0, 115, 115, 115, 0, 0, 0, 0, 41, 41, 0, 0, 0, 31, 30, 41, 41, 73, 0, 67, 66, 0, 69, 68, 0, 85, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 131, 0, 0, 0, 41, 0, 0, 0, 27, 26, 0, 0, 41, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 23, 22, 0, 35, 34, 41, 0, 65, 64, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 19, 18, 41, 93, 0, 92, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 41, 95, 94, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 41, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 41, 0, 99, 0, 98, 0, 103, 0, 102, 0, 109, 0, 108, 0, 0, 0, 0, 0, 0, 41, 97, 96, 101, 100, 107, 106, 0, 0, 0, 12, 0, 12, 0, 41, 0, 0, 0, 0, 0, 41, 0, 0, 0, 0, 0, 11, 0, 37, 36, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 13, 0 } ; static yyconst YY_CHAR yy_ec[256] = { 0, 1, 1, 1, 1, 1, 1, 1, 1, 2, 3, 1, 2, 4, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 5, 6, 7, 1, 1, 8, 9, 1, 1, 1, 1, 1, 10, 11, 12, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 14, 15, 16, 17, 18, 19, 1, 20, 21, 22, 23, 24, 21, 25, 25, 25, 25, 25, 26, 27, 25, 28, 29, 25, 25, 30, 31, 25, 25, 25, 25, 32, 25, 33, 1, 34, 1, 25, 1, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 25, 57, 58, 59, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 } ; static yyconst YY_CHAR yy_meta[60] = { 0, 1, 2, 2, 2, 1, 1, 1, 1, 1, 3, 3, 1, 4, 5, 1, 1, 1, 6, 1, 7, 7, 7, 7, 7, 5, 5, 5, 5, 5, 5, 5, 5, 1, 1, 7, 7, 7, 7, 7, 7, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5 } ; static yyconst flex_uint16_t yy_base[814] = { 0, 0, 0, 0, 3, 6, 9, 24, 27, 11, 14, 15, 17, 29, 38, 45, 52, 59, 61, 67, 70, 95, 0, 73, 76, 153, 156, 159, 162, 177, 180, 183, 186, 201, 204, 207, 210, 226, 278, 245, 248, 251, 254, 297, 300, 303, 306, 330, 0, 388, 391, 394, 397, 412, 415, 418, 421, 437, 489, 456, 459, 541, 0, 462, 465, 598, 600, 2053, 2135, 2135, 269, 272, 47, 62, 2135, 321, 172, 2135, 2135, 2033, 2135, 2135, 2022, 2135, 2033, 2031, 475, 2135, 2135, 2135, 2135, 2027, 2025, 1987, 468, 2135, 324, 1998, 0, 196, 2135, 8, 1968, 1958, 1952, 1863, 1858, 35, 2135, 481, 600, 608, 604, 612, 641, 2135, 509, 1878, 0, 258, 2135, 1855, 2135, 515, 660, 663, 682, 2135, 525, 1875, 0, 310, 2135, 1854, 1852, 1855, 1834, 1849, 2135, 528, 704, 701, 723, 2135, 531, 1869, 0, 407, 2135, 1847, 2135, 534, 606, 2135, 626, 1857, 0, 469, 2135, 1831, 1829, 1821, 1830, 1814, 32, 2135, 667, 621, 1851, 1820, 671, 628, 1784, 2135, 1827, 1818, 2135, 2135, 2135, 2, 44, 1781, 1780, 1778, 1814, 1793, 2135, 0, 1787, 1774, 1777, 1784, 1784, 1772, 1747, 1752, 1753, 1752, 1748, 1742, 1722, 1735, 1712, 1715, 1720, 2135, 0, 1713, 1695, 1693, 2135, 0, 686, 1670, 1671, 1668, 1660, 1666, 1656, 2135, 0, 1661, 1650, 2135, 0, 1652, 1644, 1635, 1646, 1627, 1648, 1632, 1626, 1634, 1648, 1646, 1649, 458, 1611, 2135, 72, 0, 1606, 1602, 1635, 1633, 1594, 2135, 1607, 1590, 1586, 1594, 1592, 1601, 1583, 1567, 1567, 1600, 759, 762, 1577, 1564, 1570, 1547, 1559, 1560, 1544, 727, 766, 772, 1534, 779, 783, 1545, 1536, 1520, 1519, 791, 795, 1528, 799, 802, 808, 1527, 1517, 1517, 1513, 1506, 1496, 1503, 815, 819, 1519, 2135, 1521, 2135, 2135, 17, 1488, 2135, 1514, 1513, 1473, 2135, 2135, 1464, 1475, 1477, 1459, 1490, 1453, 1450, 825, 1455, 1443, 1438, 828, 832, 2135, 836, 1449, 1442, 845, 1447, 1432, 849, 1437, 853, 856, 864, 871, 2135, 874, 1433, 877, 2135, 2135, 1425, 891, 1426, 1421, 894, 897, 2135, 900, 1410, 903, 907, 919, 922, 2135, 925, 928, 931, 1411, 1406, 947, 1404, 950, 953, 956, 2135, 966, 1422, 646, 976, 2135, 2135, 1429, 1427, 959, 1419, 979, 1386, 983, 1379, 986, 995, 1374, 1004, 1381, 1007, 989, 1379, 1010, 1367, 1013, 1020, 1369, 1024, 2135, 2135, 1032, 1357, 1035, 1038, 1054, 1352, 1353, 1347, 1041, 1062, 1065, 2135, 2135, 1073, 1077, 1080, 1093, 1096, 1333, 1325, 1104, 1111, 1334, 1120, 1123, 1131, 1352, 507, 1353, 1136, 2135, 1347, 2135, 2135, 1139, 1315, 1142, 1149, 1291, 1159, 1162, 1297, 1176, 2135, 2135, 1295, 1170, 1285, 1187, 1298, 1195, 1198, 1201, 1275, 1204, 1207, 1212, 1274, 1215, 1218, 1231, 2135, 2135, 1282, 1273, 1263, 1239, 2135, 1242, 1245, 1249, 2135, 2135, 1258, 124, 125, 1274, 1262, 1266, 2135, 2135, 1259, 1274, 2135, 2135, 1282, 1261, 491, 152, 5, 2135, 1235, 1285, 2135, 2135, 1234, 1293, 2135, 2135, 1227, 1301, 1226, 1304, 2135, 2135, 1235, 1312, 2135, 1317, 1320, 1323, 1329, 1332, 1335, 2135, 2135, 1236, 1346, 1349, 1352, 1223, 1211, 1217, 1204, 1210, 1202, 1187, 1355, 2135, 1218, 174, 158, 384, 1188, 1177, 1358, 1365, 1374, 1186, 1181, 1382, 1385, 2135, 1388, 1187, 1391, 1395, 1407, 1410, 1418, 1175, 1169, 1162, 1141, 1423, 1426, 1429, 1161, 1435, 176, 461, 1134, 1445, 1448, 1452, 1464, 2135, 2135, 1472, 1151, 1475, 1478, 2135, 2135, 1491, 2135, 2135, 1499, 2135, 1128, 1131, 1122, 1120, 1502, 1505, 1518, 1521, 1529, 1534, 2135, 1545, 228, 179, 1548, 1551, 1555, 1567, 2135, 2135, 1575, 1579, 1143, 1587, 1591, 1113, 1105, 1103, 1101, 1599, 567, 570, 1607, 582, 626, 1615, 668, 671, 280, 222, 643, 1623, 1626, 1635, 2135, 2135, 1643, 2135, 2135, 1101, 1651, 2135, 2135, 1142, 1093, 1127, 1077, 1095, 1076, 1084, 1066, 1081, 1059, 1074, 1052, 1068, 1049, 1058, 1040, 715, 244, 385, 1659, 2135, 2135, 1053, 2135, 1082, 2135, 1078, 1026, 1015, 1016, 1006, 1008, 992, 1000, 974, 973, 921, 929, 918, 388, 391, 1667, 937, 2135, 2135, 912, 907, 889, 902, 861, 851, 835, 848, 832, 830, 792, 776, 1670, 1673, 1676, 1693, 753, 714, 746, 695, 712, 666, 678, 644, 648, 603, 609, 501, 471, 1701, 1704, 1711, 1721, 1739, 1773, 408, 429, 2135, 421, 2135, 421, 2135, 402, 2135, 402, 2135, 396, 2135, 725, 1791, 675, 506, 1809, 1843, 272, 2135, 2135, 2135, 2135, 2135, 2135, 684, 706, 200, 311, 731, 260, 1731, 220, 668, 218, 202, 207, 1765, 1836, 296, 153, 1861, 1757, 728, 2135, 1799, 2135, 2135, 1734, 1864, 439, 709, 753, 773, 806, 807, 761, 873, 823, 901, 948, 1031, 1013, 1035, 1014, 929, 1877, 1881, 2135, 2135, 1899, 1906, 1913, 1920, 1927, 1934, 1941, 1948, 1955, 1962, 1969, 1976, 1983, 1990, 1995, 2000, 2005, 2010, 2015, 2022, 2025, 2028, 2031, 2034, 2037, 2044, 2048, 2054, 2060, 2066, 2072, 2078, 2085, 2092, 2099, 2106, 2113, 2120, 2127 } ; static yyconst flex_int16_t yy_def[814] = { 0, 775, 775, 776, 776, 776, 776, 777, 777, 778, 778, 779, 779, 780, 780, 780, 780, 781, 781, 775, 775, 774, 21, 782, 782, 782, 782, 775, 775, 782, 782, 775, 775, 782, 782, 782, 782, 783, 783, 784, 784, 775, 775, 784, 784, 784, 784, 774, 47, 785, 785, 775, 775, 785, 785, 785, 785, 786, 786, 787, 787, 774, 61, 788, 788, 780, 780, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 789, 774, 774, 789, 789, 789, 789, 789, 789, 789, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 790, 774, 774, 790, 774, 774, 774, 774, 774, 774, 774, 774, 791, 774, 774, 791, 791, 791, 791, 791, 774, 774, 774, 774, 774, 774, 774, 774, 792, 774, 774, 792, 774, 774, 774, 774, 774, 774, 793, 774, 774, 793, 793, 793, 793, 793, 793, 774, 774, 774, 774, 774, 774, 794, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 789, 789, 789, 789, 789, 789, 789, 789, 789, 789, 795, 774, 774, 774, 774, 774, 774, 774, 774, 790, 790, 796, 774, 774, 791, 791, 791, 791, 791, 791, 797, 774, 774, 792, 792, 798, 774, 793, 793, 793, 793, 793, 793, 793, 793, 799, 774, 774, 800, 774, 800, 774, 774, 774, 801, 774, 774, 774, 774, 774, 774, 774, 789, 789, 789, 789, 789, 789, 789, 789, 789, 802, 802, 774, 774, 774, 774, 774, 774, 774, 790, 803, 803, 774, 774, 774, 791, 791, 791, 791, 804, 804, 774, 792, 805, 805, 793, 793, 793, 793, 793, 793, 793, 806, 806, 774, 774, 800, 774, 774, 800, 774, 774, 801, 774, 774, 774, 774, 774, 774, 789, 789, 789, 789, 789, 789, 789, 789, 789, 774, 802, 774, 802, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 803, 774, 803, 774, 774, 774, 774, 791, 791, 791, 791, 774, 804, 774, 804, 774, 774, 774, 774, 805, 774, 805, 793, 793, 793, 793, 793, 793, 793, 774, 806, 774, 806, 774, 800, 807, 774, 774, 774, 774, 774, 789, 789, 789, 789, 789, 774, 774, 789, 789, 789, 802, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 803, 774, 791, 774, 774, 791, 791, 791, 804, 774, 774, 774, 774, 805, 774, 774, 774, 774, 793, 793, 774, 774, 793, 774, 774, 806, 774, 800, 807, 807, 774, 807, 774, 774, 774, 789, 774, 774, 789, 774, 774, 789, 774, 774, 774, 789, 774, 789, 774, 789, 802, 774, 774, 774, 774, 774, 803, 774, 774, 774, 774, 774, 774, 791, 791, 791, 774, 774, 774, 805, 774, 774, 774, 774, 774, 774, 793, 793, 774, 774, 774, 793, 774, 774, 774, 806, 774, 800, 807, 807, 774, 789, 774, 774, 774, 789, 774, 774, 774, 789, 789, 789, 774, 774, 774, 789, 774, 774, 774, 774, 774, 803, 774, 774, 774, 774, 791, 791, 791, 805, 774, 774, 774, 774, 793, 793, 793, 774, 774, 774, 800, 807, 807, 789, 789, 789, 774, 774, 789, 789, 774, 774, 774, 774, 791, 774, 774, 774, 774, 805, 774, 774, 774, 774, 793, 793, 793, 774, 800, 807, 807, 789, 789, 774, 774, 774, 774, 774, 789, 789, 791, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 800, 807, 807, 789, 774, 774, 774, 774, 774, 774, 774, 789, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 800, 807, 807, 774, 774, 774, 774, 774, 774, 774, 774, 789, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 800, 807, 807, 774, 774, 774, 789, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 800, 807, 807, 789, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 800, 807, 807, 807, 789, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 800, 807, 807, 807, 808, 809, 789, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 800, 807, 810, 811, 808, 809, 789, 774, 774, 774, 774, 774, 774, 800, 810, 807, 812, 811, 813, 807, 789, 800, 812, 774, 813, 807, 789, 800, 774, 774, 774, 800, 774, 774, 774, 774, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 800, 774, 0, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774 } ; static yyconst flex_uint16_t yy_nxt[2195] = { 0, 774, 70, 71, 70, 70, 71, 70, 70, 71, 70, 70, 71, 70, 78, 240, 72, 78, 81, 72, 81, 79, 73, 422, 79, 73, 75, 71, 75, 75, 71, 75, 84, 85, 82, 295, 82, 86, 87, 364, 76, 84, 85, 76, 524, 88, 86, 87, 84, 85, 188, 89, 171, 86, 88, 84, 85, 189, 89, 241, 86, 88, 91, 92, 91, 92, 172, 171, 88, 71, 71, 71, 71, 71, 71, 109, 71, 109, 109, 71, 109, 173, 195, 94, 230, 240, 94, 299, 196, 110, 231, 242, 110, 93, 243, 93, 95, 96, 71, 96, 95, 95, 95, 95, 95, 95, 95, 97, 95, 98, 95, 99, 95, 100, 95, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 95, 95, 98, 98, 101, 98, 98, 102, 98, 98, 103, 104, 98, 98, 98, 105, 98, 98, 98, 98, 98, 98, 98, 106, 107, 98, 98, 109, 71, 109, 109, 71, 109, 71, 71, 71, 71, 71, 71, 512, 514, 111, 422, 749, 111, 513, 515, 112, 422, 174, 112, 109, 71, 109, 109, 71, 109, 71, 71, 71, 71, 71, 71, 173, 295, 110, 422, 551, 110, 422, 550, 113, 523, 174, 113, 109, 71, 109, 109, 71, 109, 109, 71, 109, 109, 71, 109, 173, 740, 111, 422, 423, 111, 745, 611, 114, 740, 583, 114, 115, 116, 71, 116, 115, 115, 115, 115, 115, 115, 115, 117, 115, 422, 115, 119, 115, 120, 115, 422, 123, 71, 123, 123, 71, 123, 71, 71, 71, 71, 71, 71, 115, 115, 124, 422, 174, 124, 641, 610, 125, 743, 740, 125, 170, 170, 170, 170, 170, 170, 173, 121, 115, 116, 71, 116, 115, 115, 115, 115, 115, 115, 115, 117, 115, 664, 115, 119, 115, 120, 115, 295, 123, 71, 123, 123, 71, 123, 123, 71, 123, 123, 71, 123, 115, 115, 124, 295, 174, 124, 740, 640, 126, 748, 737, 126, 170, 170, 170, 170, 170, 170, 173, 121, 127, 128, 71, 128, 127, 127, 127, 127, 127, 127, 127, 129, 127, 130, 127, 131, 127, 132, 127, 130, 130, 130, 130, 130, 130, 130, 130, 130, 130, 130, 130, 130, 127, 127, 130, 130, 130, 130, 130, 130, 130, 130, 133, 130, 130, 134, 130, 135, 130, 130, 130, 136, 130, 130, 130, 137, 130, 130, 130, 139, 71, 139, 139, 71, 139, 71, 71, 71, 71, 71, 71, 422, 422, 140, 729, 295, 140, 728, 422, 141, 727, 174, 141, 139, 71, 139, 139, 71, 139, 139, 71, 139, 139, 71, 139, 173, 726, 140, 681, 725, 140, 682, 665, 142, 724, 552, 142, 143, 144, 71, 144, 143, 143, 143, 143, 143, 143, 143, 145, 143, 723, 143, 147, 143, 148, 143, 295, 151, 71, 151, 151, 71, 151, 166, 71, 166, 166, 71, 166, 143, 143, 152, 174, 174, 152, 295, 757, 167, 422, 716, 167, 179, 170, 170, 170, 297, 173, 173, 149, 143, 144, 71, 144, 143, 143, 143, 143, 143, 143, 143, 145, 143, 185, 143, 147, 143, 148, 143, 295, 180, 170, 170, 170, 584, 732, 181, 170, 170, 170, 522, 182, 143, 143, 735, 295, 183, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 170, 479, 715, 149, 153, 154, 71, 154, 153, 153, 153, 153, 153, 153, 153, 155, 153, 156, 153, 157, 153, 158, 153, 156, 156, 156, 156, 156, 156, 156, 156, 156, 156, 156, 156, 156, 153, 153, 156, 156, 156, 156, 156, 159, 156, 156, 156, 156, 156, 160, 156, 156, 161, 156, 156, 162, 163, 164, 156, 156, 156, 156, 156, 84, 85, 84, 85, 174, 86, 628, 86, 174, 630, 174, 197, 174, 168, 714, 168, 174, 222, 173, 197, 629, 632, 173, 631, 173, 174, 173, 170, 170, 170, 173, 169, 232, 169, 198, 633, 199, 236, 198, 173, 204, 713, 198, 200, 199, 174, 198, 200, 199, 201, 237, 203, 197, 201, 202, 203, 712, 201, 202, 173, 422, 201, 202, 295, 174, 634, 202, 174, 170, 170, 170, 208, 170, 170, 170, 198, 419, 204, 173, 635, 732, 173, 711, 710, 203, 295, 174, 271, 271, 271, 201, 642, 733, 208, 198, 202, 204, 198, 744, 204, 173, 295, 272, 203, 709, 174, 203, 636, 174, 209, 638, 732, 209, 738, 202, 217, 198, 202, 204, 173, 708, 637, 173, 733, 639, 203, 295, 174, 327, 327, 327, 201, 295, 707, 217, 198, 202, 204, 198, 732, 204, 173, 295, 328, 203, 295, 758, 203, 735, 663, 201, 706, 705, 201, 753, 218, 730, 198, 218, 204, 316, 316, 316, 316, 316, 316, 203, 329, 329, 329, 295, 763, 201, 329, 329, 329, 318, 202, 295, 318, 271, 271, 271, 331, 334, 334, 334, 759, 335, 331, 295, 336, 341, 341, 341, 272, 341, 341, 341, 319, 346, 346, 346, 348, 348, 348, 704, 760, 343, 348, 348, 348, 343, 332, 697, 347, 359, 359, 359, 350, 359, 359, 359, 295, 295, 350, 376, 376, 376, 316, 316, 316, 361, 316, 316, 316, 361, 316, 316, 316, 295, 377, 351, 344, 696, 318, 384, 384, 384, 318, 387, 387, 387, 318, 327, 327, 327, 389, 389, 389, 765, 390, 761, 762, 391, 329, 329, 329, 695, 328, 381, 362, 329, 329, 329, 329, 329, 329, 334, 334, 334, 331, 335, 764, 694, 336, 693, 692, 331, 691, 295, 331, 395, 395, 395, 341, 341, 341, 341, 341, 341, 341, 341, 341, 346, 346, 346, 396, 402, 402, 402, 343, 403, 690, 343, 404, 392, 343, 295, 347, 348, 348, 348, 348, 348, 348, 348, 348, 348, 406, 406, 406, 408, 408, 408, 400, 350, 771, 766, 350, 689, 688, 350, 397, 407, 687, 295, 409, 412, 412, 412, 415, 415, 415, 359, 359, 359, 359, 359, 359, 426, 426, 426, 413, 686, 295, 416, 359, 359, 359, 361, 685, 680, 361, 679, 678, 405, 421, 421, 421, 428, 428, 428, 361, 431, 431, 431, 376, 376, 376, 443, 443, 443, 422, 423, 429, 434, 434, 434, 432, 435, 767, 377, 436, 417, 438, 438, 438, 316, 316, 316, 384, 384, 384, 446, 446, 446, 439, 677, 771, 440, 387, 387, 387, 318, 389, 389, 389, 676, 390, 295, 295, 391, 329, 329, 329, 450, 450, 450, 395, 395, 395, 458, 458, 458, 675, 674, 442, 295, 331, 769, 451, 295, 673, 396, 452, 452, 452, 459, 453, 672, 671, 454, 460, 460, 460, 402, 402, 402, 670, 403, 669, 770, 404, 348, 348, 348, 448, 406, 406, 406, 462, 462, 462, 768, 463, 668, 667, 464, 666, 350, 662, 661, 407, 408, 408, 408, 465, 465, 465, 660, 466, 659, 658, 467, 412, 412, 412, 657, 409, 656, 461, 470, 470, 470, 655, 471, 654, 653, 472, 413, 415, 415, 415, 474, 474, 474, 652, 475, 651, 650, 476, 359, 359, 359, 649, 416, 421, 421, 421, 426, 426, 426, 428, 428, 428, 648, 647, 361, 646, 484, 484, 484, 422, 485, 627, 626, 486, 429, 625, 431, 431, 431, 488, 488, 488, 624, 489, 620, 599, 490, 438, 438, 438, 480, 432, 598, 434, 434, 434, 597, 435, 596, 477, 436, 593, 440, 585, 494, 494, 494, 481, 495, 581, 574, 496, 498, 498, 498, 443, 443, 443, 500, 500, 500, 446, 446, 446, 502, 502, 502, 573, 499, 329, 329, 329, 450, 450, 450, 505, 505, 505, 572, 506, 571, 562, 507, 561, 560, 331, 554, 451, 452, 452, 452, 553, 453, 549, 548, 454, 458, 458, 458, 460, 460, 460, 348, 348, 348, 503, 462, 462, 462, 547, 463, 546, 459, 464, 545, 465, 465, 465, 350, 466, 544, 543, 467, 470, 470, 470, 542, 471, 536, 531, 472, 474, 474, 474, 530, 475, 527, 526, 476, 519, 519, 519, 484, 484, 484, 525, 485, 521, 511, 486, 488, 488, 488, 518, 489, 520, 517, 490, 528, 528, 528, 494, 494, 494, 516, 495, 510, 509, 496, 498, 498, 498, 508, 529, 500, 500, 500, 532, 532, 532, 502, 502, 502, 504, 501, 499, 533, 533, 533, 535, 535, 535, 505, 505, 505, 497, 506, 493, 492, 507, 491, 487, 534, 537, 537, 537, 539, 539, 539, 348, 348, 348, 519, 519, 519, 555, 555, 555, 538, 483, 482, 540, 528, 528, 528, 350, 422, 478, 520, 473, 556, 557, 557, 557, 469, 558, 468, 529, 559, 532, 532, 532, 533, 533, 533, 535, 535, 535, 537, 537, 537, 457, 563, 563, 563, 456, 564, 455, 534, 565, 449, 541, 447, 538, 539, 539, 539, 566, 566, 566, 445, 567, 444, 441, 568, 569, 569, 569, 437, 540, 575, 575, 575, 577, 577, 577, 579, 579, 579, 433, 430, 570, 582, 582, 582, 576, 427, 425, 578, 424, 418, 580, 586, 586, 586, 555, 555, 555, 295, 588, 588, 588, 414, 589, 411, 410, 590, 587, 401, 399, 556, 557, 557, 557, 398, 558, 394, 393, 559, 591, 591, 591, 594, 594, 594, 563, 563, 563, 388, 564, 386, 385, 565, 383, 592, 382, 380, 595, 566, 566, 566, 379, 567, 378, 375, 568, 569, 569, 569, 575, 575, 575, 600, 600, 600, 374, 601, 373, 372, 602, 371, 370, 570, 369, 576, 577, 577, 577, 603, 603, 603, 368, 604, 367, 366, 605, 579, 579, 579, 365, 578, 606, 606, 606, 295, 607, 363, 358, 608, 357, 356, 580, 582, 582, 582, 612, 612, 612, 586, 586, 586, 355, 614, 614, 614, 354, 615, 353, 295, 616, 613, 352, 345, 587, 588, 588, 588, 340, 589, 339, 338, 590, 591, 591, 591, 609, 617, 617, 617, 337, 618, 333, 326, 619, 594, 594, 594, 592, 621, 621, 621, 325, 622, 324, 323, 623, 600, 600, 600, 595, 601, 322, 321, 602, 603, 603, 603, 320, 604, 315, 314, 605, 606, 606, 606, 313, 607, 312, 311, 608, 612, 612, 612, 643, 643, 643, 310, 644, 309, 308, 645, 307, 614, 614, 614, 613, 615, 306, 305, 616, 617, 617, 617, 304, 618, 303, 302, 619, 621, 621, 621, 301, 622, 298, 296, 623, 643, 643, 643, 295, 644, 293, 292, 645, 683, 683, 683, 698, 698, 698, 699, 699, 699, 683, 683, 683, 291, 289, 288, 684, 422, 287, 286, 295, 285, 700, 422, 284, 684, 422, 701, 701, 701, 283, 702, 282, 280, 703, 698, 698, 698, 699, 699, 699, 279, 278, 422, 276, 718, 718, 718, 275, 719, 274, 295, 720, 700, 422, 701, 701, 701, 273, 702, 270, 422, 703, 717, 269, 742, 742, 742, 754, 754, 754, 422, 420, 420, 420, 420, 420, 420, 420, 420, 420, 422, 423, 420, 295, 267, 420, 420, 420, 422, 420, 750, 750, 750, 266, 751, 265, 264, 752, 742, 742, 742, 263, 262, 420, 420, 420, 420, 420, 420, 420, 420, 420, 420, 420, 422, 261, 420, 260, 259, 420, 420, 420, 422, 420, 718, 718, 718, 257, 719, 256, 255, 720, 750, 750, 750, 480, 751, 420, 420, 752, 422, 420, 420, 420, 420, 420, 736, 420, 420, 420, 254, 253, 420, 252, 251, 420, 420, 420, 422, 420, 250, 249, 248, 247, 246, 245, 244, 239, 236, 746, 746, 746, 238, 420, 420, 420, 420, 420, 420, 420, 420, 420, 420, 736, 747, 234, 420, 233, 229, 420, 420, 420, 422, 420, 746, 746, 746, 754, 754, 754, 228, 755, 227, 226, 756, 225, 223, 420, 420, 747, 772, 772, 772, 295, 772, 772, 772, 221, 219, 216, 215, 214, 213, 212, 210, 207, 773, 205, 194, 193, 773, 68, 68, 68, 68, 68, 68, 68, 69, 69, 69, 69, 69, 69, 69, 74, 74, 74, 74, 74, 74, 74, 77, 77, 77, 77, 77, 77, 77, 80, 80, 80, 80, 80, 80, 80, 83, 83, 83, 83, 83, 83, 83, 90, 90, 90, 90, 90, 90, 90, 108, 108, 108, 108, 108, 108, 108, 118, 118, 118, 118, 118, 118, 118, 122, 122, 122, 122, 122, 122, 122, 138, 138, 138, 138, 138, 138, 138, 146, 146, 146, 146, 146, 146, 146, 150, 150, 150, 150, 150, 150, 150, 165, 165, 165, 165, 165, 165, 165, 187, 187, 187, 192, 187, 206, 206, 206, 191, 206, 211, 211, 211, 190, 211, 220, 220, 220, 186, 220, 224, 224, 224, 184, 224, 235, 235, 235, 235, 235, 178, 235, 258, 177, 258, 268, 178, 268, 277, 177, 277, 281, 176, 281, 290, 175, 290, 294, 294, 294, 294, 294, 294, 294, 300, 774, 774, 300, 317, 317, 317, 317, 317, 317, 330, 330, 330, 330, 330, 330, 342, 342, 342, 342, 342, 342, 349, 349, 349, 349, 349, 349, 360, 360, 360, 360, 360, 360, 420, 420, 420, 420, 420, 420, 420, 721, 721, 721, 721, 721, 721, 721, 722, 722, 722, 722, 722, 722, 722, 731, 731, 731, 731, 731, 731, 731, 734, 734, 734, 734, 734, 734, 734, 739, 739, 739, 739, 739, 739, 739, 741, 741, 741, 741, 741, 741, 741, 67, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774 } ; static yyconst flex_int16_t yy_chk[2195] = { 0, 0, 3, 3, 3, 4, 4, 4, 5, 5, 5, 6, 6, 6, 9, 179, 3, 10, 11, 4, 12, 9, 5, 481, 10, 6, 7, 7, 7, 8, 8, 8, 13, 13, 11, 297, 12, 13, 13, 297, 7, 14, 14, 8, 481, 13, 14, 14, 15, 15, 101, 15, 72, 15, 14, 16, 16, 101, 16, 179, 16, 15, 17, 17, 18, 18, 72, 73, 16, 19, 19, 19, 20, 20, 20, 23, 23, 23, 24, 24, 24, 73, 107, 19, 164, 240, 20, 240, 107, 23, 164, 180, 24, 17, 180, 18, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 25, 25, 25, 26, 26, 26, 27, 27, 27, 28, 28, 28, 466, 467, 25, 480, 745, 26, 466, 467, 27, 523, 76, 28, 29, 29, 29, 30, 30, 30, 31, 31, 31, 32, 32, 32, 76, 522, 29, 551, 523, 30, 584, 522, 31, 480, 99, 32, 33, 33, 33, 34, 34, 34, 35, 35, 35, 36, 36, 36, 99, 741, 33, 732, 732, 34, 740, 584, 35, 739, 551, 36, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 610, 37, 37, 37, 37, 37, 583, 39, 39, 39, 40, 40, 40, 41, 41, 41, 42, 42, 42, 37, 37, 39, 641, 119, 40, 610, 583, 41, 737, 735, 42, 70, 70, 70, 71, 71, 71, 119, 37, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 641, 38, 38, 38, 38, 38, 609, 43, 43, 43, 44, 44, 44, 45, 45, 45, 46, 46, 46, 38, 38, 43, 744, 131, 44, 733, 609, 45, 744, 723, 46, 75, 75, 75, 96, 96, 96, 131, 38, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 47, 49, 49, 49, 50, 50, 50, 51, 51, 51, 52, 52, 52, 524, 642, 49, 715, 663, 50, 713, 664, 51, 711, 147, 52, 53, 53, 53, 54, 54, 54, 55, 55, 55, 56, 56, 56, 147, 709, 53, 663, 707, 54, 664, 642, 55, 705, 524, 56, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 704, 57, 57, 57, 57, 57, 755, 59, 59, 59, 60, 60, 60, 63, 63, 63, 64, 64, 64, 57, 57, 59, 94, 157, 60, 237, 755, 63, 552, 697, 64, 86, 109, 109, 109, 237, 94, 157, 57, 58, 58, 58, 58, 58, 58, 58, 58, 58, 58, 58, 58, 58, 94, 58, 58, 58, 58, 58, 479, 86, 116, 116, 116, 552, 720, 86, 123, 123, 123, 479, 86, 58, 58, 720, 419, 86, 128, 128, 128, 139, 139, 139, 144, 144, 144, 151, 151, 151, 419, 696, 58, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 65, 65, 66, 66, 110, 65, 601, 66, 112, 602, 152, 110, 111, 65, 695, 66, 113, 152, 110, 111, 601, 604, 112, 602, 152, 167, 111, 154, 154, 154, 113, 65, 167, 66, 110, 604, 110, 171, 112, 167, 112, 694, 111, 110, 111, 114, 113, 112, 113, 110, 171, 111, 114, 112, 110, 113, 693, 111, 112, 114, 611, 113, 111, 364, 124, 605, 113, 125, 166, 166, 166, 124, 170, 170, 170, 114, 364, 114, 124, 605, 719, 125, 692, 691, 114, 738, 126, 212, 212, 212, 114, 611, 719, 126, 124, 114, 124, 125, 738, 125, 126, 730, 212, 124, 690, 141, 125, 607, 140, 124, 608, 731, 125, 730, 124, 140, 126, 125, 126, 141, 689, 607, 140, 731, 608, 126, 756, 142, 267, 267, 267, 126, 640, 688, 142, 141, 126, 141, 140, 734, 140, 142, 717, 267, 141, 748, 756, 140, 734, 640, 141, 687, 686, 140, 748, 141, 717, 142, 140, 142, 258, 258, 258, 259, 259, 259, 142, 268, 268, 268, 757, 761, 142, 269, 269, 269, 258, 142, 761, 259, 271, 271, 271, 268, 272, 272, 272, 757, 272, 269, 758, 272, 277, 277, 277, 271, 278, 278, 278, 259, 280, 280, 280, 281, 281, 281, 685, 758, 277, 282, 282, 282, 278, 269, 680, 280, 290, 290, 290, 281, 291, 291, 291, 759, 760, 282, 312, 312, 312, 316, 316, 316, 290, 317, 317, 317, 291, 319, 319, 319, 763, 312, 282, 278, 679, 316, 322, 322, 322, 317, 325, 325, 325, 319, 327, 327, 327, 328, 328, 328, 763, 328, 759, 760, 328, 329, 329, 329, 678, 327, 319, 291, 330, 330, 330, 332, 332, 332, 334, 334, 334, 329, 334, 762, 677, 334, 676, 675, 330, 674, 762, 332, 338, 338, 338, 341, 341, 341, 342, 342, 342, 344, 344, 344, 346, 346, 346, 338, 347, 347, 347, 341, 347, 673, 342, 347, 332, 344, 764, 346, 348, 348, 348, 349, 349, 349, 351, 351, 351, 352, 352, 352, 353, 353, 353, 344, 348, 770, 764, 349, 672, 671, 351, 338, 352, 670, 770, 353, 356, 356, 356, 358, 358, 358, 359, 359, 359, 360, 360, 360, 370, 370, 370, 356, 669, 765, 358, 362, 362, 362, 359, 666, 662, 360, 661, 660, 351, 365, 365, 365, 372, 372, 372, 362, 374, 374, 374, 376, 376, 376, 382, 382, 382, 365, 365, 372, 377, 377, 377, 374, 377, 765, 376, 377, 362, 379, 379, 379, 381, 381, 381, 384, 384, 384, 386, 386, 386, 379, 659, 769, 379, 387, 387, 387, 381, 389, 389, 389, 658, 389, 767, 769, 389, 392, 392, 392, 394, 394, 394, 395, 395, 395, 400, 400, 400, 657, 656, 381, 766, 392, 767, 394, 768, 655, 395, 396, 396, 396, 400, 396, 654, 653, 396, 401, 401, 401, 402, 402, 402, 652, 402, 651, 768, 402, 405, 405, 405, 392, 406, 406, 406, 407, 407, 407, 766, 407, 650, 648, 407, 646, 405, 639, 638, 406, 408, 408, 408, 409, 409, 409, 637, 409, 636, 635, 409, 412, 412, 412, 634, 408, 633, 405, 413, 413, 413, 632, 413, 631, 630, 413, 412, 415, 415, 415, 416, 416, 416, 629, 416, 628, 627, 416, 417, 417, 417, 626, 415, 421, 421, 421, 426, 426, 426, 428, 428, 428, 625, 624, 417, 620, 429, 429, 429, 421, 429, 599, 598, 429, 428, 597, 431, 431, 431, 432, 432, 432, 596, 432, 593, 574, 432, 438, 438, 438, 421, 431, 573, 434, 434, 434, 572, 434, 571, 417, 434, 561, 438, 553, 440, 440, 440, 421, 440, 549, 545, 440, 442, 442, 442, 443, 443, 443, 444, 444, 444, 446, 446, 446, 447, 447, 447, 544, 442, 448, 448, 448, 450, 450, 450, 451, 451, 451, 543, 451, 542, 536, 451, 531, 530, 448, 526, 450, 452, 452, 452, 525, 452, 521, 518, 452, 458, 458, 458, 460, 460, 460, 461, 461, 461, 448, 462, 462, 462, 517, 462, 516, 458, 462, 515, 465, 465, 465, 461, 465, 514, 513, 465, 470, 470, 470, 512, 470, 508, 497, 470, 474, 474, 474, 493, 474, 491, 487, 474, 477, 477, 477, 484, 484, 484, 483, 484, 478, 461, 484, 488, 488, 488, 473, 488, 477, 469, 488, 492, 492, 492, 494, 494, 494, 468, 494, 457, 456, 494, 498, 498, 498, 455, 492, 500, 500, 500, 501, 501, 501, 502, 502, 502, 449, 445, 498, 503, 503, 503, 504, 504, 504, 505, 505, 505, 441, 505, 439, 437, 505, 433, 430, 503, 509, 509, 509, 510, 510, 510, 511, 511, 511, 519, 519, 519, 527, 527, 527, 509, 427, 423, 510, 528, 528, 528, 511, 420, 418, 519, 414, 527, 529, 529, 529, 411, 529, 410, 528, 529, 532, 532, 532, 533, 533, 533, 535, 535, 535, 537, 537, 537, 399, 538, 538, 538, 398, 538, 397, 533, 538, 393, 511, 388, 537, 539, 539, 539, 540, 540, 540, 385, 540, 383, 380, 540, 541, 541, 541, 378, 539, 546, 546, 546, 547, 547, 547, 548, 548, 548, 375, 373, 541, 550, 550, 550, 546, 371, 369, 547, 368, 363, 548, 554, 554, 554, 555, 555, 555, 550, 556, 556, 556, 357, 556, 355, 354, 556, 554, 345, 340, 555, 557, 557, 557, 339, 557, 337, 333, 557, 560, 560, 560, 562, 562, 562, 563, 563, 563, 326, 563, 324, 323, 563, 321, 560, 320, 315, 562, 566, 566, 566, 314, 566, 313, 311, 566, 569, 569, 569, 575, 575, 575, 576, 576, 576, 310, 576, 309, 308, 576, 307, 306, 569, 305, 575, 577, 577, 577, 578, 578, 578, 302, 578, 301, 300, 578, 579, 579, 579, 298, 577, 580, 580, 580, 294, 580, 292, 289, 580, 288, 287, 579, 582, 582, 582, 585, 585, 585, 586, 586, 586, 286, 587, 587, 587, 285, 587, 284, 582, 587, 585, 283, 279, 586, 588, 588, 588, 276, 588, 275, 274, 588, 591, 591, 591, 582, 592, 592, 592, 273, 592, 270, 266, 592, 594, 594, 594, 591, 595, 595, 595, 265, 595, 264, 263, 595, 600, 600, 600, 594, 600, 262, 261, 600, 603, 603, 603, 260, 603, 257, 256, 603, 606, 606, 606, 255, 606, 254, 253, 606, 612, 612, 612, 613, 613, 613, 252, 613, 251, 250, 613, 249, 614, 614, 614, 612, 614, 248, 246, 614, 617, 617, 617, 245, 617, 244, 243, 617, 621, 621, 621, 242, 621, 238, 236, 621, 643, 643, 643, 235, 643, 234, 233, 643, 665, 665, 665, 681, 681, 681, 682, 682, 682, 683, 683, 683, 232, 231, 230, 665, 665, 229, 228, 681, 227, 682, 682, 226, 683, 683, 684, 684, 684, 225, 684, 222, 221, 684, 698, 698, 698, 699, 699, 699, 218, 217, 684, 216, 700, 700, 700, 215, 700, 214, 698, 700, 699, 699, 701, 701, 701, 213, 701, 209, 700, 701, 698, 208, 736, 736, 736, 753, 753, 753, 701, 702, 702, 702, 702, 702, 702, 702, 702, 702, 736, 736, 702, 753, 207, 702, 702, 702, 702, 702, 747, 747, 747, 204, 747, 203, 202, 747, 742, 742, 742, 201, 200, 702, 702, 703, 703, 703, 703, 703, 703, 703, 703, 703, 742, 199, 703, 198, 197, 703, 703, 703, 703, 703, 718, 718, 718, 196, 718, 195, 194, 718, 750, 750, 750, 742, 750, 703, 703, 750, 718, 721, 721, 721, 721, 721, 721, 721, 721, 721, 193, 192, 721, 191, 190, 721, 721, 721, 721, 721, 189, 188, 185, 184, 183, 182, 181, 175, 174, 743, 743, 743, 172, 721, 721, 722, 722, 722, 722, 722, 722, 722, 722, 722, 743, 169, 722, 168, 163, 722, 722, 722, 722, 722, 746, 746, 746, 754, 754, 754, 162, 754, 161, 160, 754, 159, 155, 722, 722, 746, 771, 771, 771, 754, 772, 772, 772, 149, 145, 137, 136, 135, 134, 133, 129, 121, 771, 117, 106, 105, 772, 775, 775, 775, 775, 775, 775, 775, 776, 776, 776, 776, 776, 776, 776, 777, 777, 777, 777, 777, 777, 777, 778, 778, 778, 778, 778, 778, 778, 779, 779, 779, 779, 779, 779, 779, 780, 780, 780, 780, 780, 780, 780, 781, 781, 781, 781, 781, 781, 781, 782, 782, 782, 782, 782, 782, 782, 783, 783, 783, 783, 783, 783, 783, 784, 784, 784, 784, 784, 784, 784, 785, 785, 785, 785, 785, 785, 785, 786, 786, 786, 786, 786, 786, 786, 787, 787, 787, 787, 787, 787, 787, 788, 788, 788, 788, 788, 788, 788, 789, 789, 789, 104, 789, 790, 790, 790, 103, 790, 791, 791, 791, 102, 791, 792, 792, 792, 97, 792, 793, 793, 793, 93, 793, 794, 794, 794, 794, 794, 92, 794, 795, 91, 795, 796, 85, 796, 797, 84, 797, 798, 82, 798, 799, 79, 799, 800, 800, 800, 800, 800, 800, 800, 801, 67, 0, 801, 802, 802, 802, 802, 802, 802, 803, 803, 803, 803, 803, 803, 804, 804, 804, 804, 804, 804, 805, 805, 805, 805, 805, 805, 806, 806, 806, 806, 806, 806, 807, 807, 807, 807, 807, 807, 807, 808, 808, 808, 808, 808, 808, 808, 809, 809, 809, 809, 809, 809, 809, 810, 810, 810, 810, 810, 810, 810, 811, 811, 811, 811, 811, 811, 811, 812, 812, 812, 812, 812, 812, 812, 813, 813, 813, 813, 813, 813, 813, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774, 774 } ; /* Table of booleans, true if rule could match eol. */ static yyconst flex_int32_t yy_rule_can_match_eol[141] = { 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, }; static yy_state_type yy_last_accepting_state; static char *yy_last_accepting_cpos; extern int dax__flex_debug; int dax__flex_debug = 0; /* The intent behind this definition is that it'll catch * any uses of REJECT which flex missed. */ #define REJECT reject_used_but_not_detected #define yymore() yymore_used_but_not_detected #define YY_MORE_ADJ 0 #define YY_RESTORE_YY_MORE_OFFSET char *dax_text; /* Validating XML processor for src/simdag/dax.dtd. * * This program was generated with the FleXML XML processor generator. * FleXML is Copyright (C) 1999-2005 Kristoffer Rose. All rights reserved. * FleXML is Copyright (C) 2003-2013 Martin Quinson. All rights reserved. * (1.9.6). * * There are two, intertwined parts to this program, part A and part B. * * Part A * ------ * * Some parts, here collectively called "Part A", are found in the * FleXML package. They are Copyright (C) 1999-2005 Kristoffer Rose * and Copyright (C) 2003-2013 Martin Quinson. All rights reserved. * * You can redistribute, use, perform, display and/or modify "Part A" * provided the following two conditions hold: * * 1. The program is distributed WITHOUT ANY WARRANTY from the author of * FleXML; without even the implied warranty of MERCHANTABILITY or * FITNESS FOR A PARTICULAR PURPOSE. * * 2. The program distribution conditions do not in any way affect the * distribution conditions of the FleXML system used to generate this * file or any version of FleXML derived from that system. * * Notice that these are explicit rights granted to you for files * generated by the FleXML system. For your rights in connection with * the FleXML system itself please consult the GNU General Public License. * * Part B * ------ * * The other parts, here collectively called "Part B", and which came * from the DTD used by FleXML to generate this program, can be * distributed (or not, as the case may be) under the terms of whoever * wrote them, provided these terms respect and obey the two conditions * above under the heading "Part A". * * The author of and contributors to FleXML specifically disclaim * any copyright interest in "Part B", unless "Part B" was written * by the author of or contributors to FleXML. * */ /* Version strings. */ const char dax__flexml_version[] = "1.9.6"; /* ANSI headers. */ #include /* for realloc() -- needed here when using flex 2.5.4 */ #include #include #include #include #include #if defined(_WIN32) || defined(__WIN32__) || defined(WIN32) || defined(__TOS_WIN__) # ifndef __STRICT_ANSI__ # include # include # endif #else # include #endif #ifndef FLEXML_INDEXSTACKSIZE #define FLEXML_INDEXSTACKSIZE 1000 #endif /* Generated definitions. */ #define FLEXML_yylineno #ifndef FLEXML_BUFFERSTACKSIZE #define FLEXML_BUFFERSTACKSIZE 1000000 #endif #define FLEXML_NEED_BUFFERLIT /* XML processor api. */ /* FleXML-provided data. */ int dax__pcdata_ix; extern char *dax__bufferstack; #define dax__pcdata (dax__bufferstack + dax__pcdata_ix) AT_dax__adag_childCount AX_dax__adag_childCount; #define A_dax__adag_childCount (dax__bufferstack + AX_dax__adag_childCount) short int dax__adag_childCount_isset; AT_dax__adag_count AX_dax__adag_count; #define A_dax__adag_count (dax__bufferstack + AX_dax__adag_count) short int dax__adag_count_isset; AT_dax__adag_fileCount AX_dax__adag_fileCount; #define A_dax__adag_fileCount (dax__bufferstack + AX_dax__adag_fileCount) short int dax__adag_fileCount_isset; AT_dax__adag_index AX_dax__adag_index; #define A_dax__adag_index (dax__bufferstack + AX_dax__adag_index) short int dax__adag_index_isset; AT_dax__adag_jobCount AX_dax__adag_jobCount; #define A_dax__adag_jobCount (dax__bufferstack + AX_dax__adag_jobCount) short int dax__adag_jobCount_isset; AT_dax__adag_name AX_dax__adag_name; #define A_dax__adag_name (dax__bufferstack + AX_dax__adag_name) short int dax__adag_name_isset; AT_dax__adag_version AX_dax__adag_version; #define A_dax__adag_version (dax__bufferstack + AX_dax__adag_version) short int dax__adag_version_isset; AT_dax__adag_xmlns AX_dax__adag_xmlns; #define A_dax__adag_xmlns (dax__bufferstack + AX_dax__adag_xmlns) short int dax__adag_xmlns_isset; AT_dax__adag_xmlns_c_xsi AX_dax__adag_xmlns_c_xsi; #define A_dax__adag_xmlns_c_xsi (dax__bufferstack + AX_dax__adag_xmlns_c_xsi) short int dax__adag_xmlns_c_xsi_isset; AT_dax__adag_xsi_c_schemaLocation AX_dax__adag_xsi_c_schemaLocation; #define A_dax__adag_xsi_c_schemaLocation (dax__bufferstack + AX_dax__adag_xsi_c_schemaLocation) short int dax__adag_xsi_c_schemaLocation_isset; AT_dax__child_ref AX_dax__child_ref; #define A_dax__child_ref (dax__bufferstack + AX_dax__child_ref) short int dax__child_ref_isset; AT_dax__job_id AX_dax__job_id; #define A_dax__job_id (dax__bufferstack + AX_dax__job_id) short int dax__job_id_isset; AT_dax__job_level AX_dax__job_level; #define A_dax__job_level (dax__bufferstack + AX_dax__job_level) short int dax__job_level_isset; AT_dax__job_name AX_dax__job_name; #define A_dax__job_name (dax__bufferstack + AX_dax__job_name) short int dax__job_name_isset; AT_dax__job_namespace AX_dax__job_namespace; #define A_dax__job_namespace (dax__bufferstack + AX_dax__job_namespace) short int dax__job_namespace_isset; AT_dax__job_runtime AX_dax__job_runtime; #define A_dax__job_runtime (dax__bufferstack + AX_dax__job_runtime) short int dax__job_runtime_isset; AT_dax__job_version AX_dax__job_version; #define A_dax__job_version (dax__bufferstack + AX_dax__job_version) short int dax__job_version_isset; AT_dax__parent_ref AX_dax__parent_ref; #define A_dax__parent_ref (dax__bufferstack + AX_dax__parent_ref) short int dax__parent_ref_isset; AT_dax__uses_file AX_dax__uses_file; #define A_dax__uses_file (dax__bufferstack + AX_dax__uses_file) short int dax__uses_file_isset; AT_dax__uses_link AX_dax__uses_link; #define A_dax__uses_link AX_dax__uses_link short int dax__uses_link_isset; AT_dax__uses_optional AX_dax__uses_optional; #define A_dax__uses_optional AX_dax__uses_optional short int dax__uses_optional_isset; AT_dax__uses_register AX_dax__uses_register; #define A_dax__uses_register AX_dax__uses_register short int dax__uses_register_isset; AT_dax__uses_size AX_dax__uses_size; #define A_dax__uses_size (dax__bufferstack + AX_dax__uses_size) short int dax__uses_size_isset; AT_dax__uses_transfer AX_dax__uses_transfer; #define A_dax__uses_transfer AX_dax__uses_transfer short int dax__uses_transfer_isset; AT_dax__uses_type AX_dax__uses_type; #define A_dax__uses_type (dax__bufferstack + AX_dax__uses_type) short int dax__uses_type_isset; /* XML state. */ #ifdef FLEX_DEBUG # define ENTER(state) debug_enter(state,#state) # define LEAVE debug_leave() # define SET(state) debug_set(state,#state) static void debug_enter(int, const char*); static void debug_leave(void); static void debug_set(int, const char*); #else # define ENTER(state) (yy_push_state(state)) # define LEAVE (yy_pop_state()) # define SET(state) BEGIN(state) #endif /* Generic actions. */ #define SKIP /*skip*/ #define SUCCEED CLEANUP; return 0 #define FAIL return fail static int fail(const char*, ...); enum {flexml_max_err_msg_size = 512}; static char flexml_err_msg[flexml_max_err_msg_size]; const char * dax__parse_err_msg() { return flexml_err_msg; } static void reset_dax__parse_err_msg() { flexml_err_msg[0] = '\0'; } /* Cleanup */ static void cleanup(void); #define CLEANUP cleanup() /* Text buffer stack handling. */ char *dax__bufferstack = NULL; static int blimit = FLEXML_BUFFERSTACKSIZE; static int bnext = 1; static int *indexstack = NULL; static int ilimit = FLEXML_INDEXSTACKSIZE; static int inext = 1; #define BUFFERSET(P) (P = bnext) #define BUFFERPUTC(C) (ck_blimit(), dax__bufferstack[bnext++] = (C)) #define BUFFERDONE (BUFFERPUTC('\0')) #define BUFFERLITERAL(C, P) dax__bufferliteral(C, &(P), dax_text) /* after this is called, there are at least 2 slots left in the stack */ static int ck_blimit() { if (bnext >= blimit) { blimit += FLEXML_BUFFERSTACKSIZE + 2; { char *temp = (char *) realloc(dax__bufferstack, blimit); assert(temp); dax__bufferstack = temp; } } return 0; } /* after this is called, there are at least 2 slots left in the stack */ static int ck_ilimit() { if (inext >= ilimit) { ilimit += FLEXML_INDEXSTACKSIZE + 2; { int *temp = (int *) realloc(indexstack, ilimit); assert(temp); indexstack = temp; } } return 0; } #ifdef FLEXML_NEED_BUFFERLIT static void dax__bufferliteral(char c, int* pp, const char* text) { BUFFERSET(*pp); if (c) { const char *s = strchr(text, c), *e = strrchr(text, c); assert(s && e && s <= e); ++s; while (s < e) { if (isspace(*s)) { BUFFERPUTC(' '); do ++s; while (s < e && isspace(*s)); } else BUFFERPUTC(*s++); } } else { const char *s = text; while (*s) BUFFERPUTC(*s++); } BUFFERDONE; } #endif static void pushbuffer(int p) { ck_ilimit(); indexstack[inext++] = p; indexstack[inext++] = bnext; } static int popbuffer(void) { assert(inext >= 2); bnext = indexstack[--inext]; return indexstack[--inext]; } /* General internal entities are `unput' back onto the input stream... */ #define ENTITYTEXT(T) \ { char *s = (T), *e = s+strlen(s);\ while (--e >= s) { unput(*e); }} /* Flex standard options. */ #define YY_NO_INPUT 1 /* Flex user-requested options. */ /* XML character classes (currently restricted to ASCII). */ /* "Common syntactic structures." */ /* "Names and Tokens." */ /* Miscellaneous. */ /* Parser states (flex `exclusive start conditions'): * * PROLOG the XML prolog of the document before * DOCTYPE the XML prolog of the document after * EPILOG after the root element * INCOMMENT inside an XML comment * INPI inside an XML PI * VALUE1 inside a '...'-delimited literal * VALUE2 inside a "..."-delimited literal * CDATA inside a section. * ROOT_ expect root element * AL_ inside the attribute list for * IN_ inside a with element contents (ready for end tag) * IMPOSSIBLE dummy to permit disabling rules; must be last */ /* State names. */ const char* *dax__statenames=NULL; #define INITIAL 0 #define PROLOG 1 #define DOCTYPE 2 #define EPILOG 3 #define INCOMMENT 4 #define INPI 5 #define VALUE1 6 #define VALUE2 7 #define CDATA 8 #define ROOT_dax__adag 9 #define AL_dax__adag 10 #define S_dax__adag 11 #define S_dax__adag_1 12 #define S_dax__adag_2 13 #define S_dax__adag_3 14 #define S_dax__adag_4 15 #define S_dax__adag_5 16 #define E_dax__adag 17 #define AL_dax__child 18 #define S_dax__child 19 #define S_dax__child_1 20 #define S_dax__child_2 21 #define E_dax__child 22 #define AL_dax__job 23 #define S_dax__job 24 #define S_dax__job_1 25 #define S_dax__job_2 26 #define E_dax__job 27 #define AL_dax__parent 28 #define E_dax__parent 29 #define AL_dax__uses 30 #define E_dax__uses 31 #define IMPOSSIBLE 32 #ifndef YY_NO_UNISTD_H /* Special case for "unistd.h", since it is non-ANSI. We include it way * down here because we want the user's section 1 to have been scanned first. * The user has a chance to override it with an option. */ #if defined(_WIN32) # ifndef __STRICT_ANSI__ # include # include # endif #else # include #endif #endif #ifndef YY_EXTRA_TYPE #define YY_EXTRA_TYPE void * #endif static int yy_init_globals (void ); /* Accessor methods to globals. These are made visible to non-reentrant scanners for convenience. */ int dax_lex_destroy (void ); int dax_get_debug (void ); void dax_set_debug (int debug_flag ); YY_EXTRA_TYPE dax_get_extra (void ); void dax_set_extra (YY_EXTRA_TYPE user_defined ); FILE *dax_get_in (void ); void dax_set_in (FILE * _in_str ); FILE *dax_get_out (void ); void dax_set_out (FILE * _out_str ); yy_size_t dax_get_leng (void ); char *dax_get_text (void ); int dax_get_lineno (void ); void dax_set_lineno (int _line_number ); /* Macros after this point can all be overridden by user definitions in * section 1. */ #ifndef YY_SKIP_YYWRAP #ifdef __cplusplus extern "C" int dax_wrap (void ); #else extern int dax_wrap (void ); #endif #endif #ifndef YY_NO_UNPUT #endif #ifndef yytext_ptr static void yy_flex_strncpy (char *,yyconst char *,int ); #endif #ifdef YY_NEED_STRLEN static int yy_flex_strlen (yyconst char * ); #endif #ifndef YY_NO_INPUT #ifdef __cplusplus static int yyinput (void ); #else static int input (void ); #endif #endif static int yy_start_stack_ptr = 0; static int yy_start_stack_depth = 0; static int *yy_start_stack = NULL; static void yy_push_state (int _new_state ); static void yy_pop_state (void ); /* Amount of stuff to slurp up with each read. */ #ifndef YY_READ_BUF_SIZE #ifdef __ia64__ /* On IA-64, the buffer size is 16k, not 8k */ #define YY_READ_BUF_SIZE 16384 #else #define YY_READ_BUF_SIZE 8192 #endif /* __ia64__ */ #endif /* Copy whatever the last rule matched to the standard output. */ #ifndef ECHO /* This used to be an fputs(), but since the string might contain NUL's, * we now use fwrite(). */ #define ECHO do { if (fwrite( dax_text, dax_leng, 1, dax_out )) {} } while (0) #endif /* Gets input and stuffs it into "buf". number of characters read, or YY_NULL, * is returned in "result". */ #ifndef YY_INPUT #define YY_INPUT(buf,result,max_size) \ if ( YY_CURRENT_BUFFER_LVALUE->yy_is_interactive ) \ { \ int c = '*'; \ size_t n; \ for ( n = 0; n < max_size && \ (c = getc( dax_in )) != EOF && c != '\n'; ++n ) \ buf[n] = (char) c; \ if ( c == '\n' ) \ buf[n++] = (char) c; \ if ( c == EOF && ferror( dax_in ) ) \ YY_FATAL_ERROR( "input in flex scanner failed" ); \ result = n; \ } \ else \ { \ errno=0; \ while ( (result = fread(buf, 1, max_size, dax_in))==0 && ferror(dax_in)) \ { \ if( errno != EINTR) \ { \ YY_FATAL_ERROR( "input in flex scanner failed" ); \ break; \ } \ errno=0; \ clearerr(dax_in); \ } \ }\ \ #endif /* No semi-colon after return; correct usage is to write "yyterminate();" - * we don't want an extra ';' after the "return" because that will cause * some compilers to complain about unreachable statements. */ #ifndef yyterminate #define yyterminate() return YY_NULL #endif /* Number of entries by which start-condition stack grows. */ #ifndef YY_START_STACK_INCR #define YY_START_STACK_INCR 25 #endif /* Report a fatal error. */ #ifndef YY_FATAL_ERROR #define YY_FATAL_ERROR(msg) yy_fatal_error( msg ) #endif /* end tables serialization structures and prototypes */ /* Default declaration of generated scanner - a define so the user can * easily add parameters. */ #ifndef YY_DECL #define YY_DECL_IS_OURS 1 extern int dax_lex (void); #define YY_DECL int dax_lex (void) #endif /* !YY_DECL */ /* Code executed at the beginning of each rule, after dax_text and dax_leng * have been set up. */ #ifndef YY_USER_ACTION #define YY_USER_ACTION #endif /* Code executed at the end of each rule. */ #ifndef YY_BREAK #define YY_BREAK /*LINTED*/break; #endif #define YY_RULE_SETUP \ YY_USER_ACTION /** The main scanner function which does all the work. */ YY_DECL { yy_state_type yy_current_state; char *yy_cp, *yy_bp; int yy_act; if ( !(yy_init) ) { (yy_init) = 1; #ifdef YY_USER_INIT YY_USER_INIT; #endif if ( ! (yy_start) ) (yy_start) = 1; /* first start state */ if ( ! dax_in ) dax_in = stdin; if ( ! dax_out ) dax_out = stdout; if ( ! YY_CURRENT_BUFFER ) { dax_ensure_buffer_stack (); YY_CURRENT_BUFFER_LVALUE = dax__create_buffer(dax_in,YY_BUF_SIZE ); } dax__load_buffer_state( ); } { /* Bypass Flex's default INITIAL state and begin by parsing the XML prolog. */ SET(PROLOG); reset_dax__parse_err_msg(); dax__bufferstack = (char *) malloc(FLEXML_BUFFERSTACKSIZE); assert(dax__bufferstack); #ifdef FLEX_DEBUG { int i; for (i = 0; i < blimit; i++) { dax__bufferstack[i] = '\377'; } } #endif dax__bufferstack[0] = '\0'; indexstack = (int *) malloc(FLEXML_INDEXSTACKSIZE * sizeof(int)); assert(indexstack); indexstack[0] = 0; /* FleXML_init */ bnext = inext = 1; dax__bufferliteral('\0', &bnext, "http://pegasus.isi.edu/schema/DAX"); dax__bufferliteral('\0', &bnext, "http://www.w3.org/2001/XMLSchema-instance"); dax__bufferliteral('\0', &bnext, "http://pegasus.isi.edu/schema/DAX http://pegasus.isi.edu/schema/dax-2.1.xsd"); dax__bufferliteral('\0', &bnext, "1.0"); dax__bufferliteral('\0', &bnext, "0.0"); dax__bufferliteral('\0', &bnext, "data"); if(!dax__statenames) {dax__statenames= (const char **)calloc(IMPOSSIBLE,sizeof(char*)); dax__statenames[PROLOG] = NULL; dax__statenames[DOCTYPE] = NULL; dax__statenames[EPILOG] = NULL; dax__statenames[INCOMMENT] = NULL; dax__statenames[INPI] = NULL; dax__statenames[VALUE1] = NULL; dax__statenames[VALUE2] = NULL; dax__statenames[CDATA] = NULL; dax__statenames[ROOT_dax__adag] = NULL; dax__statenames[AL_dax__adag] = NULL; dax__statenames[S_dax__adag] = "adag"; dax__statenames[S_dax__adag_1] = "adag"; dax__statenames[S_dax__adag_2] = "adag"; dax__statenames[S_dax__adag_3] = "adag"; dax__statenames[S_dax__adag_4] = "adag"; dax__statenames[S_dax__adag_5] = "adag"; dax__statenames[E_dax__adag] = "adag"; dax__statenames[AL_dax__child] = NULL; dax__statenames[S_dax__child] = "child"; dax__statenames[S_dax__child_1] = "child"; dax__statenames[S_dax__child_2] = "child"; dax__statenames[E_dax__child] = "child"; dax__statenames[AL_dax__job] = NULL; dax__statenames[S_dax__job] = "job"; dax__statenames[S_dax__job_1] = "job"; dax__statenames[S_dax__job_2] = "job"; dax__statenames[E_dax__job] = "job"; dax__statenames[AL_dax__parent] = NULL; dax__statenames[E_dax__parent] = "parent"; dax__statenames[AL_dax__uses] = NULL; dax__statenames[E_dax__uses] = "uses"; } /* COMMENTS and PIs: handled uniformly for efficiency. */ while ( /*CONSTCOND*/1 ) /* loops until end-of-file is reached */ { yy_cp = (yy_c_buf_p); /* Support of dax_text. */ *yy_cp = (yy_hold_char); /* yy_bp points to the position in yy_ch_buf of the start of * the current run. */ yy_bp = yy_cp; yy_current_state = (yy_start); yy_match: do { YY_CHAR yy_c = yy_ec[YY_SC_TO_UI(*yy_cp)] ; if ( yy_accept[yy_current_state] ) { (yy_last_accepting_state) = yy_current_state; (yy_last_accepting_cpos) = yy_cp; } while ( yy_chk[yy_base[yy_current_state] + yy_c] != yy_current_state ) { yy_current_state = (int) yy_def[yy_current_state]; if ( yy_current_state >= 775 ) yy_c = yy_meta[(unsigned int) yy_c]; } yy_current_state = yy_nxt[yy_base[yy_current_state] + (unsigned int) yy_c]; ++yy_cp; } while ( yy_base[yy_current_state] != 2135 ); yy_find_action: yy_act = yy_accept[yy_current_state]; if ( yy_act == 0 ) { /* have to back up */ yy_cp = (yy_last_accepting_cpos); yy_current_state = (yy_last_accepting_state); yy_act = yy_accept[yy_current_state]; } YY_DO_BEFORE_ACTION; if ( yy_act != YY_END_OF_BUFFER && yy_rule_can_match_eol[yy_act] ) { yy_size_t yyl; for ( yyl = 0; yyl < dax_leng; ++yyl ) if ( dax_text[yyl] == '\n' ) dax_lineno++; ; } do_action: /* This label is used only to access EOF actions. */ switch ( yy_act ) { /* beginning of action switch */ case 0: /* must back up */ /* undo the effects of YY_DO_BEFORE_ACTION */ *yy_cp = (yy_hold_char); yy_cp = (yy_last_accepting_cpos); yy_current_state = (yy_last_accepting_state); goto yy_find_action; case 1: YY_RULE_SETUP ENTER(INCOMMENT); YY_BREAK case 2: YY_RULE_SETUP ENTER(INPI); YY_BREAK case 3: YY_RULE_SETUP LEAVE; YY_BREAK case 4: case 5: case 6: /* rule 6 can match eol */ YY_RULE_SETUP SKIP; YY_BREAK case YY_STATE_EOF(INCOMMENT): FAIL("EOF in comment."); YY_BREAK case 7: YY_RULE_SETUP LEAVE; YY_BREAK case 8: case 9: /* rule 9 can match eol */ YY_RULE_SETUP SKIP; YY_BREAK case YY_STATE_EOF(INPI): FAIL("EOF in PI (processing instruction)."); YY_BREAK /* SPACES: skipped uniformly */ case 10: /* rule 10 can match eol */ YY_RULE_SETUP SKIP; YY_BREAK /* PROLOG: determine root element and process it. */ case 11: /* rule 11 can match eol */ YY_RULE_SETUP SET(ROOT_dax__adag); YY_BREAK case 12: /* rule 12 can match eol */ YY_RULE_SETUP FAIL("Bad declaration %s.",dax_text); YY_BREAK case 13: /* rule 13 can match eol */ YY_RULE_SETUP SET(ROOT_dax__adag); YY_BREAK case 14: /* rule 14 can match eol */ YY_RULE_SETUP FAIL("Bad declaration %s.",dax_text); YY_BREAK case 15: YY_RULE_SETUP FAIL("Unexpected character `%c' in prolog.", dax_text[0]); YY_BREAK case YY_STATE_EOF(PROLOG): case YY_STATE_EOF(DOCTYPE): FAIL("EOF in prolog."); YY_BREAK /* RULES DERIVED FROM DTD. */ /* */ case 16: /* rule 16 can match eol */ YY_RULE_SETUP FAIL("Starting tag is not allowed here."); YY_BREAK case 17: /* rule 17 can match eol */ YY_RULE_SETUP { AX_dax__adag_childCount = 0; dax__adag_childCount_isset = 0; AX_dax__adag_count = 0; dax__adag_count_isset = 0; AX_dax__adag_fileCount = 0; dax__adag_fileCount_isset = 0; AX_dax__adag_index = 0; dax__adag_index_isset = 0; AX_dax__adag_jobCount = 0; dax__adag_jobCount_isset = 0; AX_dax__adag_name = 0; dax__adag_name_isset = 0; AX_dax__adag_version = 153; dax__adag_version_isset = 0; AX_dax__adag_xmlns = 1; dax__adag_xmlns_isset = 0; AX_dax__adag_xmlns_c_xsi = 35; dax__adag_xmlns_c_xsi_isset = 0; AX_dax__adag_xsi_c_schemaLocation = 77; dax__adag_xsi_c_schemaLocation_isset = 0; ENTER(AL_dax__adag); pushbuffer(0); } YY_BREAK case 18: /* rule 18 can match eol */ YY_RULE_SETUP if (dax__adag_childCount_isset != 0) {FAIL("Multiple definition of attribute childCount in ");} dax__adag_childCount_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__adag_childCount); YY_BREAK case 19: /* rule 19 can match eol */ YY_RULE_SETUP if (dax__adag_childCount_isset != 0) {FAIL("Multiple definition of attribute childCount in ");} dax__adag_childCount_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__adag_childCount); YY_BREAK case 20: /* rule 20 can match eol */ YY_RULE_SETUP if (dax__adag_count_isset != 0) {FAIL("Multiple definition of attribute count in ");} dax__adag_count_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__adag_count); YY_BREAK case 21: /* rule 21 can match eol */ YY_RULE_SETUP if (dax__adag_count_isset != 0) {FAIL("Multiple definition of attribute count in ");} dax__adag_count_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__adag_count); YY_BREAK case 22: /* rule 22 can match eol */ YY_RULE_SETUP if (dax__adag_fileCount_isset != 0) {FAIL("Multiple definition of attribute fileCount in ");} dax__adag_fileCount_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__adag_fileCount); YY_BREAK case 23: /* rule 23 can match eol */ YY_RULE_SETUP if (dax__adag_fileCount_isset != 0) {FAIL("Multiple definition of attribute fileCount in ");} dax__adag_fileCount_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__adag_fileCount); YY_BREAK case 24: /* rule 24 can match eol */ YY_RULE_SETUP if (dax__adag_index_isset != 0) {FAIL("Multiple definition of attribute index in ");} dax__adag_index_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__adag_index); YY_BREAK case 25: /* rule 25 can match eol */ YY_RULE_SETUP if (dax__adag_index_isset != 0) {FAIL("Multiple definition of attribute index in ");} dax__adag_index_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__adag_index); YY_BREAK case 26: /* rule 26 can match eol */ YY_RULE_SETUP if (dax__adag_jobCount_isset != 0) {FAIL("Multiple definition of attribute jobCount in ");} dax__adag_jobCount_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__adag_jobCount); YY_BREAK case 27: /* rule 27 can match eol */ YY_RULE_SETUP if (dax__adag_jobCount_isset != 0) {FAIL("Multiple definition of attribute jobCount in ");} dax__adag_jobCount_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__adag_jobCount); YY_BREAK case 28: /* rule 28 can match eol */ YY_RULE_SETUP if (dax__adag_name_isset != 0) {FAIL("Multiple definition of attribute name in ");} dax__adag_name_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__adag_name); YY_BREAK case 29: /* rule 29 can match eol */ YY_RULE_SETUP if (dax__adag_name_isset != 0) {FAIL("Multiple definition of attribute name in ");} dax__adag_name_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__adag_name); YY_BREAK case 30: /* rule 30 can match eol */ YY_RULE_SETUP if (dax__adag_version_isset != 0) {FAIL("Multiple definition of attribute version in ");} dax__adag_version_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__adag_version); YY_BREAK case 31: /* rule 31 can match eol */ YY_RULE_SETUP if (dax__adag_version_isset != 0) {FAIL("Multiple definition of attribute version in ");} dax__adag_version_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__adag_version); YY_BREAK case 32: /* rule 32 can match eol */ YY_RULE_SETUP if (dax__adag_xmlns_isset != 0) {FAIL("Multiple definition of attribute xmlns in ");} dax__adag_xmlns_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__adag_xmlns); YY_BREAK case 33: /* rule 33 can match eol */ YY_RULE_SETUP if (dax__adag_xmlns_isset != 0) {FAIL("Multiple definition of attribute xmlns in ");} dax__adag_xmlns_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__adag_xmlns); YY_BREAK case 34: /* rule 34 can match eol */ YY_RULE_SETUP if (dax__adag_xmlns_c_xsi_isset != 0) {FAIL("Multiple definition of attribute xmlns:xsi in ");} dax__adag_xmlns_c_xsi_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__adag_xmlns_c_xsi); YY_BREAK case 35: /* rule 35 can match eol */ YY_RULE_SETUP if (dax__adag_xmlns_c_xsi_isset != 0) {FAIL("Multiple definition of attribute xmlns:xsi in ");} dax__adag_xmlns_c_xsi_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__adag_xmlns_c_xsi); YY_BREAK case 36: /* rule 36 can match eol */ YY_RULE_SETUP if (dax__adag_xsi_c_schemaLocation_isset != 0) {FAIL("Multiple definition of attribute xsi:schemaLocation in ");} dax__adag_xsi_c_schemaLocation_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__adag_xsi_c_schemaLocation); YY_BREAK case 37: /* rule 37 can match eol */ YY_RULE_SETUP if (dax__adag_xsi_c_schemaLocation_isset != 0) {FAIL("Multiple definition of attribute xsi:schemaLocation in ");} dax__adag_xsi_c_schemaLocation_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__adag_xsi_c_schemaLocation); YY_BREAK case 38: YY_RULE_SETUP { LEAVE; STag_dax__adag();dax__pcdata_ix = 0; ENTER(S_dax__adag); } YY_BREAK case 39: YY_RULE_SETUP { LEAVE; STag_dax__adag(); dax__pcdata_ix = 0; ETag_dax__adag(); popbuffer(); /* attribute */ switch (YY_START) { case ROOT_dax__adag: SET(EPILOG); break; } } YY_BREAK case 40: YY_RULE_SETUP FAIL("Unexpected character `%c' in attribute list of adag element.", dax_text[0]); YY_BREAK case 41: YY_RULE_SETUP FAIL("Bad attribute `%s' in `adag' element start tag.",dax_text); YY_BREAK case YY_STATE_EOF(AL_dax__adag): FAIL("EOF in attribute list of `adag' element."); YY_BREAK case 42: /* rule 42 can match eol */ YY_RULE_SETUP { LEAVE; ETag_dax__adag(); popbuffer(); /* attribute */ switch (YY_START) { case ROOT_dax__adag: SET(EPILOG); break; } } YY_BREAK case 43: /* rule 43 can match eol */ YY_RULE_SETUP FAIL("Unexpected end-tag `%s': `' expected.",dax_text); YY_BREAK case 44: YY_RULE_SETUP FAIL("Unexpected character `%c': `' expected.",dax_text[0]); YY_BREAK case YY_STATE_EOF(E_dax__adag): case YY_STATE_EOF(S_dax__adag): case YY_STATE_EOF(S_dax__adag_1): case YY_STATE_EOF(S_dax__adag_3): case YY_STATE_EOF(S_dax__adag_5): FAIL("Premature EOF: `' expected."); YY_BREAK case 45: /* rule 45 can match eol */ YY_RULE_SETUP FAIL("Starting tag is not allowed here."); YY_BREAK case 46: /* rule 46 can match eol */ YY_RULE_SETUP { AX_dax__child_ref = 0; dax__child_ref_isset = 0; ENTER(AL_dax__child); pushbuffer(0); } YY_BREAK case 47: /* rule 47 can match eol */ YY_RULE_SETUP if (dax__child_ref_isset != 0) {FAIL("Multiple definition of attribute ref in ");} dax__child_ref_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__child_ref); YY_BREAK case 48: /* rule 48 can match eol */ YY_RULE_SETUP if (dax__child_ref_isset != 0) {FAIL("Multiple definition of attribute ref in ");} dax__child_ref_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__child_ref); YY_BREAK case 49: YY_RULE_SETUP { if (!AX_dax__child_ref) FAIL("Required attribute `ref' not set for `child' element."); LEAVE; STag_dax__child();dax__pcdata_ix = 0; ENTER(S_dax__child); } YY_BREAK case 50: YY_RULE_SETUP { if (!AX_dax__child_ref) FAIL("Required attribute `ref' not set for `child' element."); LEAVE; STag_dax__child(); dax__pcdata_ix = 0; ETag_dax__child(); popbuffer(); /* attribute */ switch (YY_START) { case S_dax__adag: case S_dax__adag_1: case S_dax__adag_3: case S_dax__adag_4: case S_dax__adag_5: SET(S_dax__adag_5); break; } } YY_BREAK case 51: YY_RULE_SETUP FAIL("Unexpected character `%c' in attribute list of child element.", dax_text[0]); YY_BREAK case 52: YY_RULE_SETUP FAIL("Bad attribute `%s' in `child' element start tag.",dax_text); YY_BREAK case YY_STATE_EOF(AL_dax__child): FAIL("EOF in attribute list of `child' element."); YY_BREAK case 53: /* rule 53 can match eol */ YY_RULE_SETUP { LEAVE; ETag_dax__child(); popbuffer(); /* attribute */ switch (YY_START) { case S_dax__adag: case S_dax__adag_1: case S_dax__adag_3: case S_dax__adag_4: case S_dax__adag_5: SET(S_dax__adag_5); break; } } YY_BREAK case 54: /* rule 54 can match eol */ YY_RULE_SETUP FAIL("Unexpected end-tag `%s': `' expected.",dax_text); YY_BREAK case 55: YY_RULE_SETUP FAIL("Unexpected character `%c': `' expected.",dax_text[0]); YY_BREAK case YY_STATE_EOF(E_dax__child): case YY_STATE_EOF(S_dax__child): case YY_STATE_EOF(S_dax__child_2): FAIL("Premature EOF: `' expected."); YY_BREAK case 56: /* rule 56 can match eol */ YY_RULE_SETUP FAIL("Starting tag is not allowed here."); YY_BREAK case 57: /* rule 57 can match eol */ YY_RULE_SETUP { AX_dax__job_id = 0; dax__job_id_isset = 0; AX_dax__job_level = 0; dax__job_level_isset = 0; AX_dax__job_name = 0; dax__job_name_isset = 0; AX_dax__job_namespace = 0; dax__job_namespace_isset = 0; AX_dax__job_runtime = 0; dax__job_runtime_isset = 0; AX_dax__job_version = 157; dax__job_version_isset = 0; ENTER(AL_dax__job); pushbuffer(0); } YY_BREAK case 58: /* rule 58 can match eol */ YY_RULE_SETUP if (dax__job_id_isset != 0) {FAIL("Multiple definition of attribute id in ");} dax__job_id_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__job_id); YY_BREAK case 59: /* rule 59 can match eol */ YY_RULE_SETUP if (dax__job_id_isset != 0) {FAIL("Multiple definition of attribute id in ");} dax__job_id_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__job_id); YY_BREAK case 60: /* rule 60 can match eol */ YY_RULE_SETUP if (dax__job_level_isset != 0) {FAIL("Multiple definition of attribute level in ");} dax__job_level_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__job_level); YY_BREAK case 61: /* rule 61 can match eol */ YY_RULE_SETUP if (dax__job_level_isset != 0) {FAIL("Multiple definition of attribute level in ");} dax__job_level_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__job_level); YY_BREAK case 62: /* rule 62 can match eol */ YY_RULE_SETUP if (dax__job_name_isset != 0) {FAIL("Multiple definition of attribute name in ");} dax__job_name_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__job_name); YY_BREAK case 63: /* rule 63 can match eol */ YY_RULE_SETUP if (dax__job_name_isset != 0) {FAIL("Multiple definition of attribute name in ");} dax__job_name_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__job_name); YY_BREAK case 64: /* rule 64 can match eol */ YY_RULE_SETUP if (dax__job_namespace_isset != 0) {FAIL("Multiple definition of attribute namespace in ");} dax__job_namespace_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__job_namespace); YY_BREAK case 65: /* rule 65 can match eol */ YY_RULE_SETUP if (dax__job_namespace_isset != 0) {FAIL("Multiple definition of attribute namespace in ");} dax__job_namespace_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__job_namespace); YY_BREAK case 66: /* rule 66 can match eol */ YY_RULE_SETUP if (dax__job_runtime_isset != 0) {FAIL("Multiple definition of attribute runtime in ");} dax__job_runtime_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__job_runtime); YY_BREAK case 67: /* rule 67 can match eol */ YY_RULE_SETUP if (dax__job_runtime_isset != 0) {FAIL("Multiple definition of attribute runtime in ");} dax__job_runtime_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__job_runtime); YY_BREAK case 68: /* rule 68 can match eol */ YY_RULE_SETUP if (dax__job_version_isset != 0) {FAIL("Multiple definition of attribute version in ");} dax__job_version_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__job_version); YY_BREAK case 69: /* rule 69 can match eol */ YY_RULE_SETUP if (dax__job_version_isset != 0) {FAIL("Multiple definition of attribute version in ");} dax__job_version_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__job_version); YY_BREAK case 70: YY_RULE_SETUP { if (!AX_dax__job_id) FAIL("Required attribute `id' not set for `job' element."); if (!AX_dax__job_name) FAIL("Required attribute `name' not set for `job' element."); if (!AX_dax__job_runtime) FAIL("Required attribute `runtime' not set for `job' element."); LEAVE; STag_dax__job();dax__pcdata_ix = 0; ENTER(S_dax__job); } YY_BREAK case 71: YY_RULE_SETUP { if (!AX_dax__job_id) FAIL("Required attribute `id' not set for `job' element."); if (!AX_dax__job_name) FAIL("Required attribute `name' not set for `job' element."); if (!AX_dax__job_runtime) FAIL("Required attribute `runtime' not set for `job' element."); LEAVE; STag_dax__job(); dax__pcdata_ix = 0; ETag_dax__job(); popbuffer(); /* attribute */ switch (YY_START) { case S_dax__adag: case S_dax__adag_2: case S_dax__adag_3: SET(S_dax__adag_3); break; } } YY_BREAK case 72: YY_RULE_SETUP FAIL("Unexpected character `%c' in attribute list of job element.", dax_text[0]); YY_BREAK case 73: YY_RULE_SETUP FAIL("Bad attribute `%s' in `job' element start tag.",dax_text); YY_BREAK case YY_STATE_EOF(AL_dax__job): FAIL("EOF in attribute list of `job' element."); YY_BREAK case 74: /* rule 74 can match eol */ YY_RULE_SETUP { LEAVE; ETag_dax__job(); popbuffer(); /* attribute */ switch (YY_START) { case S_dax__adag: case S_dax__adag_2: case S_dax__adag_3: SET(S_dax__adag_3); break; } } YY_BREAK case 75: /* rule 75 can match eol */ YY_RULE_SETUP FAIL("Unexpected end-tag `%s': `' expected.",dax_text); YY_BREAK case 76: YY_RULE_SETUP FAIL("Unexpected character `%c': `' expected.",dax_text[0]); YY_BREAK case YY_STATE_EOF(E_dax__job): case YY_STATE_EOF(S_dax__job): case YY_STATE_EOF(S_dax__job_2): FAIL("Premature EOF: `' expected."); YY_BREAK case 77: /* rule 77 can match eol */ YY_RULE_SETUP FAIL("Starting tag is not allowed here."); YY_BREAK case 78: /* rule 78 can match eol */ YY_RULE_SETUP { AX_dax__parent_ref = 0; dax__parent_ref_isset = 0; ENTER(AL_dax__parent); pushbuffer(0); } YY_BREAK case 79: /* rule 79 can match eol */ YY_RULE_SETUP if (dax__parent_ref_isset != 0) {FAIL("Multiple definition of attribute ref in ");} dax__parent_ref_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__parent_ref); YY_BREAK case 80: /* rule 80 can match eol */ YY_RULE_SETUP if (dax__parent_ref_isset != 0) {FAIL("Multiple definition of attribute ref in ");} dax__parent_ref_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__parent_ref); YY_BREAK case 81: YY_RULE_SETUP { if (!AX_dax__parent_ref) FAIL("Required attribute `ref' not set for `parent' element."); LEAVE; STag_dax__parent();dax__pcdata_ix = 0; ENTER(E_dax__parent); } YY_BREAK case 82: YY_RULE_SETUP { if (!AX_dax__parent_ref) FAIL("Required attribute `ref' not set for `parent' element."); LEAVE; STag_dax__parent(); dax__pcdata_ix = 0; ETag_dax__parent(); popbuffer(); /* attribute */ switch (YY_START) { case S_dax__child: case S_dax__child_1: case S_dax__child_2: SET(S_dax__child_2); break; } } YY_BREAK case 83: YY_RULE_SETUP FAIL("Unexpected character `%c' in attribute list of parent element.", dax_text[0]); YY_BREAK case 84: YY_RULE_SETUP FAIL("Bad attribute `%s' in `parent' element start tag.",dax_text); YY_BREAK case YY_STATE_EOF(AL_dax__parent): FAIL("EOF in attribute list of `parent' element."); YY_BREAK case 85: /* rule 85 can match eol */ YY_RULE_SETUP { LEAVE; ETag_dax__parent(); popbuffer(); /* attribute */ switch (YY_START) { case S_dax__child: case S_dax__child_1: case S_dax__child_2: SET(S_dax__child_2); break; } } YY_BREAK case 86: /* rule 86 can match eol */ YY_RULE_SETUP FAIL("Unexpected end-tag `%s': `' expected.",dax_text); YY_BREAK case 87: YY_RULE_SETUP FAIL("Unexpected character `%c': `' expected.",dax_text[0]); YY_BREAK case YY_STATE_EOF(E_dax__parent): FAIL("Premature EOF: `' expected."); YY_BREAK /* * */ case 88: /* rule 88 can match eol */ YY_RULE_SETUP FAIL("Starting tag is not allowed here."); YY_BREAK case 89: /* rule 89 can match eol */ YY_RULE_SETUP { AX_dax__uses_file = 0; dax__uses_file_isset = 0; AX_dax__uses_link = AU_dax__uses_link; dax__uses_link_isset = 0; AX_dax__uses_optional = A_dax__uses_optional_false; dax__uses_optional_isset = 0; AX_dax__uses_register = A_dax__uses_register_true; dax__uses_register_isset = 0; AX_dax__uses_size = 0; dax__uses_size_isset = 0; AX_dax__uses_transfer = A_dax__uses_transfer_true; dax__uses_transfer_isset = 0; AX_dax__uses_type = 161; dax__uses_type_isset = 0; ENTER(AL_dax__uses); pushbuffer(0); } YY_BREAK case 90: /* rule 90 can match eol */ YY_RULE_SETUP if (dax__uses_file_isset != 0) {FAIL("Multiple definition of attribute file in ");} dax__uses_file_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__uses_file); YY_BREAK case 91: /* rule 91 can match eol */ YY_RULE_SETUP if (dax__uses_file_isset != 0) {FAIL("Multiple definition of attribute file in ");} dax__uses_file_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__uses_file); YY_BREAK case 92: /* rule 92 can match eol */ case 93: /* rule 93 can match eol */ YY_RULE_SETUP A_dax__uses_link = A_dax__uses_link_input; YY_BREAK case 94: /* rule 94 can match eol */ case 95: /* rule 95 can match eol */ YY_RULE_SETUP A_dax__uses_link = A_dax__uses_link_output; YY_BREAK case 96: /* rule 96 can match eol */ case 97: /* rule 97 can match eol */ YY_RULE_SETUP A_dax__uses_optional = A_dax__uses_optional_false; YY_BREAK case 98: /* rule 98 can match eol */ case 99: /* rule 99 can match eol */ YY_RULE_SETUP A_dax__uses_optional = A_dax__uses_optional_true; YY_BREAK case 100: /* rule 100 can match eol */ case 101: /* rule 101 can match eol */ YY_RULE_SETUP A_dax__uses_register = A_dax__uses_register_false; YY_BREAK case 102: /* rule 102 can match eol */ case 103: /* rule 103 can match eol */ YY_RULE_SETUP A_dax__uses_register = A_dax__uses_register_true; YY_BREAK case 104: /* rule 104 can match eol */ YY_RULE_SETUP if (dax__uses_size_isset != 0) {FAIL("Multiple definition of attribute size in ");} dax__uses_size_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__uses_size); YY_BREAK case 105: /* rule 105 can match eol */ YY_RULE_SETUP if (dax__uses_size_isset != 0) {FAIL("Multiple definition of attribute size in ");} dax__uses_size_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__uses_size); YY_BREAK case 106: /* rule 106 can match eol */ case 107: /* rule 107 can match eol */ YY_RULE_SETUP A_dax__uses_transfer = A_dax__uses_transfer_false; YY_BREAK case 108: /* rule 108 can match eol */ case 109: /* rule 109 can match eol */ YY_RULE_SETUP A_dax__uses_transfer = A_dax__uses_transfer_true; YY_BREAK case 110: /* rule 110 can match eol */ YY_RULE_SETUP if (dax__uses_type_isset != 0) {FAIL("Multiple definition of attribute type in ");} dax__uses_type_isset = 1; ENTER(VALUE1); BUFFERSET(AX_dax__uses_type); YY_BREAK case 111: /* rule 111 can match eol */ YY_RULE_SETUP if (dax__uses_type_isset != 0) {FAIL("Multiple definition of attribute type in ");} dax__uses_type_isset = 1; ENTER(VALUE2); BUFFERSET(AX_dax__uses_type); YY_BREAK case 112: YY_RULE_SETUP { if (!AX_dax__uses_file) FAIL("Required attribute `file' not set for `uses' element."); if (!AX_dax__uses_size) FAIL("Required attribute `size' not set for `uses' element."); LEAVE; STag_dax__uses();dax__pcdata_ix = 0; ENTER(E_dax__uses); } YY_BREAK case 113: YY_RULE_SETUP { if (!AX_dax__uses_file) FAIL("Required attribute `file' not set for `uses' element."); if (!AX_dax__uses_size) FAIL("Required attribute `size' not set for `uses' element."); LEAVE; STag_dax__uses(); dax__pcdata_ix = 0; ETag_dax__uses(); popbuffer(); /* attribute */ switch (YY_START) { case S_dax__job: case S_dax__job_1: case S_dax__job_2: SET(S_dax__job_2); break; } } YY_BREAK case 114: YY_RULE_SETUP FAIL("Unexpected character `%c' in attribute list of uses element.", dax_text[0]); YY_BREAK case 115: YY_RULE_SETUP FAIL("Bad attribute `%s' in `uses' element start tag.",dax_text); YY_BREAK case YY_STATE_EOF(AL_dax__uses): FAIL("EOF in attribute list of `uses' element."); YY_BREAK case 116: /* rule 116 can match eol */ YY_RULE_SETUP { LEAVE; ETag_dax__uses(); popbuffer(); /* attribute */ switch (YY_START) { case S_dax__job: case S_dax__job_1: case S_dax__job_2: SET(S_dax__job_2); break; } } YY_BREAK case 117: /* rule 117 can match eol */ YY_RULE_SETUP FAIL("Unexpected end-tag `%s': `' expected.",dax_text); YY_BREAK case 118: YY_RULE_SETUP FAIL("Unexpected character `%c': `' expected.",dax_text[0]); YY_BREAK case YY_STATE_EOF(E_dax__uses): FAIL("Premature EOF: `' expected."); YY_BREAK /* EPILOG: after the root element. */ case 119: YY_RULE_SETUP {SET(PROLOG); yyless(0); CLEANUP; return -1;} YY_BREAK case YY_STATE_EOF(EPILOG): SUCCEED; YY_BREAK /* CHARACTER DATA. */ /* Non-defined standard entities... */ case 120: YY_RULE_SETUP BUFFERPUTC('&'); YY_BREAK case 121: YY_RULE_SETUP BUFFERPUTC('<'); YY_BREAK case 122: YY_RULE_SETUP BUFFERPUTC('>'); YY_BREAK case 123: YY_RULE_SETUP BUFFERPUTC('\''); YY_BREAK case 124: YY_RULE_SETUP BUFFERPUTC('"'); YY_BREAK /* Character entities. */ case 125: YY_RULE_SETUP BUFFERPUTC((unsigned char)atoi(dax_text+2)); YY_BREAK case 126: YY_RULE_SETUP BUFFERPUTC((unsigned char)strtol(dax_text+3,NULL,16)); YY_BREAK case 127: /* rule 127 can match eol */ case 128: /* rule 128 can match eol */ case 129: /* rule 129 can match eol */ case 130: /* rule 130 can match eol */ YY_RULE_SETUP BUFFERPUTC('\n'); YY_BREAK case 131: YY_RULE_SETUP ENTER(CDATA); YY_BREAK case 132: YY_RULE_SETUP FAIL("Unexpected `]""]>' in character data."); YY_BREAK case 133: YY_RULE_SETUP BUFFERDONE; LEAVE; YY_BREAK case YY_STATE_EOF(VALUE1): FAIL("EOF in literal (\"'\" expected)."); YY_BREAK case 134: YY_RULE_SETUP BUFFERDONE; LEAVE; YY_BREAK case YY_STATE_EOF(VALUE2): FAIL("EOF in literal (`\"' expected)."); YY_BREAK case 135: /* rule 135 can match eol */ YY_RULE_SETUP BUFFERPUTC(dax_text[0]); YY_BREAK case 136: YY_RULE_SETUP FAIL("Spurious `%c' in character data.",dax_text[0]); YY_BREAK case 137: YY_RULE_SETUP LEAVE; YY_BREAK /* "]""]" BUFFERPUTC(dax_text[0]); BUFFERPUTC(dax_text[1]); */ case 138: YY_RULE_SETUP BUFFERPUTC(dax_text[0]); YY_BREAK case YY_STATE_EOF(CDATA): FAIL("EOF in CDATA section."); YY_BREAK /* Impossible rules to avoid warnings from flex(1). */ /* Ideally, this should be replaced by code in flexml.pl that generates just the states not covered by other rules. */ case 139: /* rule 139 can match eol */ YY_RULE_SETUP FAIL("Syntax error on character `%c'.", dax_text[0]); YY_BREAK case 140: YY_RULE_SETUP ECHO; YY_BREAK case YY_STATE_EOF(INITIAL): case YY_STATE_EOF(ROOT_dax__adag): case YY_STATE_EOF(S_dax__adag_2): case YY_STATE_EOF(S_dax__adag_4): case YY_STATE_EOF(S_dax__child_1): case YY_STATE_EOF(S_dax__job_1): case YY_STATE_EOF(IMPOSSIBLE): yyterminate(); case YY_END_OF_BUFFER: { /* Amount of text matched not including the EOB char. */ int yy_amount_of_matched_text = (int) (yy_cp - (yytext_ptr)) - 1; /* Undo the effects of YY_DO_BEFORE_ACTION. */ *yy_cp = (yy_hold_char); YY_RESTORE_YY_MORE_OFFSET if ( YY_CURRENT_BUFFER_LVALUE->yy_buffer_status == YY_BUFFER_NEW ) { /* We're scanning a new file or input source. It's * possible that this happened because the user * just pointed dax_in at a new source and called * dax_lex(). If so, then we have to assure * consistency between YY_CURRENT_BUFFER and our * globals. Here is the right place to do so, because * this is the first action (other than possibly a * back-up) that will match for the new input source. */ (yy_n_chars) = YY_CURRENT_BUFFER_LVALUE->yy_n_chars; YY_CURRENT_BUFFER_LVALUE->yy_input_file = dax_in; YY_CURRENT_BUFFER_LVALUE->yy_buffer_status = YY_BUFFER_NORMAL; } /* Note that here we test for yy_c_buf_p "<=" to the position * of the first EOB in the buffer, since yy_c_buf_p will * already have been incremented past the NUL character * (since all states make transitions on EOB to the * end-of-buffer state). Contrast this with the test * in input(). */ if ( (yy_c_buf_p) <= &YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[(yy_n_chars)] ) { /* This was really a NUL. */ yy_state_type yy_next_state; (yy_c_buf_p) = (yytext_ptr) + yy_amount_of_matched_text; yy_current_state = yy_get_previous_state( ); /* Okay, we're now positioned to make the NUL * transition. We couldn't have * yy_get_previous_state() go ahead and do it * for us because it doesn't know how to deal * with the possibility of jamming (and we don't * want to build jamming into it because then it * will run more slowly). */ yy_next_state = yy_try_NUL_trans( yy_current_state ); yy_bp = (yytext_ptr) + YY_MORE_ADJ; if ( yy_next_state ) { /* Consume the NUL. */ yy_cp = ++(yy_c_buf_p); yy_current_state = yy_next_state; goto yy_match; } else { yy_cp = (yy_c_buf_p); goto yy_find_action; } } else switch ( yy_get_next_buffer( ) ) { case EOB_ACT_END_OF_FILE: { (yy_did_buffer_switch_on_eof) = 0; if ( dax_wrap( ) ) { /* Note: because we've taken care in * yy_get_next_buffer() to have set up * dax_text, we can now set up * yy_c_buf_p so that if some total * hoser (like flex itself) wants to * call the scanner after we return the * YY_NULL, it'll still work - another * YY_NULL will get returned. */ (yy_c_buf_p) = (yytext_ptr) + YY_MORE_ADJ; yy_act = YY_STATE_EOF(YY_START); goto do_action; } else { if ( ! (yy_did_buffer_switch_on_eof) ) YY_NEW_FILE; } break; } case EOB_ACT_CONTINUE_SCAN: (yy_c_buf_p) = (yytext_ptr) + yy_amount_of_matched_text; yy_current_state = yy_get_previous_state( ); yy_cp = (yy_c_buf_p); yy_bp = (yytext_ptr) + YY_MORE_ADJ; goto yy_match; case EOB_ACT_LAST_MATCH: (yy_c_buf_p) = &YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[(yy_n_chars)]; yy_current_state = yy_get_previous_state( ); yy_cp = (yy_c_buf_p); yy_bp = (yytext_ptr) + YY_MORE_ADJ; goto yy_find_action; } break; } default: YY_FATAL_ERROR( "fatal flex scanner internal error--no action found" ); } /* end of action switch */ } /* end of scanning one token */ } /* end of user's declarations */ } /* end of dax_lex */ /* yy_get_next_buffer - try to read in a new buffer * * Returns a code representing an action: * EOB_ACT_LAST_MATCH - * EOB_ACT_CONTINUE_SCAN - continue scanning from current position * EOB_ACT_END_OF_FILE - end of file */ static int yy_get_next_buffer (void) { char *dest = YY_CURRENT_BUFFER_LVALUE->yy_ch_buf; char *source = (yytext_ptr); yy_size_t number_to_move, i; int ret_val; if ( (yy_c_buf_p) > &YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[(yy_n_chars) + 1] ) YY_FATAL_ERROR( "fatal flex scanner internal error--end of buffer missed" ); if ( YY_CURRENT_BUFFER_LVALUE->yy_fill_buffer == 0 ) { /* Don't try to fill the buffer, so this is an EOF. */ if ( (yy_c_buf_p) - (yytext_ptr) - YY_MORE_ADJ == 1 ) { /* We matched a single character, the EOB, so * treat this as a final EOF. */ return EOB_ACT_END_OF_FILE; } else { /* We matched some text prior to the EOB, first * process it. */ return EOB_ACT_LAST_MATCH; } } /* Try to read more data. */ /* First move last chars to start of buffer. */ number_to_move = (yy_size_t) ((yy_c_buf_p) - (yytext_ptr)) - 1; for ( i = 0; i < number_to_move; ++i ) *(dest++) = *(source++); if ( YY_CURRENT_BUFFER_LVALUE->yy_buffer_status == YY_BUFFER_EOF_PENDING ) /* don't do the read, it's not guaranteed to return an EOF, * just force an EOF */ YY_CURRENT_BUFFER_LVALUE->yy_n_chars = (yy_n_chars) = 0; else { yy_size_t num_to_read = YY_CURRENT_BUFFER_LVALUE->yy_buf_size - number_to_move - 1; while ( num_to_read <= 0 ) { /* Not enough room in the buffer - grow it. */ /* just a shorter name for the current buffer */ YY_BUFFER_STATE b = YY_CURRENT_BUFFER_LVALUE; int yy_c_buf_p_offset = (int) ((yy_c_buf_p) - b->yy_ch_buf); if ( b->yy_is_our_buffer ) { yy_size_t new_size = b->yy_buf_size * 2; if ( new_size <= 0 ) b->yy_buf_size += b->yy_buf_size / 8; else b->yy_buf_size *= 2; b->yy_ch_buf = (char *) /* Include room in for 2 EOB chars. */ dax_realloc((void *) b->yy_ch_buf,b->yy_buf_size + 2 ); } else /* Can't grow it, we don't own it. */ b->yy_ch_buf = 0; if ( ! b->yy_ch_buf ) YY_FATAL_ERROR( "fatal error - scanner input buffer overflow" ); (yy_c_buf_p) = &b->yy_ch_buf[yy_c_buf_p_offset]; num_to_read = YY_CURRENT_BUFFER_LVALUE->yy_buf_size - number_to_move - 1; } if ( num_to_read > YY_READ_BUF_SIZE ) num_to_read = YY_READ_BUF_SIZE; /* Read in more data. */ YY_INPUT( (&YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[number_to_move]), (yy_n_chars), num_to_read ); YY_CURRENT_BUFFER_LVALUE->yy_n_chars = (yy_n_chars); } if ( (yy_n_chars) == 0 ) { if ( number_to_move == YY_MORE_ADJ ) { ret_val = EOB_ACT_END_OF_FILE; dax_restart(dax_in ); } else { ret_val = EOB_ACT_LAST_MATCH; YY_CURRENT_BUFFER_LVALUE->yy_buffer_status = YY_BUFFER_EOF_PENDING; } } else ret_val = EOB_ACT_CONTINUE_SCAN; if ((unsigned int) ((yy_n_chars) + number_to_move) > YY_CURRENT_BUFFER_LVALUE->yy_buf_size) { /* Extend the array by 50%, plus the number we really need. */ int new_size = (yy_n_chars) + number_to_move + ((yy_n_chars) >> 1); YY_CURRENT_BUFFER_LVALUE->yy_ch_buf = (char *) dax_realloc((void *) YY_CURRENT_BUFFER_LVALUE->yy_ch_buf,new_size ); if ( ! YY_CURRENT_BUFFER_LVALUE->yy_ch_buf ) YY_FATAL_ERROR( "out of dynamic memory in yy_get_next_buffer()" ); } (yy_n_chars) += number_to_move; YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[(yy_n_chars)] = YY_END_OF_BUFFER_CHAR; YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[(yy_n_chars) + 1] = YY_END_OF_BUFFER_CHAR; (yytext_ptr) = &YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[0]; return ret_val; } /* yy_get_previous_state - get the state just before the EOB char was reached */ static yy_state_type yy_get_previous_state (void) { yy_state_type yy_current_state; char *yy_cp; yy_current_state = (yy_start); for ( yy_cp = (yytext_ptr) + YY_MORE_ADJ; yy_cp < (yy_c_buf_p); ++yy_cp ) { YY_CHAR yy_c = (*yy_cp ? yy_ec[YY_SC_TO_UI(*yy_cp)] : 1); if ( yy_accept[yy_current_state] ) { (yy_last_accepting_state) = yy_current_state; (yy_last_accepting_cpos) = yy_cp; } while ( yy_chk[yy_base[yy_current_state] + yy_c] != yy_current_state ) { yy_current_state = (int) yy_def[yy_current_state]; if ( yy_current_state >= 775 ) yy_c = yy_meta[(unsigned int) yy_c]; } yy_current_state = yy_nxt[yy_base[yy_current_state] + (unsigned int) yy_c]; } return yy_current_state; } /* yy_try_NUL_trans - try to make a transition on the NUL character * * synopsis * next_state = yy_try_NUL_trans( current_state ); */ static yy_state_type yy_try_NUL_trans (yy_state_type yy_current_state ) { int yy_is_jam; char *yy_cp = (yy_c_buf_p); YY_CHAR yy_c = 1; if ( yy_accept[yy_current_state] ) { (yy_last_accepting_state) = yy_current_state; (yy_last_accepting_cpos) = yy_cp; } while ( yy_chk[yy_base[yy_current_state] + yy_c] != yy_current_state ) { yy_current_state = (int) yy_def[yy_current_state]; if ( yy_current_state >= 775 ) yy_c = yy_meta[(unsigned int) yy_c]; } yy_current_state = yy_nxt[yy_base[yy_current_state] + (unsigned int) yy_c]; yy_is_jam = (yy_current_state == 774); return yy_is_jam ? 0 : yy_current_state; } #ifndef YY_NO_UNPUT #endif #ifndef YY_NO_INPUT #ifdef __cplusplus static int yyinput (void) #else static int input (void) #endif { int c; *(yy_c_buf_p) = (yy_hold_char); if ( *(yy_c_buf_p) == YY_END_OF_BUFFER_CHAR ) { /* yy_c_buf_p now points to the character we want to return. * If this occurs *before* the EOB characters, then it's a * valid NUL; if not, then we've hit the end of the buffer. */ if ( (yy_c_buf_p) < &YY_CURRENT_BUFFER_LVALUE->yy_ch_buf[(yy_n_chars)] ) /* This was really a NUL. */ *(yy_c_buf_p) = '\0'; else { /* need more input */ yy_size_t offset = (yy_c_buf_p) - (yytext_ptr); ++(yy_c_buf_p); switch ( yy_get_next_buffer( ) ) { case EOB_ACT_LAST_MATCH: /* This happens because yy_g_n_b() * sees that we've accumulated a * token and flags that we need to * try matching the token before * proceeding. But for input(), * there's no matching to consider. * So convert the EOB_ACT_LAST_MATCH * to EOB_ACT_END_OF_FILE. */ /* Reset buffer status. */ dax_restart(dax_in ); /*FALLTHROUGH*/ case EOB_ACT_END_OF_FILE: { if ( dax_wrap( ) ) return EOF; if ( ! (yy_did_buffer_switch_on_eof) ) YY_NEW_FILE; #ifdef __cplusplus return yyinput(); #else return input(); #endif } case EOB_ACT_CONTINUE_SCAN: (yy_c_buf_p) = (yytext_ptr) + offset; break; } } } c = *(unsigned char *) (yy_c_buf_p); /* cast for 8-bit char's */ *(yy_c_buf_p) = '\0'; /* preserve dax_text */ (yy_hold_char) = *++(yy_c_buf_p); if ( c == '\n' ) dax_lineno++; ; return c; } #endif /* ifndef YY_NO_INPUT */ /** Immediately switch to a different input stream. * @param input_file A readable stream. * * @note This function does not reset the start condition to @c INITIAL . */ void dax_restart (FILE * input_file ) { if ( ! YY_CURRENT_BUFFER ){ dax_ensure_buffer_stack (); YY_CURRENT_BUFFER_LVALUE = dax__create_buffer(dax_in,YY_BUF_SIZE ); } dax__init_buffer(YY_CURRENT_BUFFER,input_file ); dax__load_buffer_state( ); } /** Switch to a different input buffer. * @param new_buffer The new input buffer. * */ void dax__switch_to_buffer (YY_BUFFER_STATE new_buffer ) { /* TODO. We should be able to replace this entire function body * with * dax_pop_buffer_state(); * dax_push_buffer_state(new_buffer); */ dax_ensure_buffer_stack (); if ( YY_CURRENT_BUFFER == new_buffer ) return; if ( YY_CURRENT_BUFFER ) { /* Flush out information for old buffer. */ *(yy_c_buf_p) = (yy_hold_char); YY_CURRENT_BUFFER_LVALUE->yy_buf_pos = (yy_c_buf_p); YY_CURRENT_BUFFER_LVALUE->yy_n_chars = (yy_n_chars); } YY_CURRENT_BUFFER_LVALUE = new_buffer; dax__load_buffer_state( ); /* We don't actually know whether we did this switch during * EOF (dax_wrap()) processing, but the only time this flag * is looked at is after dax_wrap() is called, so it's safe * to go ahead and always set it. */ (yy_did_buffer_switch_on_eof) = 1; } static void dax__load_buffer_state (void) { (yy_n_chars) = YY_CURRENT_BUFFER_LVALUE->yy_n_chars; (yytext_ptr) = (yy_c_buf_p) = YY_CURRENT_BUFFER_LVALUE->yy_buf_pos; dax_in = YY_CURRENT_BUFFER_LVALUE->yy_input_file; (yy_hold_char) = *(yy_c_buf_p); } /** Allocate and initialize an input buffer state. * @param file A readable stream. * @param size The character buffer size in bytes. When in doubt, use @c YY_BUF_SIZE. * * @return the allocated buffer state. */ YY_BUFFER_STATE dax__create_buffer (FILE * file, int size ) { YY_BUFFER_STATE b; b = (YY_BUFFER_STATE) dax_alloc(sizeof( struct yy_buffer_state ) ); if ( ! b ) YY_FATAL_ERROR( "out of dynamic memory in dax__create_buffer()" ); b->yy_buf_size = (yy_size_t)size; /* yy_ch_buf has to be 2 characters longer than the size given because * we need to put in 2 end-of-buffer characters. */ b->yy_ch_buf = (char *) dax_alloc(b->yy_buf_size + 2 ); if ( ! b->yy_ch_buf ) YY_FATAL_ERROR( "out of dynamic memory in dax__create_buffer()" ); b->yy_is_our_buffer = 1; dax__init_buffer(b,file ); return b; } /** Destroy the buffer. * @param b a buffer created with dax__create_buffer() * */ void dax__delete_buffer (YY_BUFFER_STATE b ) { if ( ! b ) return; if ( b == YY_CURRENT_BUFFER ) /* Not sure if we should pop here. */ YY_CURRENT_BUFFER_LVALUE = (YY_BUFFER_STATE) 0; if ( b->yy_is_our_buffer ) dax_free((void *) b->yy_ch_buf ); dax_free((void *) b ); } /* Initializes or reinitializes a buffer. * This function is sometimes called more than once on the same buffer, * such as during a dax_restart() or at EOF. */ static void dax__init_buffer (YY_BUFFER_STATE b, FILE * file ) { int oerrno = errno; dax__flush_buffer(b ); b->yy_input_file = file; b->yy_fill_buffer = 1; /* If b is the current buffer, then dax__init_buffer was _probably_ * called from dax_restart() or through yy_get_next_buffer. * In that case, we don't want to reset the lineno or column. */ if (b != YY_CURRENT_BUFFER){ b->yy_bs_lineno = 1; b->yy_bs_column = 0; } b->yy_is_interactive = file ? (isatty( fileno(file) ) > 0) : 0; errno = oerrno; } /** Discard all buffered characters. On the next scan, YY_INPUT will be called. * @param b the buffer state to be flushed, usually @c YY_CURRENT_BUFFER. * */ void dax__flush_buffer (YY_BUFFER_STATE b ) { if ( ! b ) return; b->yy_n_chars = 0; /* We always need two end-of-buffer characters. The first causes * a transition to the end-of-buffer state. The second causes * a jam in that state. */ b->yy_ch_buf[0] = YY_END_OF_BUFFER_CHAR; b->yy_ch_buf[1] = YY_END_OF_BUFFER_CHAR; b->yy_buf_pos = &b->yy_ch_buf[0]; b->yy_at_bol = 1; b->yy_buffer_status = YY_BUFFER_NEW; if ( b == YY_CURRENT_BUFFER ) dax__load_buffer_state( ); } /** Pushes the new state onto the stack. The new state becomes * the current state. This function will allocate the stack * if necessary. * @param new_buffer The new state. * */ void dax_push_buffer_state (YY_BUFFER_STATE new_buffer ) { if (new_buffer == NULL) return; dax_ensure_buffer_stack(); /* This block is copied from dax__switch_to_buffer. */ if ( YY_CURRENT_BUFFER ) { /* Flush out information for old buffer. */ *(yy_c_buf_p) = (yy_hold_char); YY_CURRENT_BUFFER_LVALUE->yy_buf_pos = (yy_c_buf_p); YY_CURRENT_BUFFER_LVALUE->yy_n_chars = (yy_n_chars); } /* Only push if top exists. Otherwise, replace top. */ if (YY_CURRENT_BUFFER) (yy_buffer_stack_top)++; YY_CURRENT_BUFFER_LVALUE = new_buffer; /* copied from dax__switch_to_buffer. */ dax__load_buffer_state( ); (yy_did_buffer_switch_on_eof) = 1; } /** Removes and deletes the top of the stack, if present. * The next element becomes the new top. * */ void dax_pop_buffer_state (void) { if (!YY_CURRENT_BUFFER) return; dax__delete_buffer(YY_CURRENT_BUFFER ); YY_CURRENT_BUFFER_LVALUE = NULL; if ((yy_buffer_stack_top) > 0) --(yy_buffer_stack_top); if (YY_CURRENT_BUFFER) { dax__load_buffer_state( ); (yy_did_buffer_switch_on_eof) = 1; } } /* Allocates the stack if it does not exist. * Guarantees space for at least one push. */ static void dax_ensure_buffer_stack (void) { yy_size_t num_to_alloc; if (!(yy_buffer_stack)) { /* First allocation is just for 2 elements, since we don't know if this * scanner will even need a stack. We use 2 instead of 1 to avoid an * immediate realloc on the next call. */ num_to_alloc = 1; /* After all that talk, this was set to 1 anyways... */ (yy_buffer_stack) = (struct yy_buffer_state**)dax_alloc (num_to_alloc * sizeof(struct yy_buffer_state*) ); if ( ! (yy_buffer_stack) ) YY_FATAL_ERROR( "out of dynamic memory in dax_ensure_buffer_stack()" ); memset((yy_buffer_stack), 0, num_to_alloc * sizeof(struct yy_buffer_state*)); (yy_buffer_stack_max) = num_to_alloc; (yy_buffer_stack_top) = 0; return; } if ((yy_buffer_stack_top) >= ((yy_buffer_stack_max)) - 1){ /* Increase the buffer to prepare for a possible push. */ yy_size_t grow_size = 8 /* arbitrary grow size */; num_to_alloc = (yy_buffer_stack_max) + grow_size; (yy_buffer_stack) = (struct yy_buffer_state**)dax_realloc ((yy_buffer_stack), num_to_alloc * sizeof(struct yy_buffer_state*) ); if ( ! (yy_buffer_stack) ) YY_FATAL_ERROR( "out of dynamic memory in dax_ensure_buffer_stack()" ); /* zero only the new slots.*/ memset((yy_buffer_stack) + (yy_buffer_stack_max), 0, grow_size * sizeof(struct yy_buffer_state*)); (yy_buffer_stack_max) = num_to_alloc; } } /** Setup the input buffer state to scan directly from a user-specified character buffer. * @param base the character buffer * @param size the size in bytes of the character buffer * * @return the newly allocated buffer state object. */ YY_BUFFER_STATE dax__scan_buffer (char * base, yy_size_t size ) { YY_BUFFER_STATE b; if ( size < 2 || base[size-2] != YY_END_OF_BUFFER_CHAR || base[size-1] != YY_END_OF_BUFFER_CHAR ) /* They forgot to leave room for the EOB's. */ return 0; b = (YY_BUFFER_STATE) dax_alloc(sizeof( struct yy_buffer_state ) ); if ( ! b ) YY_FATAL_ERROR( "out of dynamic memory in dax__scan_buffer()" ); b->yy_buf_size = size - 2; /* "- 2" to take care of EOB's */ b->yy_buf_pos = b->yy_ch_buf = base; b->yy_is_our_buffer = 0; b->yy_input_file = 0; b->yy_n_chars = b->yy_buf_size; b->yy_is_interactive = 0; b->yy_at_bol = 1; b->yy_fill_buffer = 0; b->yy_buffer_status = YY_BUFFER_NEW; dax__switch_to_buffer(b ); return b; } /** Setup the input buffer state to scan a string. The next call to dax_lex() will * scan from a @e copy of @a str. * @param yystr a NUL-terminated string to scan * * @return the newly allocated buffer state object. * @note If you want to scan bytes that may contain NUL values, then use * dax__scan_bytes() instead. */ YY_BUFFER_STATE dax__scan_string (yyconst char * yystr ) { return dax__scan_bytes(yystr,strlen(yystr) ); } /** Setup the input buffer state to scan the given bytes. The next call to dax_lex() will * scan from a @e copy of @a bytes. * @param yybytes the byte buffer to scan * @param _yybytes_len the number of bytes in the buffer pointed to by @a bytes. * * @return the newly allocated buffer state object. */ YY_BUFFER_STATE dax__scan_bytes (yyconst char * yybytes, yy_size_t _yybytes_len ) { YY_BUFFER_STATE b; char *buf; yy_size_t n; yy_size_t i; /* Get memory for full buffer, including space for trailing EOB's. */ n = _yybytes_len + 2; buf = (char *) dax_alloc(n ); if ( ! buf ) YY_FATAL_ERROR( "out of dynamic memory in dax__scan_bytes()" ); for ( i = 0; i < _yybytes_len; ++i ) buf[i] = yybytes[i]; buf[_yybytes_len] = buf[_yybytes_len+1] = YY_END_OF_BUFFER_CHAR; b = dax__scan_buffer(buf,n ); if ( ! b ) YY_FATAL_ERROR( "bad buffer in dax__scan_bytes()" ); /* It's okay to grow etc. this buffer, and we should throw it * away when we're done. */ b->yy_is_our_buffer = 1; return b; } static void yy_push_state (int _new_state ) { if ( (yy_start_stack_ptr) >= (yy_start_stack_depth) ) { yy_size_t new_size; (yy_start_stack_depth) += YY_START_STACK_INCR; new_size = (yy_start_stack_depth) * sizeof( int ); if ( ! (yy_start_stack) ) (yy_start_stack) = (int *) dax_alloc(new_size ); else (yy_start_stack) = (int *) dax_realloc((void *) (yy_start_stack),new_size ); if ( ! (yy_start_stack) ) YY_FATAL_ERROR( "out of memory expanding start-condition stack" ); } (yy_start_stack)[(yy_start_stack_ptr)++] = YY_START; BEGIN(_new_state); } static void yy_pop_state (void) { if ( --(yy_start_stack_ptr) < 0 ) YY_FATAL_ERROR( "start-condition stack underflow" ); BEGIN((yy_start_stack)[(yy_start_stack_ptr)]); } #ifndef YY_EXIT_FAILURE #define YY_EXIT_FAILURE 2 #endif static void yy_fatal_error (yyconst char* msg ) { (void) fprintf( stderr, "%s\n", msg ); exit( YY_EXIT_FAILURE ); } /* Redefine yyless() so it works in section 3 code. */ #undef yyless #define yyless(n) \ do \ { \ /* Undo effects of setting up dax_text. */ \ int yyless_macro_arg = (n); \ YY_LESS_LINENO(yyless_macro_arg);\ dax_text[dax_leng] = (yy_hold_char); \ (yy_c_buf_p) = dax_text + yyless_macro_arg; \ (yy_hold_char) = *(yy_c_buf_p); \ *(yy_c_buf_p) = '\0'; \ dax_leng = yyless_macro_arg; \ } \ while ( 0 ) /* Accessor methods (get/set functions) to struct members. */ /** Get the current line number. * */ int dax_get_lineno (void) { return dax_lineno; } /** Get the input stream. * */ FILE *dax_get_in (void) { return dax_in; } /** Get the output stream. * */ FILE *dax_get_out (void) { return dax_out; } /** Get the length of the current token. * */ yy_size_t dax_get_leng (void) { return dax_leng; } /** Get the current token. * */ char *dax_get_text (void) { return dax_text; } /** Set the current line number. * @param _line_number line number * */ void dax_set_lineno (int _line_number ) { dax_lineno = _line_number; } /** Set the input stream. This does not discard the current * input buffer. * @param _in_str A readable stream. * * @see dax__switch_to_buffer */ void dax_set_in (FILE * _in_str ) { dax_in = _in_str ; } void dax_set_out (FILE * _out_str ) { dax_out = _out_str ; } int dax_get_debug (void) { return dax__flex_debug; } void dax_set_debug (int _bdebug ) { dax__flex_debug = _bdebug ; } static int yy_init_globals (void) { /* Initialization is the same as for the non-reentrant scanner. * This function is called from dax_lex_destroy(), so don't allocate here. */ /* We do not touch dax_lineno unless the option is enabled. */ dax_lineno = 1; (yy_buffer_stack) = 0; (yy_buffer_stack_top) = 0; (yy_buffer_stack_max) = 0; (yy_c_buf_p) = (char *) 0; (yy_init) = 0; (yy_start) = 0; (yy_start_stack_ptr) = 0; (yy_start_stack_depth) = 0; (yy_start_stack) = NULL; /* Defined in main.c */ #ifdef YY_STDINIT dax_in = stdin; dax_out = stdout; #else dax_in = (FILE *) 0; dax_out = (FILE *) 0; #endif /* For future reference: Set errno on error, since we are called by * dax_lex_init() */ return 0; } /* dax_lex_destroy is for both reentrant and non-reentrant scanners. */ int dax_lex_destroy (void) { /* Pop the buffer stack, destroying each element. */ while(YY_CURRENT_BUFFER){ dax__delete_buffer(YY_CURRENT_BUFFER ); YY_CURRENT_BUFFER_LVALUE = NULL; dax_pop_buffer_state(); } /* Destroy the stack itself. */ dax_free((yy_buffer_stack) ); (yy_buffer_stack) = NULL; /* Destroy the start condition stack. */ dax_free((yy_start_stack) ); (yy_start_stack) = NULL; /* Reset the globals. This is important in a non-reentrant scanner so the next time * dax_lex() is called, initialization will occur. */ yy_init_globals( ); return 0; } /* * Internal utility routines. */ #ifndef yytext_ptr static void yy_flex_strncpy (char* s1, yyconst char * s2, int n ) { int i; for ( i = 0; i < n; ++i ) s1[i] = s2[i]; } #endif #ifdef YY_NEED_STRLEN static int yy_flex_strlen (yyconst char * s ) { int n; for ( n = 0; s[n]; ++n ) ; return n; } #endif void *dax_alloc (yy_size_t size ) { return (void *) malloc( size ); } void *dax_realloc (void * ptr, yy_size_t size ) { /* The cast to (char *) in the following accommodates both * implementations that use char* generic pointers, and those * that use void* generic pointers. It works with the latter * because both ANSI C and C++ allow castless assignment from * any pointer type to void*, and deal with argument conversions * as though doing an assignment. */ return (void *) realloc( (char *) ptr, size ); } void dax_free (void * ptr ) { free( (char *) ptr ); /* see dax_realloc() for (char *) cast */ } #define YYTABLES_NAME "yytables" /* Element context stack lookup. */ int dax__element_context(int i) { return (0(); runnable_tasks = new std::set(); completed_tasks = new std::set(); return_set = new std::set(); } Global::~Global(){ delete initial_tasks; delete runnable_tasks; delete completed_tasks; delete return_set; } std::set* simulate(double how_long){ XBT_VERB("Run simulation for %f seconds", how_long); sd_global->watch_point_reached = false; sd_global->return_set->clear(); /* explore the runnable tasks */ while(!sd_global->runnable_tasks->empty()) SD_task_run(*(sd_global->runnable_tasks->begin())); double elapsed_time = 0.0; double total_time = 0.0; /* main loop */ while (elapsed_time >= 0 && (how_long < 0 || 0.00001 < (how_long -total_time)) && !sd_global->watch_point_reached) { XBT_DEBUG("Total time: %f", total_time); elapsed_time = surf_solve(how_long > 0 ? surf_get_clock() + how_long - total_time: -1.0); XBT_DEBUG("surf_solve() returns %f", elapsed_time); if (elapsed_time > 0.0) total_time += elapsed_time; /* let's see which tasks are done */ for (auto model : *all_existing_models) { surf_action_t action = surf_model_extract_done_action_set(model); while (action != nullptr) { SD_task_t task = static_cast(action->getData()); XBT_VERB("Task '%s' done", SD_task_get_name(task)); SD_task_set_state(task, SD_DONE); /* the state has changed. Add it only if it's the first change */ if (sd_global->return_set->find(task) == sd_global->return_set->end()) sd_global->return_set->insert(task); /* remove the dependencies after this task */ for (auto succ : *task->successors) { succ->predecessors->erase(task); succ->inputs->erase(task); XBT_DEBUG("Release dependency on %s: %zu remain(s). Becomes schedulable if %zu=0", SD_task_get_name(succ), succ->predecessors->size()+succ->inputs->size(), succ->predecessors->size()); if (SD_task_get_state(succ) == SD_NOT_SCHEDULED && succ->predecessors->empty()) SD_task_set_state(succ, SD_SCHEDULABLE); if (SD_task_get_state(succ) == SD_SCHEDULED && succ->predecessors->empty() && succ->inputs->empty()) SD_task_set_state(succ, SD_RUNNABLE); if (SD_task_get_state(succ) == SD_RUNNABLE && !sd_global->watch_point_reached) SD_task_run(succ); } task->successors->clear(); for (auto output : *task->outputs) { output->start_time = task->finish_time; output->predecessors->erase(task); if (SD_task_get_state(output) == SD_SCHEDULED) SD_task_set_state(output, SD_RUNNABLE); else SD_task_set_state(output, SD_SCHEDULABLE); SD_task_t comm_dst = *(output->successors->begin()); if (SD_task_get_state(comm_dst) == SD_NOT_SCHEDULED && comm_dst->predecessors->empty()){ XBT_DEBUG("%s is a transfer, %s may be ready now if %zu=0", SD_task_get_name(output), SD_task_get_name(comm_dst), comm_dst->predecessors->size()); SD_task_set_state(comm_dst, SD_SCHEDULABLE); } if (SD_task_get_state(output) == SD_RUNNABLE && !sd_global->watch_point_reached) SD_task_run(output); } task->outputs->clear(); action = surf_model_extract_done_action_set(model); } /* let's see which tasks have just failed */ action = surf_model_extract_failed_action_set(model); while (action != nullptr) { SD_task_t task = static_cast(action->getData()); XBT_VERB("Task '%s' failed", SD_task_get_name(task)); SD_task_set_state(task, SD_FAILED); sd_global->return_set->insert(task); action = surf_model_extract_failed_action_set(model); } } } if (!sd_global->watch_point_reached && how_long < 0 && !sd_global->initial_tasks->empty()) { XBT_WARN("Simulation is finished but %zu tasks are still not done", sd_global->initial_tasks->size()); for (auto t : *sd_global->initial_tasks) XBT_WARN("%s is in %s state", SD_task_get_name(t), __get_state_name(SD_task_get_state(t))); } XBT_DEBUG("elapsed_time = %f, total_time = %f, watch_point_reached = %d", elapsed_time, total_time, sd_global->watch_point_reached); XBT_DEBUG("current time = %f", surf_get_clock()); return sd_global->return_set; } } } /** * \brief helper for pretty printing of task state * \param state the state of a task * \return the equivalent as a readable string */ const char *__get_state_name(e_SD_task_state_t state){ static std::string state_names[7] = { "not scheduled", "schedulable", "scheduled", "runnable","running", "done", "failed" }; return state_names[static_cast(log2(static_cast(state)))].data(); } /** * \brief Initializes SD internal data * * This function must be called before any other SD function. Then you should call SD_create_environment(). * * \param argc argument number * \param argv argument list * \see SD_create_environment(), SD_exit() */ void SD_init(int *argc, char **argv) { xbt_assert(sd_global == nullptr, "SD_init() already called"); sd_global = new simgrid::sd::Global(); surf_init(argc, argv); xbt_cfg_setdefault_string("host/model", "ptask_L07"); if (_sg_cfg_exit_asap) { SD_exit(); exit(0); } } /** \brief set a configuration variable * * Do --help on any simgrid binary to see the list of currently existing configuration variables, and * see Section @ref options. * * Example: SD_config("host/model","default"); */ void SD_config(const char *key, const char *value){ xbt_assert(sd_global,"ERROR: Please call SD_init() before using SD_config()"); xbt_cfg_set_as_string(key, value); } /** * \brief Creates the environment * * The environment (i.e. the \ref SD_host_api "hosts" and the \ref SD_link_api "links") is created with * the data stored in the given XML platform file. * * \param platform_file name of an XML file describing the environment to create * \see SD_host_api, SD_link_api * * The XML file follows this DTD: * * \include simgrid.dtd * * Here is a small example of such a platform: * * \include small_platform.xml */ void SD_create_environment(const char *platform_file) { simgrid::s4u::Engine::instance()->loadPlatform(platform_file); XBT_DEBUG("Host number: %zu, link number: %d", sg_host_count(), sg_link_count()); #if HAVE_JEDULE jedule_sd_init(); #endif XBT_VERB("Starting simulation..."); surf_presolve(); /* Takes traces into account */ } /** * \brief Launches the simulation. * * The function will execute the \ref SD_RUNNABLE runnable tasks. * If \a how_long is positive, then the simulation will be stopped either when time reaches \a how_long or when a watch * point is reached. * A non-positive value for \a how_long means no time limit, in which case the simulation will be stopped either when a * watch point is reached or when no more task can be executed. * Then you can call SD_simulate() again. * * \param how_long maximum duration of the simulation (a negative value means no time limit) * \return a dynar of \ref SD_task_t whose state has changed. * \see SD_task_schedule(), SD_task_watch() */ xbt_dynar_t SD_simulate(double how_long) { std::set *changed_tasks = simgrid::sd::simulate(how_long); xbt_dynar_t changed_tasks_dynar = xbt_dynar_new(sizeof(SD_task_t), nullptr); for (auto task: *changed_tasks) xbt_dynar_push(changed_tasks_dynar, &task); return changed_tasks_dynar; } /** @brief Returns the current clock, in seconds */ double SD_get_clock() { return surf_get_clock(); } /** * \brief Destroys all SD internal data * This function should be called when the simulation is over. Don't forget to destroy too. * \see SD_init(), SD_task_destroy() */ void SD_exit() { TRACE_end(); #if HAVE_JEDULE jedule_sd_exit(); #endif delete sd_global; } SimGrid-3.14.159/src/simdag/dax.dtd000644 001750 001750 00000002332 13031031273 020375 0ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/src/simdag/simdag_private.hpp000644 001750 001750 00000004067 13031031273 022642 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include "simgrid/simdag.h" #include "surf/surf.h" #ifndef SIMDAG_PRIVATE_HPP #define SIMDAG_PRIVATE_HPP #if HAVE_JEDULE #include "simgrid/jedule/jedule_sd_binding.h" #endif namespace simgrid{ namespace sd{ class Global { public: explicit Global(); ~Global(); bool watch_point_reached; /* has a task just reached a watch point? */ std::set *initial_tasks; std::set *runnable_tasks; std::set *completed_tasks; std::set *return_set; }; std::set* simulate (double how_long); } } SG_BEGIN_DECL() extern XBT_PRIVATE simgrid::sd::Global *sd_global; /* Task */ typedef struct SD_task { e_SD_task_state_t state; void *data; /* user data */ char *name; e_SD_task_kind_t kind; double amount; double alpha; /* used by typed parallel tasks */ double start_time; double finish_time; surf_action_t surf_action; unsigned short watch_points; /* bit field xor()ed with masks */ int marked; /* used to check if the task DAG has some cycle*/ /* dependencies */ std::set *inputs; std::set *outputs; std::set *predecessors; std::set *successors; /* scheduling parameters (only exist in state SD_SCHEDULED) */ std::vector *allocation; double *flops_amount; double *bytes_amount; double rate; } s_SD_task_t; /* SimDag private functions */ XBT_PRIVATE void SD_task_set_state(SD_task_t task, e_SD_task_state_t new_state); XBT_PRIVATE void SD_task_run(SD_task_t task); XBT_PRIVATE bool acyclic_graph_detail(xbt_dynar_t dag); XBT_PRIVATE void uniq_transfer_task_name(SD_task_t task); XBT_PRIVATE const char *__get_state_name(e_SD_task_state_t state); SG_END_DECL() #endif SimGrid-3.14.159/src/simdag/sd_workstation.cpp000644 001750 001750 00000004675 13031031271 022714 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/s4u/host.hpp" #include "simdag_private.hpp" #include "src/surf/HostImpl.hpp" #include "surf/surf.h" /** @brief Returns the route between two workstations * * Use SD_route_get_size() to know the array size. * * \param src a host * \param dst another host * \return an array of the \ref SD_link_t composing the route * \see SD_route_get_size(), SD_link_t */ SD_link_t *SD_route_get_list(sg_host_t src, sg_host_t dst) { std::vector route; src->routeTo(dst, &route, nullptr); int cpt=0; SD_link_t* list = xbt_new(SD_link_t, route.size()); for (auto link : route) { list[cpt] = link; cpt++; } return list; } /** * \brief Returns the number of links on the route between two workstations * * \param src a workstation * \param dst another workstation * \return the number of links on the route between these two workstations * \see SD_route_get_list() */ int SD_route_get_size(sg_host_t src, sg_host_t dst) { std::vector route; src->routeTo(dst, &route, nullptr); int size = route.size(); return size; } /** * \brief Returns the latency of the route between two workstations. * * \param src the first workstation * \param dst the second workstation * \return the latency of the route between the two workstations (in seconds) * \see SD_route_get_bandwidth() */ double SD_route_get_latency(sg_host_t src, sg_host_t dst) { double latency = 0; std::vector route; src->routeTo(dst, &route, &latency); return latency; } /** * \brief Returns the bandwidth of the route between two workstations, * i.e. the minimum link bandwidth of all between the workstations. * * \param src the first workstation * \param dst the second workstation * \return the bandwidth of the route between the two workstations (in bytes/second) * \see SD_route_get_latency() */ double SD_route_get_bandwidth(sg_host_t src, sg_host_t dst) { double min_bandwidth = -1.0; std::vector route; src->routeTo(dst, &route, nullptr); for (auto link : route) { double bandwidth = sg_link_bandwidth(link); if (bandwidth < min_bandwidth || min_bandwidth < 0.0) min_bandwidth = bandwidth; } return min_bandwidth; } SimGrid-3.14.159/src/simdag/sd_daxloader.cpp000644 001750 001750 00000025774 13031031271 022276 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2009-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/simdag.h" #include "xbt/misc.h" #include "xbt/log.h" #include "xbt/str.h" #include "xbt/file.h" /* xbt_basename() */ #include "simdag_private.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(sd_daxparse, sd, "Parsing DAX files"); extern "C" { #undef CLEANUP #include "dax_dtd.h" #define register /* g++ don't like register, so don't say it */ #include "dax_dtd.c" #undef register } /* Ensure that transfer tasks have unique names even though a file is used several times */ void uniq_transfer_task_name(SD_task_t task) { SD_task_t child = *(task->successors->begin()); SD_task_t parent = *(task->predecessors->begin()); char *new_name = bprintf("%s_%s_%s", SD_task_get_name(parent), SD_task_get_name(task), SD_task_get_name(child)); SD_task_set_name(task, new_name); free(new_name); } static bool children_are_marked(SD_task_t task){ for (SD_task_t it : *task->successors) if (it->marked == 0) return false; for (SD_task_t it : *task->outputs) if (it->marked == 0) return false; return true; } static bool parents_are_marked(SD_task_t task){ for (SD_task_t it : *task->predecessors) if (it->marked == 0) return false; for (SD_task_t it : *task->inputs) if (it->marked == 0) return false; return true; } bool acyclic_graph_detail(xbt_dynar_t dag){ unsigned int count; bool all_marked = true; SD_task_t task = nullptr; std::vector current; xbt_dynar_foreach(dag,count,task){ if(task->kind != SD_TASK_COMM_E2E){ task->marked = 0; if(task->successors->empty() && task->outputs->empty()) current.push_back(task); } } while(!current.empty()){ std::vector next; for (auto t: current){ //Mark task t->marked = 1; for (SD_task_t input : *t->inputs){ input->marked=1; // Inputs are communication, hence they can have only one predecessor SD_task_t input_pred = *(input->predecessors->begin()); if (children_are_marked(input_pred)) next.push_back(input_pred); } for (SD_task_t pred : *t->predecessors) { if (children_are_marked(pred)) next.push_back(pred); } } current.clear(); current = next; } all_marked = true; //test if all tasks are marked xbt_dynar_foreach(dag,count,task){ if(task->kind != SD_TASK_COMM_E2E && task->marked == 0){ XBT_WARN("the task %s is not marked",task->name); all_marked = false; break; } } if(!all_marked){ XBT_VERB("there is at least one cycle in your task graph"); xbt_dynar_foreach(dag,count,task){ if(task->kind != SD_TASK_COMM_E2E && task->predecessors->empty() && task->inputs->empty()){ task->marked = 1; current.push_back(task); } } //test if something has to be done for the next iteration while(!current.empty()){ std::vector next; //test if the current iteration is done for (auto t: current){ t->marked = 1; for (SD_task_t output : *t->outputs) { output->marked = 1; // outputs are communication, hence they can have only one successor SD_task_t output_succ = *(output->successors->begin()); if (parents_are_marked(output_succ)) next.push_back(output_succ); } for (SD_task_t succ : *t->successors) { if (parents_are_marked(succ)) next.push_back(succ); } current.clear(); current = next; } } all_marked = true; xbt_dynar_foreach(dag,count,task){ if(task->kind != SD_TASK_COMM_E2E && task->marked == 0){ XBT_WARN("the task %s is in a cycle",task->name); all_marked = false; } } } return all_marked; } static YY_BUFFER_STATE input_buffer; static xbt_dynar_t result; static xbt_dict_t jobs; static xbt_dict_t files; static SD_task_t current_job; static SD_task_t root_task; static SD_task_t end_task; static void dax_task_free(void *task) { SD_task_destroy(static_cast(task)); } /** @brief loads a DAX file describing a DAG * * See https://confluence.pegasus.isi.edu/display/pegasus/WorkflowGenerator for more details. */ xbt_dynar_t SD_daxload(const char *filename) { xbt_dict_cursor_t cursor; SD_task_t file; char *name; FILE *in_file = fopen(filename, "r"); xbt_assert(in_file, "Unable to open \"%s\"\n", filename); input_buffer = dax__create_buffer(in_file, 10); dax__switch_to_buffer(input_buffer); dax_lineno = 1; result = xbt_dynar_new(sizeof(SD_task_t), dax_task_free); files = xbt_dict_new_homogeneous(&dax_task_free); jobs = xbt_dict_new_homogeneous(nullptr); root_task = SD_task_create_comp_seq("root", nullptr, 0); /* by design the root task is always SCHEDULABLE */ SD_task_set_state(root_task, SD_SCHEDULABLE); xbt_dynar_push(result, &root_task); end_task = SD_task_create_comp_seq("end", nullptr, 0); int res = dax_lex(); if (res != 0) xbt_die("Parse error in %s: %s", filename, dax__parse_err_msg()); dax__delete_buffer(input_buffer); fclose(in_file); dax_lex_destroy(); xbt_dict_free(&jobs); /* And now, post-process the files. * We want a file task per pair of computation tasks exchanging the file. Duplicate on need * Files not produced in the system are said to be produced by root task (top of DAG). * Files not consumed in the system are said to be consumed by end task (bottom of DAG). */ xbt_dict_foreach(files, cursor, name, file) { SD_task_t newfile; if (file->predecessors->empty()) { for (SD_task_t it : *file->successors) { newfile = SD_task_create_comm_e2e(file->name, nullptr, file->amount); SD_task_dependency_add(nullptr, nullptr, root_task, newfile); SD_task_dependency_add(nullptr, nullptr, newfile, it); xbt_dynar_push(result, &newfile); } } else if (file->successors->empty()) { for (SD_task_t it : *file->predecessors){ newfile = SD_task_create_comm_e2e(file->name, nullptr, file->amount); SD_task_dependency_add(nullptr, nullptr, it, newfile); SD_task_dependency_add(nullptr, nullptr, newfile, end_task); xbt_dynar_push(result, &newfile); } } else { for (SD_task_t it : *file->predecessors) { for (SD_task_t it2 : *file->successors) { if (it == it2) { XBT_WARN ("File %s is produced and consumed by task %s." "This loop dependency will prevent the execution of the task.", file->name, it->name); } newfile = SD_task_create_comm_e2e(file->name, nullptr, file->amount); SD_task_dependency_add(nullptr, nullptr, it, newfile); SD_task_dependency_add(nullptr, nullptr, newfile, it2); xbt_dynar_push(result, &newfile); } } } } /* Push end task last */ xbt_dynar_push(result, &end_task); /* Free previous copy of the files */ xbt_dict_free(&files); unsigned int cpt; xbt_dynar_foreach(result, cpt, file) { if (SD_task_get_kind(file) == SD_TASK_COMM_E2E) { uniq_transfer_task_name(file); } else if (SD_task_get_kind(file) == SD_TASK_COMP_SEQ){ /* If some tasks do not take files as input, connect them to the root * if they don't produce files, connect them to the end node. */ if ((file != root_task) && file->inputs->empty()) SD_task_dependency_add(nullptr, nullptr, root_task, file); if ((file != end_task) && file->outputs->empty()) SD_task_dependency_add(nullptr, nullptr, file, end_task); } else { THROW_IMPOSSIBLE; } } if (!acyclic_graph_detail(result)) { char* base = xbt_basename(filename); XBT_ERROR("The DAX described in %s is not a DAG. It contains a cycle.", base); free(base); xbt_dynar_foreach(result, cpt, file) SD_task_destroy(file); xbt_dynar_free_container(&result); return nullptr; } else { return result; } } void STag_dax__adag() { XBT_ATTRIB_UNUSED double version; version = xbt_str_parse_double(A_dax__adag_version, "Parse error: %s is not a double"); xbt_assert(version == 2.1, "Expected version 2.1 in tag, got %f. Fix the parser or your file", version); } void STag_dax__job() { double runtime = xbt_str_parse_double(A_dax__job_runtime, "Parse error: %s is not a double"); char *name = bprintf("%s@%s", A_dax__job_id, A_dax__job_name); runtime *= 4200000000.; /* Assume that timings were done on a 4.2GFlops machine. I mean, why not? */ XBT_DEBUG("See ",A_dax__job_id,A_dax__job_runtime,runtime); current_job = SD_task_create_comp_seq(name, nullptr, runtime); xbt_dict_set(jobs, A_dax__job_id, current_job, nullptr); free(name); xbt_dynar_push(result, ¤t_job); } void STag_dax__uses() { double size = xbt_str_parse_double(A_dax__uses_size, "Parse error: %s is not a double"); int is_input = (A_dax__uses_link == A_dax__uses_link_input); XBT_DEBUG("See ",A_dax__uses_file,(is_input?"in":"out")); SD_task_t file = static_cast(xbt_dict_get_or_null(files, A_dax__uses_file)); if (file == nullptr) { file = SD_task_create_comm_e2e(A_dax__uses_file, nullptr, size); sd_global->initial_tasks->erase(file); xbt_dict_set(files, A_dax__uses_file, file, nullptr); } else { if (file->amount < size || file->amount > size) { XBT_WARN("Ignore file %s size redefinition from %.0f to %.0f", A_dax__uses_file, SD_task_get_amount(file), size); } } if (is_input) { SD_task_dependency_add(nullptr, nullptr, file, current_job); } else { SD_task_dependency_add(nullptr, nullptr, current_job, file); if ((file->predecessors->size() + file->inputs->size()) > 1) { XBT_WARN("File %s created at more than one location...", file->name); } } } static SD_task_t current_child; void STag_dax__child() { current_child = static_cast(xbt_dict_get_or_null(jobs, A_dax__child_ref)); xbt_assert(current_child != nullptr,"Parse error on line %d: Asked to add dependencies to the non-existent %s task", dax_lineno, A_dax__child_ref); } void ETag_dax__child() { current_child = nullptr; } void STag_dax__parent() { SD_task_t parent = static_cast(xbt_dict_get_or_null(jobs, A_dax__parent_ref)); xbt_assert(parent != nullptr, "Parse error on line %d: Asked to add a dependency from %s to %s, but %s does not exist", dax_lineno, current_child->name, A_dax__parent_ref, A_dax__parent_ref); SD_task_dependency_add(nullptr, nullptr, parent, current_child); XBT_DEBUG("Control-flow dependency from %s to %s", current_child->name, parent->name); } void ETag_dax__adag() { XBT_DEBUG("See "); } void ETag_dax__job() { current_job = nullptr; XBT_DEBUG("See "); } void ETag_dax__parent() { XBT_DEBUG("See "); } void ETag_dax__uses() { XBT_DEBUG("See "); } SimGrid-3.14.159/src/simdag/sd_task.cpp000644 001750 001750 00000103016 13031031271 021257 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simdag_private.hpp" #include "src/surf/HostImpl.hpp" #include "src/surf/surf_interface.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(sd_task, sd, "Logging specific to SimDag (task)"); /* Destroys the data memorized by SD_task_schedule. Task state must be SD_SCHEDULED or SD_RUNNABLE. */ static void __SD_task_destroy_scheduling_data(SD_task_t task) { if (task->state != SD_SCHEDULED && task->state != SD_RUNNABLE) THROWF(arg_error, 0, "Task '%s' must be SD_SCHEDULED or SD_RUNNABLE", SD_task_get_name(task)); xbt_free(task->flops_amount); xbt_free(task->bytes_amount); task->bytes_amount = nullptr; task->flops_amount = nullptr; } /** * \brief Creates a new task. * * \param name the name of the task (can be \c nullptr) * \param data the user data you want to associate with the task (can be \c nullptr) * \param amount amount of the task * \return the new task * \see SD_task_destroy() */ SD_task_t SD_task_create(const char *name, void *data, double amount) { SD_task_t task = xbt_new0(s_SD_task_t, 1); task->kind = SD_TASK_NOT_TYPED; task->state= SD_NOT_SCHEDULED; sd_global->initial_tasks->insert(task); task->marked = 0; task->start_time = -1.0; task->finish_time = -1.0; task->surf_action = nullptr; task->watch_points = 0; task->inputs = new std::set(); task->outputs = new std::set(); task->predecessors = new std::set(); task->successors = new std::set(); task->data = data; task->name = xbt_strdup(name); task->amount = amount; task->allocation = new std::vector(); task->rate = -1; return task; } static inline SD_task_t SD_task_create_sized(const char *name, void *data, double amount, int count) { SD_task_t task = SD_task_create(name, data, amount); task->bytes_amount = xbt_new0(double, count * count); task->flops_amount = xbt_new0(double, count); return task; } /** @brief create a end-to-end communication task that can then be auto-scheduled * * Auto-scheduling mean that the task can be used with SD_task_schedulev(). This allows to specify the task costs at * creation, and decouple them from the scheduling process where you just specify which resource should deliver the * mandatory power. * * A end-to-end communication must be scheduled on 2 hosts, and the amount specified at creation is sent from hosts[0] * to hosts[1]. */ SD_task_t SD_task_create_comm_e2e(const char *name, void *data, double amount) { SD_task_t res = SD_task_create_sized(name, data, amount, 2); res->bytes_amount[2] = amount; res->kind = SD_TASK_COMM_E2E; return res; } /** @brief create a sequential computation task that can then be auto-scheduled * * Auto-scheduling mean that the task can be used with SD_task_schedulev(). This allows to specify the task costs at * creation, and decouple them from the scheduling process where you just specify which resource should deliver the * mandatory power. * * A sequential computation must be scheduled on 1 host, and the amount specified at creation to be run on hosts[0]. * * \param name the name of the task (can be \c nullptr) * \param data the user data you want to associate with the task (can be \c nullptr) * \param flops_amount amount of compute work to be done by the task * \return the new SD_TASK_COMP_SEQ typed task */ SD_task_t SD_task_create_comp_seq(const char *name, void *data, double flops_amount) { SD_task_t res = SD_task_create_sized(name, data, flops_amount, 1); res->flops_amount[0] = flops_amount; res->kind = SD_TASK_COMP_SEQ; return res; } /** @brief create a parallel computation task that can then be auto-scheduled * * Auto-scheduling mean that the task can be used with SD_task_schedulev(). This allows to specify the task costs at * creation, and decouple them from the scheduling process where you just specify which resource should deliver the * mandatory power. * * A parallel computation can be scheduled on any number of host. * The underlying speedup model is Amdahl's law. * To be auto-scheduled, \see SD_task_distribute_comp_amdahl has to be called first. * \param name the name of the task (can be \c nullptr) * \param data the user data you want to associate with the task (can be \c nullptr) * \param flops_amount amount of compute work to be done by the task * \param alpha purely serial fraction of the work to be done (in [0.;1.[) * \return the new task */ SD_task_t SD_task_create_comp_par_amdahl(const char *name, void *data, double flops_amount, double alpha) { xbt_assert(alpha < 1. && alpha >= 0., "Invalid parameter: alpha must be in [0.;1.["); SD_task_t res = SD_task_create(name, data, flops_amount); res->alpha = alpha; res->kind = SD_TASK_COMP_PAR_AMDAHL; return res; } /** @brief create a complex data redistribution task that can then be auto-scheduled * * Auto-scheduling mean that the task can be used with SD_task_schedulev(). * This allows to specify the task costs at creation, and decouple them from the scheduling process where you just * specify which resource should communicate. * * A data redistribution can be scheduled on any number of host. * The assumed distribution is a 1D block distribution. Each host owns the same share of the \see amount. * To be auto-scheduled, \see SD_task_distribute_comm_mxn_1d_block has to be called first. * \param name the name of the task (can be \c nullptr) * \param data the user data you want to associate with the task (can be \c nullptr) * \param amount amount of data to redistribute by the task * \return the new task */ SD_task_t SD_task_create_comm_par_mxn_1d_block(const char *name, void *data, double amount) { SD_task_t res = SD_task_create(name, data, amount); res->kind = SD_TASK_COMM_PAR_MXN_1D_BLOCK; return res; } /** * \brief Destroys a task. * * The user data (if any) should have been destroyed first. * * \param task the task you want to destroy * \see SD_task_create() */ void SD_task_destroy(SD_task_t task) { XBT_DEBUG("Destroying task %s...", SD_task_get_name(task)); /* First Remove all dependencies associated with the task. */ while (!task->predecessors->empty()) SD_task_dependency_remove(*(task->predecessors->begin()), task); while (!task->inputs->empty()) SD_task_dependency_remove(*(task->inputs->begin()), task); while (!task->successors->empty()) SD_task_dependency_remove(task, *(task->successors->begin())); while (!task->outputs->empty()) SD_task_dependency_remove(task, *(task->outputs->begin())); if (task->state == SD_SCHEDULED || task->state == SD_RUNNABLE) __SD_task_destroy_scheduling_data(task); xbt_free(task->name); if (task->surf_action != nullptr) task->surf_action->unref(); delete task->allocation; xbt_free(task->bytes_amount); xbt_free(task->flops_amount); delete task->inputs; delete task->outputs; delete task->predecessors; delete task->successors; xbt_free(task); XBT_DEBUG("Task destroyed."); } /** * \brief Returns the user data of a task * * \param task a task * \return the user data associated with this task (can be \c nullptr) * \see SD_task_set_data() */ void *SD_task_get_data(SD_task_t task) { return task->data; } /** * \brief Sets the user data of a task * * The new data can be \c nullptr. The old data should have been freed first, if it was not \c nullptr. * * \param task a task * \param data the new data you want to associate with this task * \see SD_task_get_data() */ void SD_task_set_data(SD_task_t task, void *data) { task->data = data; } /** * \brief Sets the rate of a task * * This will change the network bandwidth a task can use. This rate cannot be dynamically changed. Once the task has * started, this call is ineffective. This rate depends on both the nominal bandwidth on the route onto which the task * is scheduled (\see SD_task_get_current_bandwidth) and the amount of data to transfer. * * To divide the nominal bandwidth by 2, the rate then has to be : * rate = bandwidth/(2*amount) * * \param task a \see SD_TASK_COMM_E2E task (end-to-end communication) * \param rate the new rate you want to associate with this task. */ void SD_task_set_rate(SD_task_t task, double rate) { xbt_assert(task->kind == SD_TASK_COMM_E2E, "The rate can be modified for end-to-end communications only."); if(task->state < SD_RUNNING) { task->rate = rate; } else { XBT_WARN("Task %p has started. Changing rate is ineffective.", task); } } /** * \brief Returns the state of a task * * \param task a task * \return the current \ref e_SD_task_state_t "state" of this task: * #SD_NOT_SCHEDULED, #SD_SCHEDULED, #SD_RUNNABLE, #SD_RUNNING, #SD_DONE or #SD_FAILED * \see e_SD_task_state_t */ e_SD_task_state_t SD_task_get_state(SD_task_t task) { return task->state; } /* Changes the state of a task. Updates the sd_global->watch_point_reached flag. */ void SD_task_set_state(SD_task_t task, e_SD_task_state_t new_state) { std::set::iterator idx; XBT_DEBUG("Set state of '%s' to %d", task->name, new_state); if ((new_state == SD_NOT_SCHEDULED || new_state == SD_SCHEDULABLE) && task->state == SD_FAILED){ sd_global->completed_tasks->erase(task); sd_global->initial_tasks->insert(task); } if (new_state == SD_SCHEDULED && task->state == SD_RUNNABLE){ sd_global->initial_tasks->insert(task); sd_global->runnable_tasks->erase(task); } if (new_state == SD_RUNNABLE){ idx = sd_global->initial_tasks->find(task); if (idx != sd_global->initial_tasks->end()) { sd_global->runnable_tasks->insert(*idx); sd_global->initial_tasks->erase(idx); } } if (new_state == SD_RUNNING) sd_global->runnable_tasks->erase(task); if (new_state == SD_DONE || new_state == SD_FAILED){ sd_global->completed_tasks->insert(task); task->start_time = task->surf_action->getStartTime(); if (new_state == SD_DONE){ task->finish_time = task->surf_action->getFinishTime(); #if HAVE_JEDULE jedule_log_sd_event(task); #endif } else task->finish_time = surf_get_clock(); task->surf_action->unref(); task->surf_action = nullptr; task->allocation->clear(); } task->state = new_state; if (task->watch_points & new_state) { XBT_VERB("Watch point reached with task '%s'!", task->name); sd_global->watch_point_reached = true; SD_task_unwatch(task, new_state); /* remove the watch point */ } } /** * \brief Returns the name of a task * * \param task a task * \return the name of this task (can be \c nullptr) */ const char *SD_task_get_name(SD_task_t task) { return task->name; } /** @brief Allows to change the name of a task */ void SD_task_set_name(SD_task_t task, const char *name) { xbt_free(task->name); task->name = xbt_strdup(name); } /** @brief Returns the dynar of the parents of a task * * \param task a task * \return a newly allocated dynar comprising the parents of this task */ xbt_dynar_t SD_task_get_parents(SD_task_t task) { xbt_dynar_t parents = xbt_dynar_new(sizeof(SD_task_t), nullptr); for (auto it : *task->predecessors) xbt_dynar_push(parents, &it); for (auto it : *task->inputs) xbt_dynar_push(parents, &it); return parents; } /** @brief Returns the dynar of the parents of a task * * \param task a task * \return a newly allocated dynar comprising the parents of this task */ xbt_dynar_t SD_task_get_children(SD_task_t task) { xbt_dynar_t children = xbt_dynar_new(sizeof(SD_task_t), nullptr); for (auto it : *task->successors) xbt_dynar_push(children, &it); for (auto it : *task->outputs) xbt_dynar_push(children, &it); return children; } /** * \brief Returns the number of workstations involved in a task * * Only call this on already scheduled tasks! * \param task a task */ int SD_task_get_workstation_count(SD_task_t task) { return task->allocation->size(); } /** * \brief Returns the list of workstations involved in a task * * Only call this on already scheduled tasks! * \param task a task */ sg_host_t *SD_task_get_workstation_list(SD_task_t task) { return &(*(task->allocation))[0]; } /** * \brief Returns the total amount of work contained in a task * * \param task a task * \return the total amount of work (computation or data transfer) for this task * \see SD_task_get_remaining_amount() */ double SD_task_get_amount(SD_task_t task) { return task->amount; } /** @brief Sets the total amount of work of a task * For sequential typed tasks (COMP_SEQ and COMM_E2E), it also sets the appropriate values in the flops_amount and * bytes_amount arrays respectively. Nothing more than modifying task->amount is done for parallel typed tasks * (COMP_PAR_AMDAHL and COMM_PAR_MXN_1D_BLOCK) as the distribution of the amount of work is done at scheduling time. * * \param task a task * \param amount the new amount of work to execute */ void SD_task_set_amount(SD_task_t task, double amount) { task->amount = amount; if (task->kind == SD_TASK_COMP_SEQ) task->flops_amount[0] = amount; if (task->kind == SD_TASK_COMM_E2E) task->bytes_amount[2] = amount; } /** * \brief Returns the alpha parameter of a SD_TASK_COMP_PAR_AMDAHL task * * \param task a parallel task assuming Amdahl's law as speedup model * \return the alpha parameter (serial part of a task in percent) for this task */ double SD_task_get_alpha(SD_task_t task) { xbt_assert(SD_task_get_kind(task) == SD_TASK_COMP_PAR_AMDAHL, "Alpha parameter is not defined for this kind of task"); return task->alpha; } /** * \brief Returns the remaining amount work to do till the completion of a task * * \param task a task * \return the remaining amount of work (computation or data transfer) of this task * \see SD_task_get_amount() */ double SD_task_get_remaining_amount(SD_task_t task) { if (task->surf_action) return task->surf_action->getRemains(); else return (task->state == SD_DONE) ? 0 : task->amount; } e_SD_task_kind_t SD_task_get_kind(SD_task_t task) { return task->kind; } /** @brief Displays debugging information about a task */ void SD_task_dump(SD_task_t task) { XBT_INFO("Displaying task %s", SD_task_get_name(task)); if (task->state == SD_RUNNABLE) XBT_INFO(" - state: runnable"); else if (task->state < SD_RUNNABLE) XBT_INFO(" - state: %s not runnable", __get_state_name(task->state)); else XBT_INFO(" - state: not runnable %s", __get_state_name(task->state)); if (task->kind != 0) { switch (task->kind) { case SD_TASK_COMM_E2E: XBT_INFO(" - kind: end-to-end communication"); break; case SD_TASK_COMP_SEQ: XBT_INFO(" - kind: sequential computation"); break; case SD_TASK_COMP_PAR_AMDAHL: XBT_INFO(" - kind: parallel computation following Amdahl's law"); break; case SD_TASK_COMM_PAR_MXN_1D_BLOCK: XBT_INFO(" - kind: MxN data redistribution assuming 1D block distribution"); break; default: XBT_INFO(" - (unknown kind %d)", task->kind); } } XBT_INFO(" - amount: %.0f", SD_task_get_amount(task)); if (task->kind == SD_TASK_COMP_PAR_AMDAHL) XBT_INFO(" - alpha: %.2f", task->alpha); XBT_INFO(" - Dependencies to satisfy: %zu", task->inputs->size()+ task->predecessors->size()); if ((task->inputs->size()+ task->predecessors->size()) > 0) { XBT_INFO(" - pre-dependencies:"); for (auto it : *task->predecessors) XBT_INFO(" %s", it->name); for (auto it: *task->inputs) XBT_INFO(" %s", it->name); } if ((task->outputs->size() + task->successors->size()) > 0) { XBT_INFO(" - post-dependencies:"); for (auto it : *task->successors) XBT_INFO(" %s", it->name); for (auto it : *task->outputs) XBT_INFO(" %s", it->name); } } /** @brief Dumps the task in dotty formalism into the FILE* passed as second argument */ void SD_task_dotty(SD_task_t task, void *out) { FILE *fout = static_cast(out); fprintf(fout, " T%p [label=\"%.20s\"", task, task->name); switch (task->kind) { case SD_TASK_COMM_E2E: case SD_TASK_COMM_PAR_MXN_1D_BLOCK: fprintf(fout, ", shape=box"); break; case SD_TASK_COMP_SEQ: case SD_TASK_COMP_PAR_AMDAHL: fprintf(fout, ", shape=circle"); break; default: xbt_die("Unknown task type!"); } fprintf(fout, "];\n"); for (auto it : *task->predecessors) fprintf(fout, " T%p -> T%p;\n", it, task); for (auto it : *task->inputs) fprintf(fout, " T%p -> T%p;\n", it, task); } /** * \brief Adds a dependency between two tasks * * \a dst will depend on \a src, ie \a dst will not start before \a src is finished. * Their \ref e_SD_task_state_t "state" must be #SD_NOT_SCHEDULED, #SD_SCHEDULED or #SD_RUNNABLE. * * \param name the name of the new dependency (can be \c nullptr) * \param data the user data you want to associate with this dependency (can be \c nullptr) * \param src the task which must be executed first * \param dst the task you want to make depend on \a src * \see SD_task_dependency_remove() */ void SD_task_dependency_add(const char *name, void *data, SD_task_t src, SD_task_t dst) { if (src == dst) THROWF(arg_error, 0, "Cannot add a dependency between task '%s' and itself", SD_task_get_name(src)); if (src->state == SD_DONE || src->state == SD_FAILED) THROWF(arg_error, 0, "Task '%s' must be SD_NOT_SCHEDULED, SD_SCHEDULABLE, SD_SCHEDULED, SD_RUNNABLE, or SD_RUNNING", src->name); if (dst->state == SD_DONE || dst->state == SD_FAILED || dst->state == SD_RUNNING) THROWF(arg_error, 0, "Task '%s' must be SD_NOT_SCHEDULED, SD_SCHEDULABLE, SD_SCHEDULED, or SD_RUNNABLE", dst->name); if (dst->inputs->find(src) != dst->inputs->end() || src->outputs->find(dst) != src->outputs->end() || src->successors->find(dst) != src->successors->end() || dst->predecessors->find(src) != dst->predecessors->end()) THROWF(arg_error, 0, "A dependency already exists between task '%s' and task '%s'", src->name, dst->name); XBT_DEBUG("SD_task_dependency_add: src = %s, dst = %s", src->name, dst->name); if (src->kind == SD_TASK_COMM_E2E || src->kind == SD_TASK_COMM_PAR_MXN_1D_BLOCK){ if (dst->kind == SD_TASK_COMP_SEQ || dst->kind == SD_TASK_COMP_PAR_AMDAHL) dst->inputs->insert(src); else dst->predecessors->insert(src); src->successors->insert(dst); } else { if (dst->kind == SD_TASK_COMM_E2E|| dst->kind == SD_TASK_COMM_PAR_MXN_1D_BLOCK) src->outputs->insert(dst); else src->successors->insert(dst); dst->predecessors->insert(src); } /* if the task was runnable, the task goes back to SD_SCHEDULED because of the new dependency*/ if (dst->state == SD_RUNNABLE) { XBT_DEBUG("SD_task_dependency_add: %s was runnable and becomes scheduled!", dst->name); SD_task_set_state(dst, SD_SCHEDULED); } } /** * \brief Indicates whether there is a dependency between two tasks. * * \param src a task * \param dst a task depending on \a src * * If src is nullptr, checks whether dst has any pre-dependency. * If dst is nullptr, checks whether src has any post-dependency. */ int SD_task_dependency_exists(SD_task_t src, SD_task_t dst) { xbt_assert(src != nullptr || dst != nullptr, "Invalid parameter: both src and dst are nullptr"); if (src) { if (dst) { return (src->successors->find(dst) != src->successors->end() || src->outputs->find(dst) != src->outputs->end()); } else { return src->successors->size() + src->outputs->size(); } } else { return dst->predecessors->size() + dst->inputs->size(); } return 0; } /** * \brief Remove a dependency between two tasks * * \param src a task * \param dst a task depending on \a src * \see SD_task_dependency_add() */ void SD_task_dependency_remove(SD_task_t src, SD_task_t dst) { XBT_DEBUG("SD_task_dependency_remove: src = %s, dst = %s", SD_task_get_name(src), SD_task_get_name(dst)); if (src->successors->find(dst) == src->successors->end() && src->outputs->find(dst) == src->outputs->end()) THROWF(arg_error, 0, "No dependency found between task '%s' and '%s': task '%s' is not a successor of task '%s'", src->name, dst->name, dst->name, src->name); if (src->kind == SD_TASK_COMM_E2E || src->kind == SD_TASK_COMM_PAR_MXN_1D_BLOCK){ if (dst->kind == SD_TASK_COMP_SEQ || dst->kind == SD_TASK_COMP_PAR_AMDAHL) dst->inputs->erase(src); else dst->predecessors->erase(src); src->successors->erase(dst); } else { if (dst->kind == SD_TASK_COMM_E2E|| dst->kind == SD_TASK_COMM_PAR_MXN_1D_BLOCK) src->outputs->erase(dst); else src->successors->erase(dst); dst->predecessors->erase(src); } /* if the task was scheduled and dependencies are satisfied, we can make it runnable */ if (dst->predecessors->empty() && dst->inputs->empty() && dst->state == SD_SCHEDULED) SD_task_set_state(dst, SD_RUNNABLE); } /** * \brief Adds a watch point to a task * * SD_simulate() will stop as soon as the \ref e_SD_task_state_t "state" of this task becomes the one given in argument. * The watch point is then automatically removed. * * \param task a task * \param state the \ref e_SD_task_state_t "state" you want to watch (cannot be #SD_NOT_SCHEDULED) * \see SD_task_unwatch() */ void SD_task_watch(SD_task_t task, e_SD_task_state_t state) { if (state & SD_NOT_SCHEDULED) THROWF(arg_error, 0, "Cannot add a watch point for state SD_NOT_SCHEDULED"); task->watch_points = task->watch_points | state; } /** * \brief Removes a watch point from a task * * \param task a task * \param state the \ref e_SD_task_state_t "state" you no longer want to watch * \see SD_task_watch() */ void SD_task_unwatch(SD_task_t task, e_SD_task_state_t state) { xbt_assert(state != SD_NOT_SCHEDULED, "SimDag error: Cannot have a watch point for state SD_NOT_SCHEDULED"); task->watch_points = task->watch_points & ~state; } /** * \brief Returns an approximative estimation of the execution time of a task. * * The estimation is very approximative because the value returned is the time the task would take if it was executed * now and if it was the only task. * * \param task the task to evaluate * \param host_count number of hosts on which the task would be executed * \param host_list the hosts on which the task would be executed * \param flops_amount computation amount for each host(i.e., an array of host_count doubles) * \param bytes_amount communication amount between each pair of hosts (i.e., a matrix of host_count*host_count doubles) * \see SD_schedule() */ double SD_task_get_execution_time(SD_task_t task, int host_count, const sg_host_t *host_list, const double *flops_amount, const double *bytes_amount) { xbt_assert(host_count > 0, "Invalid parameter"); double max_time = 0.0; /* the task execution time is the maximum execution time of the parallel tasks */ for (int i = 0; i < host_count; i++) { double time = 0.0; if (flops_amount != nullptr) time = flops_amount[i] / host_list[i]->speed(); if (bytes_amount != nullptr) for (int j = 0; j < host_count; j++) if (bytes_amount[i * host_count + j] != 0) time += (SD_route_get_latency(host_list[i], host_list[j]) + bytes_amount[i * host_count + j] / SD_route_get_bandwidth(host_list[i], host_list[j])); if (time > max_time) max_time = time; } return max_time; } static inline void SD_task_do_schedule(SD_task_t task) { if (SD_task_get_state(task) > SD_SCHEDULABLE) THROWF(arg_error, 0, "Task '%s' has already been scheduled", SD_task_get_name(task)); if (task->predecessors->empty() && task->inputs->empty()) SD_task_set_state(task, SD_RUNNABLE); else SD_task_set_state(task, SD_SCHEDULED); } /** * \brief Schedules a task * * The task state must be #SD_NOT_SCHEDULED. * Once scheduled, a task is executed as soon as possible in \see SD_simulate, i.e. when its dependencies are satisfied. * * \param task the task you want to schedule * \param host_count number of hosts on which the task will be executed * \param host_list the hosts on which the task will be executed * \param flops_amount computation amount for each hosts (i.e., an array of host_count doubles) * \param bytes_amount communication amount between each pair of hosts (i.e., a matrix of host_count*host_count doubles) * \param rate task execution speed rate * \see SD_task_unschedule() */ void SD_task_schedule(SD_task_t task, int host_count, const sg_host_t * host_list, const double *flops_amount, const double *bytes_amount, double rate) { xbt_assert(host_count > 0, "host_count must be positive"); task->rate = rate; if (flops_amount) { task->flops_amount = static_cast(xbt_realloc(task->flops_amount, sizeof(double) * host_count)); memcpy(task->flops_amount, flops_amount, sizeof(double) * host_count); } else { xbt_free(task->flops_amount); task->flops_amount = nullptr; } int communication_nb = host_count * host_count; if (bytes_amount) { task->bytes_amount = static_cast(xbt_realloc(task->bytes_amount, sizeof(double) * communication_nb)); memcpy(task->bytes_amount, bytes_amount, sizeof(double) * communication_nb); } else { xbt_free(task->bytes_amount); task->bytes_amount = nullptr; } for(int i =0; iallocation->push_back(host_list[i]); SD_task_do_schedule(task); } /** * \brief Unschedules a task * * The task state must be #SD_SCHEDULED, #SD_RUNNABLE, #SD_RUNNING or #SD_FAILED. * If you call this function, the task state becomes #SD_NOT_SCHEDULED. * Call SD_task_schedule() to schedule it again. * * \param task the task you want to unschedule * \see SD_task_schedule() */ void SD_task_unschedule(SD_task_t task) { if (task->state == SD_NOT_SCHEDULED || task->state == SD_SCHEDULABLE) THROWF(arg_error, 0, "Task %s: the state must be SD_SCHEDULED, SD_RUNNABLE, SD_RUNNING or SD_FAILED", task->name); if ((task->state == SD_SCHEDULED || task->state == SD_RUNNABLE) /* if the task is scheduled or runnable */ && ((task->kind == SD_TASK_COMP_PAR_AMDAHL) || (task->kind == SD_TASK_COMM_PAR_MXN_1D_BLOCK))) { /* Don't free scheduling data for typed tasks */ __SD_task_destroy_scheduling_data(task); task->allocation->clear(); } if (SD_task_get_state(task) == SD_RUNNING) /* the task should become SD_FAILED */ task->surf_action->cancel(); else { if (task->predecessors->empty() && task->inputs->empty()) SD_task_set_state(task, SD_SCHEDULABLE); else SD_task_set_state(task, SD_NOT_SCHEDULED); } task->start_time = -1.0; } /* Runs a task. */ void SD_task_run(SD_task_t task) { xbt_assert(task->state == SD_RUNNABLE, "Task '%s' is not runnable! Task state: %d", task->name, (int) task->state); xbt_assert(task->allocation != nullptr, "Task '%s': host_list is nullptr!", task->name); XBT_VERB("Executing task '%s'", task->name); /* Copy the elements of the task into the action */ int host_nb = task->allocation->size(); sg_host_t *hosts = xbt_new(sg_host_t, host_nb); int i =0; for (auto host: *task->allocation){ hosts[i] = host; i++; } double *flops_amount = xbt_new0(double, host_nb); double *bytes_amount = xbt_new0(double, host_nb * host_nb); if(task->flops_amount) memcpy(flops_amount, task->flops_amount, sizeof(double) * host_nb); if(task->bytes_amount) memcpy(bytes_amount, task->bytes_amount, sizeof(double) * host_nb * host_nb); task->surf_action = surf_host_model->executeParallelTask(host_nb, hosts, flops_amount, bytes_amount, task->rate); task->surf_action->setData(task); XBT_DEBUG("surf_action = %p", task->surf_action); __SD_task_destroy_scheduling_data(task); /* now the scheduling data are not useful anymore */ SD_task_set_state(task, SD_RUNNING); sd_global->return_set->insert(task); } /** * \brief Returns the start time of a task * * The task state must be SD_RUNNING, SD_DONE or SD_FAILED. * * \param task: a task * \return the start time of this task */ double SD_task_get_start_time(SD_task_t task) { if (task->surf_action) return task->surf_action->getStartTime(); else return task->start_time; } /** * \brief Returns the finish time of a task * * The task state must be SD_RUNNING, SD_DONE or SD_FAILED. * If the state is not completed yet, the returned value is an estimation of the task finish time. This value can * vary until the task is completed. * * \param task: a task * \return the start time of this task */ double SD_task_get_finish_time(SD_task_t task) { if (task->surf_action) /* should never happen as actions are destroyed right after their completion */ return task->surf_action->getFinishTime(); else return task->finish_time; } void SD_task_distribute_comp_amdahl(SD_task_t task, int count) { xbt_assert(task->kind == SD_TASK_COMP_PAR_AMDAHL, "Task %s is not a SD_TASK_COMP_PAR_AMDAHL typed task." "Cannot use this function.", task->name); task->flops_amount = xbt_new0(double, count); task->bytes_amount = xbt_new0(double, count * count); for (int i=0; iflops_amount[i] = (task->alpha + (1 - task->alpha)/count) * task->amount; } } void SD_task_build_MxN_1D_block_matrix(SD_task_t task, int src_nb, int dst_nb){ xbt_assert(task->kind == SD_TASK_COMM_PAR_MXN_1D_BLOCK, "Task %s is not a SD_TASK_COMM_PAR_MXN_1D_BLOCK typed task." "Cannot use this function.", task->name); xbt_free(task->bytes_amount); task->bytes_amount = xbt_new0(double,task->allocation->size() * task->allocation->size()); for (int i=0; iamount/src_nb; double src_end = src_start + task->amount/src_nb; for (int j=0; jamount/dst_nb; double dst_end = dst_start + task->amount/dst_nb; XBT_VERB("(%d->%d): (%.2f, %.2f)-> (%.2f, %.2f)", i, j, src_start, src_end, dst_start, dst_end); task->bytes_amount[i*(src_nb+dst_nb)+src_nb+j]=0.0; if ((src_end > dst_start) && (dst_end > src_start)) { /* There is something to send */ task->bytes_amount[i*(src_nb+dst_nb)+src_nb+j] = MIN(src_end, dst_end)- MAX(src_start, dst_start); XBT_VERB("==> %.2f", task->bytes_amount[i*(src_nb+dst_nb)+src_nb+j]); } } } } /** @brief Auto-schedules a task. * * Auto-scheduling mean that the task can be used with SD_task_schedulev(). This allows to specify the task costs at * creation, and decouple them from the scheduling process where you just specify which resource should deliver the * mandatory power. * * To be auto-schedulable, a task must be a typed computation SD_TASK_COMP_SEQ or SD_TASK_COMP_PAR_AMDAHL. */ void SD_task_schedulev(SD_task_t task, int count, const sg_host_t * list) { xbt_assert(task->kind == SD_TASK_COMP_SEQ || task->kind == SD_TASK_COMP_PAR_AMDAHL, "Task %s is not typed. Cannot automatically schedule it.", SD_task_get_name(task)); for(int i =0; iallocation->push_back(list[i]); XBT_VERB("Schedule computation task %s on %zu host(s)", task->name, task->allocation->size()); if (task->kind == SD_TASK_COMP_SEQ) { if (!task->flops_amount){ /*This task has failed and is rescheduled. Reset the flops_amount*/ task->flops_amount = xbt_new0(double, 1); task->flops_amount[0] = task->amount; } XBT_VERB("It costs %.f flops", task->flops_amount[0]); } if (task->kind == SD_TASK_COMP_PAR_AMDAHL) { SD_task_distribute_comp_amdahl(task, count); XBT_VERB("%.f flops will be distributed following Amdahl's Law", task->flops_amount[0]); } SD_task_do_schedule(task); /* Iterate over all inputs and outputs to say where I am located (and start them if runnable) */ for (auto input : *task->inputs){ int src_nb = input->allocation->size(); int dst_nb = count; if (input->allocation->empty()) XBT_VERB("Sender side of '%s' not scheduled. Set receiver side to '%s''s allocation", input->name, task->name); for (int i=0; iallocation->push_back(task->allocation->at(i)); if (input->allocation->size () > task->allocation->size()) { if (task->kind == SD_TASK_COMP_PAR_AMDAHL) SD_task_build_MxN_1D_block_matrix(input, src_nb, dst_nb); SD_task_do_schedule(input); XBT_VERB ("Auto-Schedule Communication task '%s'. Send %.f bytes from %d hosts to %d hosts.", input->name,input->amount, src_nb, dst_nb); } } for (auto output : *task->outputs){ int src_nb = count; int dst_nb = output->allocation->size(); if (output->allocation->empty()) XBT_VERB("Receiver side of '%s' not scheduled. Set sender side to '%s''s allocation", output->name, task->name); for (int i=0; iallocation->insert(output->allocation->begin()+i, task->allocation->at(i)); if (output->allocation->size () > task->allocation->size()) { if (task->kind == SD_TASK_COMP_PAR_AMDAHL) SD_task_build_MxN_1D_block_matrix(output, src_nb, dst_nb); SD_task_do_schedule(output); XBT_VERB ("Auto-Schedule Communication task %s. Send %.f bytes from %d hosts to %d hosts.", output->name, output->amount, src_nb, dst_nb); } } } /** @brief autoschedule a task on a list of hosts * * This function is similar to SD_task_schedulev(), but takes the list of hosts to schedule onto as separate parameters. * It builds a proper vector of hosts and then call SD_task_schedulev() */ void SD_task_schedulel(SD_task_t task, int count, ...) { va_list ap; sg_host_t *list = xbt_new(sg_host_t, count); va_start(ap, count); for (int i=0; i #include #include #include #include #ifndef WIN32 #include #include #include "../xbt/memory_map.hpp" #include "private.h" #include "private.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_memory, smpi, "Memory layout support for SMPI"); static const int PROT_RWX = (PROT_READ | PROT_WRITE | PROT_EXEC); static const int PROT_RW = (PROT_READ | PROT_WRITE ); XBT_ATTRIB_UNUSED static const int PROT_RX = (PROT_READ | PROT_EXEC ); void smpi_get_executable_global_size() { char buffer[PATH_MAX]; char* full_name = realpath(xbt_binary_name, buffer); if (full_name == nullptr) xbt_die("Could not resolve binary file name"); std::vector map = simgrid::xbt::get_memory_map(getpid()); for (auto i = map.begin(); i != map.end() ; ++i) { // TODO, In practice, this implementation would not detect a completely // anonymous data segment. This does not happen in practice, however. // File backed RW entry: if (i->pathname == full_name && (i->prot & PROT_RWX) == PROT_RW) { smpi_start_data_exe = (char*) i->start_addr; smpi_size_data_exe = i->end_addr - i->start_addr; ++i; /* Here we are making the assumption that a suitable empty region following the rw- area is the end of the data segment. It would be better to check with the size of the data segment. */ if (i != map.end() && i->pathname.empty() && (i->prot & PROT_RWX) == PROT_RW && i->start_addr == (std::uint64_t) smpi_start_data_exe + smpi_size_data_exe) { smpi_size_data_exe = i->end_addr - (std::uint64_t) smpi_start_data_exe; } return; } } xbt_die("Did not find my data segment."); } #endif SimGrid-3.14.159/src/smpi/smpi_static_variables.cpp000644 001750 001750 00000002012 13031031272 023675 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2011-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include "private.h" typedef struct s_smpi_static { void *ptr; void_f_pvoid_t free_fn; } s_smpi_static_t; /** * \brief Holds a reference to all static variables that were registered * via smpi_register_static(). This helps to free them when * SMPI shuts down. */ static std::stack registered_static_variables_stack; void smpi_register_static(void* arg, void_f_pvoid_t free_fn) { s_smpi_static_t elm { arg, free_fn }; registered_static_variables_stack.push(elm); } void smpi_free_static() { while (!registered_static_variables_stack.empty()) { s_smpi_static_t elm = registered_static_variables_stack.top(); elm.free_fn(elm.ptr); registered_static_variables_stack.pop(); } } SimGrid-3.14.159/src/smpi/smpi_mpi_dt.cpp000644 001750 001750 00000204643 13031031272 021650 0ustar00mquinsonmquinson000000 000000 /* smpi_mpi_dt.c -- MPI primitives to handle datatypes */ /* Copyright (c) 2009-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include "private.h" #include "smpi_mpi_dt_private.h" #include "mc/mc.h" #include "xbt/replay.h" #include #include "simgrid/modelchecker.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_mpi_dt, smpi, "Logging specific to SMPI (datatype)"); xbt_dict_t smpi_type_keyvals = nullptr; int type_keyval_id=0;//avoid collisions #define CREATE_MPI_DATATYPE(name, type) \ static s_smpi_mpi_datatype_t mpi_##name = { \ (char*) # name, \ sizeof(type), /* size */ \ 0, /*was 1 sizeof_substruct*/ \ 0, /* lb */ \ sizeof(type), /* ub = lb + size */ \ DT_FLAG_BASIC, /* flags */ \ nullptr, /* attributes */ \ nullptr, /* pointer on extended struct*/ \ 0 /* in_use counter */ \ }; \ const MPI_Datatype name = &mpi_##name; #define CREATE_MPI_DATATYPE_NULL(name) \ static s_smpi_mpi_datatype_t mpi_##name = { \ (char*) # name, \ 0, /* size */ \ 0, /* was 1 sizeof_substruct*/ \ 0, /* lb */ \ 0, /* ub = lb + size */ \ DT_FLAG_BASIC, /* flags */ \ nullptr, /* attributes */ \ nullptr, /* pointer on extended struct*/ \ 0 /* in_use counter */ \ }; \ const MPI_Datatype name = &mpi_##name; //The following are datatypes for the MPI functions MPI_MAXLOC and MPI_MINLOC. typedef struct { float value; int index; } float_int; typedef struct { float value; float index; } float_float; typedef struct { long value; long index; } long_long; typedef struct { double value; double index; } double_double; typedef struct { long value; int index; } long_int; typedef struct { double value; int index; } double_int; typedef struct { short value; int index; } short_int; typedef struct { int value; int index; } int_int; typedef struct { long double value; int index; } long_double_int; typedef struct { int64_t value; int64_t index; } integer128_t; // Predefined data types CREATE_MPI_DATATYPE(MPI_CHAR, char); CREATE_MPI_DATATYPE(MPI_SHORT, short); CREATE_MPI_DATATYPE(MPI_INT, int); CREATE_MPI_DATATYPE(MPI_LONG, long); CREATE_MPI_DATATYPE(MPI_LONG_LONG, long long); CREATE_MPI_DATATYPE(MPI_SIGNED_CHAR, signed char); CREATE_MPI_DATATYPE(MPI_UNSIGNED_CHAR, unsigned char); CREATE_MPI_DATATYPE(MPI_UNSIGNED_SHORT, unsigned short); CREATE_MPI_DATATYPE(MPI_UNSIGNED, unsigned int); CREATE_MPI_DATATYPE(MPI_UNSIGNED_LONG, unsigned long); CREATE_MPI_DATATYPE(MPI_UNSIGNED_LONG_LONG, unsigned long long); CREATE_MPI_DATATYPE(MPI_FLOAT, float); CREATE_MPI_DATATYPE(MPI_DOUBLE, double); CREATE_MPI_DATATYPE(MPI_LONG_DOUBLE, long double); CREATE_MPI_DATATYPE(MPI_WCHAR, wchar_t); CREATE_MPI_DATATYPE(MPI_C_BOOL, bool); CREATE_MPI_DATATYPE(MPI_BYTE, int8_t); CREATE_MPI_DATATYPE(MPI_INT8_T, int8_t); CREATE_MPI_DATATYPE(MPI_INT16_T, int16_t); CREATE_MPI_DATATYPE(MPI_INT32_T, int32_t); CREATE_MPI_DATATYPE(MPI_INT64_T, int64_t); CREATE_MPI_DATATYPE(MPI_UINT8_T, uint8_t); CREATE_MPI_DATATYPE(MPI_UINT16_T, uint16_t); CREATE_MPI_DATATYPE(MPI_UINT32_T, uint32_t); CREATE_MPI_DATATYPE(MPI_UINT64_T, uint64_t); CREATE_MPI_DATATYPE(MPI_C_FLOAT_COMPLEX, float _Complex); CREATE_MPI_DATATYPE(MPI_C_DOUBLE_COMPLEX, double _Complex); CREATE_MPI_DATATYPE(MPI_C_LONG_DOUBLE_COMPLEX, long double _Complex); CREATE_MPI_DATATYPE(MPI_AINT, MPI_Aint); CREATE_MPI_DATATYPE(MPI_OFFSET, MPI_Offset); CREATE_MPI_DATATYPE(MPI_FLOAT_INT, float_int); CREATE_MPI_DATATYPE(MPI_LONG_INT, long_int); CREATE_MPI_DATATYPE(MPI_DOUBLE_INT, double_int); CREATE_MPI_DATATYPE(MPI_SHORT_INT, short_int); CREATE_MPI_DATATYPE(MPI_2INT, int_int); CREATE_MPI_DATATYPE(MPI_2FLOAT, float_float); CREATE_MPI_DATATYPE(MPI_2DOUBLE, double_double); CREATE_MPI_DATATYPE(MPI_2LONG, long_long); CREATE_MPI_DATATYPE(MPI_REAL, float); CREATE_MPI_DATATYPE(MPI_REAL4, float); CREATE_MPI_DATATYPE(MPI_REAL8, float); CREATE_MPI_DATATYPE(MPI_REAL16, double); CREATE_MPI_DATATYPE_NULL(MPI_COMPLEX8); CREATE_MPI_DATATYPE_NULL(MPI_COMPLEX16); CREATE_MPI_DATATYPE_NULL(MPI_COMPLEX32); CREATE_MPI_DATATYPE(MPI_INTEGER1, int); CREATE_MPI_DATATYPE(MPI_INTEGER2, int16_t); CREATE_MPI_DATATYPE(MPI_INTEGER4, int32_t); CREATE_MPI_DATATYPE(MPI_INTEGER8, int64_t); CREATE_MPI_DATATYPE(MPI_INTEGER16, integer128_t); CREATE_MPI_DATATYPE(MPI_LONG_DOUBLE_INT, long_double_int); CREATE_MPI_DATATYPE_NULL(MPI_UB); CREATE_MPI_DATATYPE_NULL(MPI_LB); CREATE_MPI_DATATYPE(MPI_PACKED, char); // Internal use only CREATE_MPI_DATATYPE(MPI_PTR, void*); /** Check if the datatype is usable for communications */ bool is_datatype_valid(MPI_Datatype datatype) { return datatype != MPI_DATATYPE_NULL && ((datatype->flags & DT_FLAG_COMMITED) != 0); } size_t smpi_datatype_size(MPI_Datatype datatype) { return datatype->size; } MPI_Aint smpi_datatype_lb(MPI_Datatype datatype) { return datatype->lb; } MPI_Aint smpi_datatype_ub(MPI_Datatype datatype) { return datatype->ub; } int smpi_datatype_dup(MPI_Datatype datatype, MPI_Datatype* new_t) { int ret=MPI_SUCCESS; *new_t= xbt_new(s_smpi_mpi_datatype_t,1); memcpy(*new_t, datatype, sizeof(s_smpi_mpi_datatype_t)); (*new_t)->in_use=1; (*new_t)->flags &= ~DT_FLAG_PREDEFINED; if (datatype->sizeof_substruct){ (*new_t)->substruct=xbt_malloc(datatype->sizeof_substruct); memcpy((*new_t)->substruct, datatype->substruct, datatype->sizeof_substruct); } if(datatype->name) (*new_t)->name = xbt_strdup(datatype->name); if(datatype->attributes !=nullptr){ (*new_t)->attributes = xbt_dict_new_homogeneous(nullptr); xbt_dict_cursor_t cursor = nullptr; int* key; int flag; void* value_in; void* value_out; xbt_dict_foreach (datatype->attributes, cursor, key, value_in) { smpi_type_key_elem elem = static_cast( xbt_dict_get_or_null_ext(smpi_type_keyvals, reinterpret_cast(key), sizeof(int))); if (elem != nullptr && elem->copy_fn != MPI_NULL_COPY_FN) { ret = elem->copy_fn(datatype, *key, nullptr, value_in, &value_out, &flag); if (ret != MPI_SUCCESS) { smpi_datatype_unuse(*new_t); *new_t = MPI_DATATYPE_NULL; xbt_dict_cursor_free(&cursor); return ret; } if (flag) xbt_dict_set_ext((*new_t)->attributes, reinterpret_cast(key), sizeof(int), value_out, nullptr); } } } return ret; } int smpi_datatype_extent(MPI_Datatype datatype, MPI_Aint * lb, MPI_Aint * extent) { if(datatype == MPI_DATATYPE_NULL){ *lb=0; *extent=0; return MPI_SUCCESS; } *lb = datatype->lb; *extent = datatype->ub - datatype->lb; return MPI_SUCCESS; } MPI_Aint smpi_datatype_get_extent(MPI_Datatype datatype){ if(datatype == MPI_DATATYPE_NULL){ return 0; } return datatype->ub - datatype->lb; } void smpi_datatype_get_name(MPI_Datatype datatype, char* name, int* length){ *length = strlen(datatype->name); strncpy(name, datatype->name, *length+1); } void smpi_datatype_set_name(MPI_Datatype datatype, char* name){ if(datatype->name!=nullptr && (datatype->flags & DT_FLAG_PREDEFINED) == 0) xbt_free(datatype->name); datatype->name = xbt_strdup(name); } int smpi_datatype_copy(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype) { int count; if(smpi_privatize_global_variables){ smpi_switch_data_segment(smpi_process_index()); } /* First check if we really have something to do */ if (recvcount > 0 && recvbuf != sendbuf) { /* FIXME: treat packed cases */ sendcount *= smpi_datatype_size(sendtype); recvcount *= smpi_datatype_size(recvtype); count = sendcount < recvcount ? sendcount : recvcount; if(sendtype->sizeof_substruct == 0 && recvtype->sizeof_substruct == 0) { if(!smpi_process_get_replaying()) memcpy(recvbuf, sendbuf, count); } else if (sendtype->sizeof_substruct == 0) { s_smpi_subtype_t *subtype = static_cast(recvtype->substruct); subtype->unserialize( sendbuf, recvbuf, recvcount/smpi_datatype_size(recvtype), subtype, MPI_REPLACE); } else if (recvtype->sizeof_substruct == 0) { s_smpi_subtype_t *subtype = static_cast(sendtype->substruct); subtype->serialize(sendbuf, recvbuf, sendcount/smpi_datatype_size(sendtype), subtype); }else{ s_smpi_subtype_t *subtype = static_cast(sendtype->substruct); void * buf_tmp = xbt_malloc(count); subtype->serialize( sendbuf, buf_tmp,count/smpi_datatype_size(sendtype), subtype); subtype = static_cast(recvtype->substruct); subtype->unserialize( buf_tmp, recvbuf,count/smpi_datatype_size(recvtype), subtype, MPI_REPLACE); xbt_free(buf_tmp); } } return sendcount > recvcount ? MPI_ERR_TRUNCATE : MPI_SUCCESS; } /* * Copies noncontiguous data into contiguous memory. * @param contiguous_vector - output vector * @param noncontiguous_vector - input vector * @param type - pointer containing : * - stride - stride of between noncontiguous data * - block_length - the width or height of blocked matrix * - count - the number of rows of matrix */ void serialize_vector( const void *noncontiguous_vector, void *contiguous_vector, int count, void *type) { s_smpi_mpi_vector_t* type_c = reinterpret_cast(type); int i; char* contiguous_vector_char = static_cast(contiguous_vector); const char* noncontiguous_vector_char = static_cast(noncontiguous_vector); for (i = 0; i < type_c->block_count * count; i++) { if (type_c->old_type->sizeof_substruct == 0) memcpy(contiguous_vector_char, noncontiguous_vector_char, type_c->block_length * type_c->size_oldtype); else static_cast(type_c->old_type->substruct)->serialize( noncontiguous_vector_char, contiguous_vector_char, type_c->block_length, type_c->old_type->substruct); contiguous_vector_char += type_c->block_length*type_c->size_oldtype; if((i+1)%type_c->block_count ==0) noncontiguous_vector_char += type_c->block_length*smpi_datatype_get_extent(type_c->old_type); else noncontiguous_vector_char += type_c->block_stride*smpi_datatype_get_extent(type_c->old_type); } } /* * Copies contiguous data into noncontiguous memory. * @param noncontiguous_vector - output vector * @param contiguous_vector - input vector * @param type - pointer contening : * - stride - stride of between noncontiguous data * - block_length - the width or height of blocked matrix * - count - the number of rows of matrix */ void unserialize_vector( const void *contiguous_vector, void *noncontiguous_vector, int count, void *type, MPI_Op op) { s_smpi_mpi_vector_t* type_c = reinterpret_cast(type); int i; const char* contiguous_vector_char = static_cast(contiguous_vector); char* noncontiguous_vector_char = static_cast(noncontiguous_vector); for (i = 0; i < type_c->block_count * count; i++) { if (type_c->old_type->sizeof_substruct == 0) smpi_op_apply(op, contiguous_vector_char, noncontiguous_vector_char, &type_c->block_length, &type_c->old_type); else static_cast(type_c->old_type->substruct)->unserialize(contiguous_vector_char, noncontiguous_vector_char, type_c->block_length,type_c->old_type->substruct, op); contiguous_vector_char += type_c->block_length*type_c->size_oldtype; if((i+1)%type_c->block_count ==0) noncontiguous_vector_char += type_c->block_length*smpi_datatype_get_extent(type_c->old_type); else noncontiguous_vector_char += type_c->block_stride*smpi_datatype_get_extent(type_c->old_type); } } /* Create a Sub type vector to be able to serialize and unserialize it the structure s_smpi_mpi_vector_t is derived * from s_smpi_subtype which required the functions unserialize and serialize */ s_smpi_mpi_vector_t* smpi_datatype_vector_create( int block_stride, int block_length, int block_count, MPI_Datatype old_type, int size_oldtype){ s_smpi_mpi_vector_t *new_t= xbt_new(s_smpi_mpi_vector_t,1); new_t->base.serialize = &serialize_vector; new_t->base.unserialize = &unserialize_vector; new_t->base.subtype_free = &free_vector; new_t->base.subtype_use = &use_vector; new_t->block_stride = block_stride; new_t->block_length = block_length; new_t->block_count = block_count; smpi_datatype_use(old_type); new_t->old_type = old_type; new_t->size_oldtype = size_oldtype; return new_t; } void smpi_datatype_create(MPI_Datatype* new_type, int size,int lb, int ub, int sizeof_substruct, void *struct_type, int flags){ MPI_Datatype new_t= xbt_new(s_smpi_mpi_datatype_t,1); new_t->name = nullptr; new_t->size = size; new_t->sizeof_substruct = size>0? sizeof_substruct:0; new_t->lb = lb; new_t->ub = ub; new_t->flags = flags; new_t->substruct = struct_type; new_t->in_use=1; new_t->attributes=nullptr; *new_type = new_t; #if HAVE_MC if(MC_is_active()) MC_ignore(&(new_t->in_use), sizeof(new_t->in_use)); #endif } void smpi_datatype_free(MPI_Datatype* type){ xbt_assert((*type)->in_use >= 0); if((*type)->flags & DT_FLAG_PREDEFINED) return; //if still used, mark for deletion if((*type)->in_use!=0){ (*type)->flags |=DT_FLAG_DESTROYED; return; } if((*type)->attributes !=nullptr){ xbt_dict_cursor_t cursor = nullptr; int* key; void * value; int flag; xbt_dict_foreach((*type)->attributes, cursor, key, value){ smpi_type_key_elem elem = static_cast(xbt_dict_get_or_null_ext(smpi_type_keyvals, reinterpret_cast(key), sizeof(int))); if(elem!=nullptr && elem->delete_fn!=nullptr) elem->delete_fn(*type,*key, value, &flag); } xbt_dict_free(&(*type)->attributes); } if ((*type)->sizeof_substruct != 0){ //((s_smpi_subtype_t *)(*type)->substruct)->subtype_free(type); xbt_free((*type)->substruct); } xbt_free((*type)->name); xbt_free(*type); *type = MPI_DATATYPE_NULL; } void smpi_datatype_use(MPI_Datatype type){ if(type != MPI_DATATYPE_NULL) type->in_use++; if(type->sizeof_substruct!=0){ static_cast((type)->substruct)->subtype_use(&type); } #if HAVE_MC if(MC_is_active()) MC_ignore(&(type->in_use), sizeof(type->in_use)); #endif } void smpi_datatype_unuse(MPI_Datatype type) { if (type == MPI_DATATYPE_NULL) return; if (type->in_use > 0) type->in_use--; if(type->sizeof_substruct!=0){ static_cast((type)->substruct)->subtype_free(&type); } if (type->in_use == 0) smpi_datatype_free(&type); #if HAVE_MC if(MC_is_active()) MC_ignore(&(type->in_use), sizeof(type->in_use)); #endif } /*Contiguous Implementation*/ /* Copies noncontiguous data into contiguous memory. * @param contiguous_hvector - output hvector * @param noncontiguous_hvector - input hvector * @param type - pointer contening : * - stride - stride of between noncontiguous data, in bytes * - block_length - the width or height of blocked matrix * - count - the number of rows of matrix */ void serialize_contiguous( const void *noncontiguous_hvector, void *contiguous_hvector, int count, void *type) { s_smpi_mpi_contiguous_t* type_c = reinterpret_cast(type); char* contiguous_vector_char = static_cast(contiguous_hvector); const char* noncontiguous_vector_char = static_cast(noncontiguous_hvector)+type_c->lb; memcpy(contiguous_vector_char, noncontiguous_vector_char, count* type_c->block_count * type_c->size_oldtype); } /* Copies contiguous data into noncontiguous memory. * @param noncontiguous_vector - output hvector * @param contiguous_vector - input hvector * @param type - pointer contening : * - stride - stride of between noncontiguous data, in bytes * - block_length - the width or height of blocked matrix * - count - the number of rows of matrix */ void unserialize_contiguous(const void *contiguous_vector, void *noncontiguous_vector, int count, void *type, MPI_Op op) { s_smpi_mpi_contiguous_t* type_c = reinterpret_cast(type); const char* contiguous_vector_char = static_cast(contiguous_vector); char* noncontiguous_vector_char = static_cast(noncontiguous_vector)+type_c->lb; int n= count* type_c->block_count; smpi_op_apply(op, contiguous_vector_char, noncontiguous_vector_char, &n, &type_c->old_type); } void free_contiguous(MPI_Datatype* d){ smpi_datatype_unuse(reinterpret_cast((*d)->substruct)->old_type); } void use_contiguous(MPI_Datatype* d){ smpi_datatype_use(reinterpret_cast((*d)->substruct)->old_type); } /* Create a Sub type contiguous to be able to serialize and unserialize it the structure s_smpi_mpi_contiguous_t is * derived from s_smpi_subtype which required the functions unserialize and serialize */ s_smpi_mpi_contiguous_t* smpi_datatype_contiguous_create( MPI_Aint lb, int block_count, MPI_Datatype old_type, int size_oldtype){ if(block_count==0) return nullptr; s_smpi_mpi_contiguous_t *new_t= xbt_new(s_smpi_mpi_contiguous_t,1); new_t->base.serialize = &serialize_contiguous; new_t->base.unserialize = &unserialize_contiguous; new_t->base.subtype_free = &free_contiguous; new_t->base.subtype_use = &use_contiguous; new_t->lb = lb; new_t->block_count = block_count; new_t->old_type = old_type; smpi_datatype_use(old_type); new_t->size_oldtype = size_oldtype; smpi_datatype_use(old_type); return new_t; } int smpi_datatype_contiguous(int count, MPI_Datatype old_type, MPI_Datatype* new_type, MPI_Aint lb) { int retval; if(old_type->sizeof_substruct){ //handle this case as a hvector with stride equals to the extent of the datatype return smpi_datatype_hvector(count, 1, smpi_datatype_get_extent(old_type), old_type, new_type); } s_smpi_mpi_contiguous_t* subtype = smpi_datatype_contiguous_create( lb, count, old_type,smpi_datatype_size(old_type)); smpi_datatype_create(new_type, count * smpi_datatype_size(old_type),lb,lb + count * smpi_datatype_size(old_type), sizeof(s_smpi_mpi_contiguous_t),subtype, DT_FLAG_CONTIGUOUS); retval=MPI_SUCCESS; return retval; } int smpi_datatype_vector(int count, int blocklen, int stride, MPI_Datatype old_type, MPI_Datatype* new_type) { int retval; if (blocklen<0) return MPI_ERR_ARG; MPI_Aint lb = 0; MPI_Aint ub = 0; if(count>0){ lb=smpi_datatype_lb(old_type); ub=((count-1)*stride+blocklen-1)*smpi_datatype_get_extent(old_type)+smpi_datatype_ub(old_type); } if(old_type->sizeof_substruct != 0 || stride != blocklen){ s_smpi_mpi_vector_t* subtype = smpi_datatype_vector_create(stride, blocklen, count, old_type, smpi_datatype_size(old_type)); smpi_datatype_create(new_type, count * (blocklen) * smpi_datatype_size(old_type), lb, ub, sizeof(s_smpi_mpi_vector_t), subtype, DT_FLAG_VECTOR); retval=MPI_SUCCESS; }else{ /* in this situation the data are contiguous thus it's not required to serialize and unserialize it*/ smpi_datatype_create(new_type, count * blocklen * smpi_datatype_size(old_type), 0, ((count -1) * stride + blocklen)* smpi_datatype_size(old_type), 0, nullptr, DT_FLAG_VECTOR|DT_FLAG_CONTIGUOUS); retval=MPI_SUCCESS; } return retval; } void free_vector(MPI_Datatype* d){ smpi_datatype_unuse(reinterpret_cast((*d)->substruct)->old_type); } void use_vector(MPI_Datatype* d){ smpi_datatype_use(reinterpret_cast((*d)->substruct)->old_type); } /* Hvector Implementation - Vector with stride in bytes */ /* Copies noncontiguous data into contiguous memory. * @param contiguous_hvector - output hvector * @param noncontiguous_hvector - input hvector * @param type - pointer contening : * - stride - stride of between noncontiguous data, in bytes * - block_length - the width or height of blocked matrix * - count - the number of rows of matrix */ void serialize_hvector( const void *noncontiguous_hvector, void *contiguous_hvector, int count, void *type) { s_smpi_mpi_hvector_t* type_c = reinterpret_cast(type); int i; char* contiguous_vector_char = static_cast(contiguous_hvector); const char* noncontiguous_vector_char = static_cast(noncontiguous_hvector); for (i = 0; i < type_c->block_count * count; i++) { if (type_c->old_type->sizeof_substruct == 0) memcpy(contiguous_vector_char, noncontiguous_vector_char, type_c->block_length * type_c->size_oldtype); else static_cast(type_c->old_type->substruct)->serialize( noncontiguous_vector_char, contiguous_vector_char, type_c->block_length, type_c->old_type->substruct); contiguous_vector_char += type_c->block_length*type_c->size_oldtype; if((i+1)%type_c->block_count ==0) noncontiguous_vector_char += type_c->block_length*type_c->size_oldtype; else noncontiguous_vector_char += type_c->block_stride; } } /* Copies contiguous data into noncontiguous memory. * @param noncontiguous_vector - output hvector * @param contiguous_vector - input hvector * @param type - pointer contening : * - stride - stride of between noncontiguous data, in bytes * - block_length - the width or height of blocked matrix * - count - the number of rows of matrix */ void unserialize_hvector( const void *contiguous_vector, void *noncontiguous_vector, int count, void *type, MPI_Op op) { s_smpi_mpi_hvector_t* type_c = reinterpret_cast(type); int i; const char* contiguous_vector_char = static_cast(contiguous_vector); char* noncontiguous_vector_char = static_cast(noncontiguous_vector); for (i = 0; i < type_c->block_count * count; i++) { if (type_c->old_type->sizeof_substruct == 0) smpi_op_apply(op, contiguous_vector_char, noncontiguous_vector_char, &type_c->block_length, &type_c->old_type); else static_cast(type_c->old_type->substruct)->unserialize( contiguous_vector_char, noncontiguous_vector_char, type_c->block_length, type_c->old_type->substruct, op); contiguous_vector_char += type_c->block_length*type_c->size_oldtype; if((i+1)%type_c->block_count ==0) noncontiguous_vector_char += type_c->block_length*type_c->size_oldtype; else noncontiguous_vector_char += type_c->block_stride; } } /* Create a Sub type vector to be able to serialize and unserialize it the structure s_smpi_mpi_vector_t is derived * from s_smpi_subtype which required the functions unserialize and serialize * */ s_smpi_mpi_hvector_t* smpi_datatype_hvector_create( MPI_Aint block_stride, int block_length, int block_count, MPI_Datatype old_type, int size_oldtype){ s_smpi_mpi_hvector_t *new_t= xbt_new(s_smpi_mpi_hvector_t,1); new_t->base.serialize = &serialize_hvector; new_t->base.unserialize = &unserialize_hvector; new_t->base.subtype_free = &free_hvector; new_t->base.subtype_use = &use_hvector; new_t->block_stride = block_stride; new_t->block_length = block_length; new_t->block_count = block_count; new_t->old_type = old_type; new_t->size_oldtype = size_oldtype; smpi_datatype_use(old_type); return new_t; } //do nothing for vector types void free_hvector(MPI_Datatype* d){ smpi_datatype_unuse(reinterpret_cast((*d)->substruct)->old_type); } void use_hvector(MPI_Datatype* d){ smpi_datatype_use(reinterpret_cast((*d)->substruct)->old_type); } int smpi_datatype_hvector(int count, int blocklen, MPI_Aint stride, MPI_Datatype old_type, MPI_Datatype* new_type) { int retval; if (blocklen<0) return MPI_ERR_ARG; MPI_Aint lb = 0; MPI_Aint ub = 0; if(count>0){ lb=smpi_datatype_lb(old_type); ub=((count-1)*stride)+(blocklen-1)*smpi_datatype_get_extent(old_type)+smpi_datatype_ub(old_type); } if(old_type->sizeof_substruct != 0 || stride != blocklen*smpi_datatype_get_extent(old_type)){ s_smpi_mpi_hvector_t* subtype = smpi_datatype_hvector_create( stride, blocklen, count, old_type, smpi_datatype_size(old_type)); smpi_datatype_create(new_type, count * blocklen * smpi_datatype_size(old_type), lb,ub, sizeof(s_smpi_mpi_hvector_t), subtype, DT_FLAG_VECTOR); retval=MPI_SUCCESS; }else{ smpi_datatype_create(new_type, count * blocklen * smpi_datatype_size(old_type),0,count * blocklen * smpi_datatype_size(old_type), 0, nullptr, DT_FLAG_VECTOR|DT_FLAG_CONTIGUOUS); retval=MPI_SUCCESS; } return retval; } /* Indexed Implementation */ /* Copies noncontiguous data into contiguous memory. * @param contiguous_indexed - output indexed * @param noncontiguous_indexed - input indexed * @param type - pointer contening : * - block_lengths - the width or height of blocked matrix * - block_indices - indices of each data, in element * - count - the number of rows of matrix */ void serialize_indexed( const void *noncontiguous_indexed, void *contiguous_indexed, int count, void *type) { s_smpi_mpi_indexed_t* type_c = reinterpret_cast(type); int i,j; char* contiguous_indexed_char = static_cast(contiguous_indexed); const char* noncontiguous_indexed_char = static_cast(noncontiguous_indexed)+type_c->block_indices[0] * type_c->size_oldtype; for(j=0; jblock_count; i++) { if (type_c->old_type->sizeof_substruct == 0) memcpy(contiguous_indexed_char, noncontiguous_indexed_char, type_c->block_lengths[i] * type_c->size_oldtype); else static_cast(type_c->old_type->substruct)->serialize( noncontiguous_indexed_char, contiguous_indexed_char, type_c->block_lengths[i], type_c->old_type->substruct); contiguous_indexed_char += type_c->block_lengths[i]*type_c->size_oldtype; if (iblock_count-1) noncontiguous_indexed_char = static_cast(noncontiguous_indexed) + type_c->block_indices[i+1]*smpi_datatype_get_extent(type_c->old_type); else noncontiguous_indexed_char += type_c->block_lengths[i]*smpi_datatype_get_extent(type_c->old_type); } noncontiguous_indexed=static_cast(noncontiguous_indexed_char); } } /* Copies contiguous data into noncontiguous memory. * @param noncontiguous_indexed - output indexed * @param contiguous_indexed - input indexed * @param type - pointer contening : * - block_lengths - the width or height of blocked matrix * - block_indices - indices of each data, in element * - count - the number of rows of matrix */ void unserialize_indexed( const void *contiguous_indexed, void *noncontiguous_indexed, int count, void *type, MPI_Op op) { s_smpi_mpi_indexed_t* type_c = reinterpret_cast(type); int i,j; const char* contiguous_indexed_char = static_cast(contiguous_indexed); char* noncontiguous_indexed_char = static_cast(noncontiguous_indexed)+type_c->block_indices[0]*smpi_datatype_get_extent(type_c->old_type); for(j=0; jblock_count; i++) { if (type_c->old_type->sizeof_substruct == 0) smpi_op_apply(op, contiguous_indexed_char, noncontiguous_indexed_char, &type_c->block_lengths[i], &type_c->old_type); else static_cast(type_c->old_type->substruct)->unserialize( contiguous_indexed_char, noncontiguous_indexed_char, type_c->block_lengths[i], type_c->old_type->substruct, op); contiguous_indexed_char += type_c->block_lengths[i]*type_c->size_oldtype; if (iblock_count-1) noncontiguous_indexed_char = static_cast(noncontiguous_indexed) + type_c->block_indices[i+1]*smpi_datatype_get_extent(type_c->old_type); else noncontiguous_indexed_char += type_c->block_lengths[i]*smpi_datatype_get_extent(type_c->old_type); } noncontiguous_indexed=static_cast(noncontiguous_indexed_char); } } void free_indexed(MPI_Datatype* type){ if((*type)->in_use==0){ xbt_free(reinterpret_cast((*type)->substruct)->block_lengths); xbt_free(reinterpret_cast((*type)->substruct)->block_indices); } smpi_datatype_unuse(reinterpret_cast((*type)->substruct)->old_type); } void use_indexed(MPI_Datatype* type){ smpi_datatype_use(reinterpret_cast((*type)->substruct)->old_type); } /* Create a Sub type indexed to be able to serialize and unserialize it the structure s_smpi_mpi_indexed_t is derived * from s_smpi_subtype which required the functions unserialize and serialize */ s_smpi_mpi_indexed_t* smpi_datatype_indexed_create( int* block_lengths, int* block_indices, int block_count, MPI_Datatype old_type, int size_oldtype){ s_smpi_mpi_indexed_t *new_t= xbt_new(s_smpi_mpi_indexed_t,1); new_t->base.serialize = &serialize_indexed; new_t->base.unserialize = &unserialize_indexed; new_t->base.subtype_free = &free_indexed; new_t->base.subtype_use = &use_indexed; new_t->block_lengths= xbt_new(int, block_count); new_t->block_indices= xbt_new(int, block_count); int i; for(i=0;iblock_lengths[i]=block_lengths[i]; new_t->block_indices[i]=block_indices[i]; } new_t->block_count = block_count; smpi_datatype_use(old_type); new_t->old_type = old_type; new_t->size_oldtype = size_oldtype; return new_t; } int smpi_datatype_indexed(int count, int* blocklens, int* indices, MPI_Datatype old_type, MPI_Datatype* new_type) { int i; int retval; int size = 0; bool contiguous=true; MPI_Aint lb = 0; MPI_Aint ub = 0; if(count>0){ lb=indices[0]*smpi_datatype_get_extent(old_type); ub=indices[0]*smpi_datatype_get_extent(old_type) + blocklens[0]*smpi_datatype_ub(old_type); } for(i=0; i< count; i++){ if (blocklens[i]<0) return MPI_ERR_ARG; size += blocklens[i]; if(indices[i]*smpi_datatype_get_extent(old_type)+smpi_datatype_lb(old_type)ub) ub = indices[i]*smpi_datatype_get_extent(old_type)+blocklens[i]*smpi_datatype_ub(old_type); if ( (i< count -1) && (indices[i]+blocklens[i] != indices[i+1]) ) contiguous=false; } if (old_type->sizeof_substruct != 0) contiguous=false; if(!contiguous){ s_smpi_mpi_indexed_t* subtype = smpi_datatype_indexed_create( blocklens, indices, count, old_type, smpi_datatype_size(old_type)); smpi_datatype_create(new_type, size * smpi_datatype_size(old_type),lb,ub,sizeof(s_smpi_mpi_indexed_t), subtype, DT_FLAG_DATA); }else{ s_smpi_mpi_contiguous_t* subtype = smpi_datatype_contiguous_create( lb, size, old_type, smpi_datatype_size(old_type)); smpi_datatype_create(new_type, size * smpi_datatype_size(old_type), lb, ub, sizeof(s_smpi_mpi_contiguous_t), subtype, DT_FLAG_DATA|DT_FLAG_CONTIGUOUS); } retval=MPI_SUCCESS; return retval; } /* Hindexed Implementation - Indexed with indices in bytes */ /* Copies noncontiguous data into contiguous memory. * @param contiguous_hindexed - output hindexed * @param noncontiguous_hindexed - input hindexed * @param type - pointer contening : * - block_lengths - the width or height of blocked matrix * - block_indices - indices of each data, in bytes * - count - the number of rows of matrix */ void serialize_hindexed( const void *noncontiguous_hindexed, void *contiguous_hindexed, int count, void *type) { s_smpi_mpi_hindexed_t* type_c = reinterpret_cast(type); int i,j; char* contiguous_hindexed_char = static_cast(contiguous_hindexed); const char* noncontiguous_hindexed_char = static_cast(noncontiguous_hindexed)+ type_c->block_indices[0]; for(j=0; jblock_count; i++) { if (type_c->old_type->sizeof_substruct == 0) memcpy(contiguous_hindexed_char, noncontiguous_hindexed_char, type_c->block_lengths[i] * type_c->size_oldtype); else static_cast(type_c->old_type->substruct)->serialize( noncontiguous_hindexed_char, contiguous_hindexed_char, type_c->block_lengths[i], type_c->old_type->substruct); contiguous_hindexed_char += type_c->block_lengths[i]*type_c->size_oldtype; if (iblock_count-1) noncontiguous_hindexed_char = static_cast(noncontiguous_hindexed) + type_c->block_indices[i+1]; else noncontiguous_hindexed_char += type_c->block_lengths[i]*smpi_datatype_get_extent(type_c->old_type); } noncontiguous_hindexed=reinterpret_cast(noncontiguous_hindexed_char); } } /* Copies contiguous data into noncontiguous memory. * @param noncontiguous_hindexed - output hindexed * @param contiguous_hindexed - input hindexed * @param type - pointer contening : * - block_lengths - the width or height of blocked matrix * - block_indices - indices of each data, in bytes * - count - the number of rows of matrix */ void unserialize_hindexed( const void *contiguous_hindexed, void *noncontiguous_hindexed, int count, void *type, MPI_Op op) { s_smpi_mpi_hindexed_t* type_c = reinterpret_cast(type); int i,j; const char* contiguous_hindexed_char = static_cast(contiguous_hindexed); char* noncontiguous_hindexed_char = static_cast(noncontiguous_hindexed)+ type_c->block_indices[0]; for(j=0; jblock_count; i++) { if (type_c->old_type->sizeof_substruct == 0) smpi_op_apply(op, contiguous_hindexed_char, noncontiguous_hindexed_char, &type_c->block_lengths[i], &type_c->old_type); else static_cast(type_c->old_type->substruct)->unserialize( contiguous_hindexed_char, noncontiguous_hindexed_char, type_c->block_lengths[i], type_c->old_type->substruct, op); contiguous_hindexed_char += type_c->block_lengths[i]*type_c->size_oldtype; if (iblock_count-1) noncontiguous_hindexed_char = static_cast(noncontiguous_hindexed) + type_c->block_indices[i+1]; else noncontiguous_hindexed_char += type_c->block_lengths[i]*smpi_datatype_get_extent(type_c->old_type); } noncontiguous_hindexed=reinterpret_cast(noncontiguous_hindexed_char); } } void free_hindexed(MPI_Datatype* type){ if((*type)->in_use==0){ xbt_free(reinterpret_cast((*type)->substruct)->block_lengths); xbt_free(reinterpret_cast((*type)->substruct)->block_indices); } smpi_datatype_unuse(reinterpret_cast((*type)->substruct)->old_type); } void use_hindexed(MPI_Datatype* type){ smpi_datatype_use(reinterpret_cast((*type)->substruct)->old_type); } /* Create a Sub type hindexed to be able to serialize and unserialize it the structure s_smpi_mpi_hindexed_t is derived * from s_smpi_subtype which required the functions unserialize and serialize */ s_smpi_mpi_hindexed_t* smpi_datatype_hindexed_create( int* block_lengths, MPI_Aint* block_indices, int block_count, MPI_Datatype old_type, int size_oldtype){ s_smpi_mpi_hindexed_t *new_t= xbt_new(s_smpi_mpi_hindexed_t,1); new_t->base.serialize = &serialize_hindexed; new_t->base.unserialize = &unserialize_hindexed; new_t->base.subtype_free = &free_hindexed; new_t->base.subtype_use = &use_hindexed; new_t->block_lengths= xbt_new(int, block_count); new_t->block_indices= xbt_new(MPI_Aint, block_count); for(int i=0;iblock_lengths[i]=block_lengths[i]; new_t->block_indices[i]=block_indices[i]; } new_t->block_count = block_count; new_t->old_type = old_type; smpi_datatype_use(old_type); new_t->size_oldtype = size_oldtype; return new_t; } int smpi_datatype_hindexed(int count, int* blocklens, MPI_Aint* indices, MPI_Datatype old_type, MPI_Datatype* new_type) { int i; int retval; int size = 0; bool contiguous=true; MPI_Aint lb = 0; MPI_Aint ub = 0; if(count>0){ lb=indices[0] + smpi_datatype_lb(old_type); ub=indices[0] + blocklens[0]*smpi_datatype_ub(old_type); } for(i=0; i< count; i++){ if (blocklens[i]<0) return MPI_ERR_ARG; size += blocklens[i]; if(indices[i]+smpi_datatype_lb(old_type)ub) ub = indices[i]+blocklens[i]*smpi_datatype_ub(old_type); if ( (i< count -1) && (indices[i]+blocklens[i]*(static_cast(smpi_datatype_size(old_type))) != indices[i+1]) ) contiguous=false; } if (old_type->sizeof_substruct != 0 || lb!=0) contiguous=false; if(!contiguous){ s_smpi_mpi_hindexed_t* subtype = smpi_datatype_hindexed_create( blocklens, indices, count, old_type, smpi_datatype_size(old_type)); smpi_datatype_create(new_type, size * smpi_datatype_size(old_type), lb, ub ,sizeof(s_smpi_mpi_hindexed_t), subtype, DT_FLAG_DATA); }else{ s_smpi_mpi_contiguous_t* subtype = smpi_datatype_contiguous_create(lb,size, old_type, smpi_datatype_size(old_type)); smpi_datatype_create(new_type, size * smpi_datatype_size(old_type), 0,size * smpi_datatype_size(old_type), 1, subtype, DT_FLAG_DATA|DT_FLAG_CONTIGUOUS); } retval=MPI_SUCCESS; return retval; } /* struct Implementation - Indexed with indices in bytes */ /* Copies noncontiguous data into contiguous memory. * @param contiguous_struct - output struct * @param noncontiguous_struct - input struct * @param type - pointer contening : * - stride - stride of between noncontiguous data * - block_length - the width or height of blocked matrix * - count - the number of rows of matrix */ void serialize_struct( const void *noncontiguous_struct, void *contiguous_struct, int count, void *type) { s_smpi_mpi_struct_t* type_c = reinterpret_cast(type); int i,j; char* contiguous_struct_char = static_cast(contiguous_struct); const char* noncontiguous_struct_char = static_cast(noncontiguous_struct)+ type_c->block_indices[0]; for(j=0; jblock_count; i++) { if (type_c->old_types[i]->sizeof_substruct == 0) memcpy(contiguous_struct_char, noncontiguous_struct_char, type_c->block_lengths[i] * smpi_datatype_size(type_c->old_types[i])); else static_cast(type_c->old_types[i]->substruct)->serialize( noncontiguous_struct_char, contiguous_struct_char, type_c->block_lengths[i], type_c->old_types[i]->substruct); contiguous_struct_char += type_c->block_lengths[i]*smpi_datatype_size(type_c->old_types[i]); if (iblock_count-1) noncontiguous_struct_char = static_cast(noncontiguous_struct) + type_c->block_indices[i+1]; else //let's hope this is MPI_UB ? noncontiguous_struct_char += type_c->block_lengths[i]*smpi_datatype_get_extent(type_c->old_types[i]); } noncontiguous_struct=reinterpret_cast(noncontiguous_struct_char); } } /* Copies contiguous data into noncontiguous memory. * @param noncontiguous_struct - output struct * @param contiguous_struct - input struct * @param type - pointer contening : * - stride - stride of between noncontiguous data * - block_length - the width or height of blocked matrix * - count - the number of rows of matrix */ void unserialize_struct( const void *contiguous_struct, void *noncontiguous_struct, int count, void *type, MPI_Op op) { s_smpi_mpi_struct_t* type_c = reinterpret_cast(type); int i,j; const char* contiguous_struct_char = static_cast(contiguous_struct); char* noncontiguous_struct_char = static_cast(noncontiguous_struct)+ type_c->block_indices[0]; for(j=0; jblock_count; i++) { if (type_c->old_types[i]->sizeof_substruct == 0) smpi_op_apply(op, contiguous_struct_char, noncontiguous_struct_char, &type_c->block_lengths[i], & type_c->old_types[i]); else static_cast(type_c->old_types[i]->substruct)->unserialize( contiguous_struct_char, noncontiguous_struct_char, type_c->block_lengths[i], type_c->old_types[i]->substruct, op); contiguous_struct_char += type_c->block_lengths[i]*smpi_datatype_size(type_c->old_types[i]); if (iblock_count-1) noncontiguous_struct_char = static_cast(noncontiguous_struct) + type_c->block_indices[i+1]; else noncontiguous_struct_char += type_c->block_lengths[i]*smpi_datatype_get_extent(type_c->old_types[i]); } noncontiguous_struct=reinterpret_cast(noncontiguous_struct_char); } } void free_struct(MPI_Datatype* type){ int i=0; for (i = 0; i < reinterpret_cast((*type)->substruct)->block_count; i++) smpi_datatype_unuse(reinterpret_cast((*type)->substruct)->old_types[i]); if((*type)->in_use==0){ xbt_free(reinterpret_cast((*type)->substruct)->block_lengths); xbt_free(reinterpret_cast((*type)->substruct)->block_indices); xbt_free(reinterpret_cast((*type)->substruct)->old_types); } } void use_struct(MPI_Datatype* type){ int i=0; for (i = 0; i < reinterpret_cast((*type)->substruct)->block_count; i++) smpi_datatype_use(reinterpret_cast((*type)->substruct)->old_types[i]); } /* Create a Sub type struct to be able to serialize and unserialize it the structure s_smpi_mpi_struct_t is derived * from s_smpi_subtype which required the functions unserialize and serialize */ s_smpi_mpi_struct_t* smpi_datatype_struct_create( int* block_lengths, MPI_Aint* block_indices, int block_count, MPI_Datatype* old_types){ s_smpi_mpi_struct_t *new_t= xbt_new(s_smpi_mpi_struct_t,1); new_t->base.serialize = &serialize_struct; new_t->base.unserialize = &unserialize_struct; new_t->base.subtype_free = &free_struct; new_t->base.subtype_use = &use_struct; new_t->block_lengths= xbt_new(int, block_count); new_t->block_indices= xbt_new(MPI_Aint, block_count); new_t->old_types= xbt_new(MPI_Datatype, block_count); int i; for(i=0;iblock_lengths[i]=block_lengths[i]; new_t->block_indices[i]=block_indices[i]; new_t->old_types[i]=old_types[i]; smpi_datatype_use(new_t->old_types[i]); } new_t->block_count = block_count; return new_t; } int smpi_datatype_struct(int count, int* blocklens, MPI_Aint* indices, MPI_Datatype* old_types, MPI_Datatype* new_type) { int i; size_t size = 0; bool contiguous=true; size = 0; MPI_Aint lb = 0; MPI_Aint ub = 0; if(count>0){ lb=indices[0] + smpi_datatype_lb(old_types[0]); ub=indices[0] + blocklens[0]*smpi_datatype_ub(old_types[0]); } bool forced_lb=false; bool forced_ub=false; for(i=0; i< count; i++){ if (blocklens[i]<0) return MPI_ERR_ARG; if (old_types[i]->sizeof_substruct != 0) contiguous=false; size += blocklens[i]*smpi_datatype_size(old_types[i]); if (old_types[i]==MPI_LB){ lb=indices[i]; forced_lb=true; } if (old_types[i]==MPI_UB){ ub=indices[i]; forced_ub=true; } if(!forced_lb && indices[i]+smpi_datatype_lb(old_types[i])ub) ub = indices[i]+blocklens[i]*smpi_datatype_ub(old_types[i]); if ( (i< count -1) && (indices[i]+blocklens[i]*static_cast(smpi_datatype_size(old_types[i])) != indices[i+1]) ) contiguous=false; } if(!contiguous){ s_smpi_mpi_struct_t* subtype = smpi_datatype_struct_create( blocklens, indices, count, old_types); smpi_datatype_create(new_type, size, lb, ub,sizeof(s_smpi_mpi_struct_t), subtype, DT_FLAG_DATA); }else{ s_smpi_mpi_contiguous_t* subtype = smpi_datatype_contiguous_create( lb, size, MPI_CHAR, 1); smpi_datatype_create(new_type, size, lb, ub,1, subtype, DT_FLAG_DATA|DT_FLAG_CONTIGUOUS); } return MPI_SUCCESS; } void smpi_datatype_commit(MPI_Datatype *datatype) { (*datatype)->flags= ((*datatype)->flags | DT_FLAG_COMMITED); } typedef struct s_smpi_mpi_op { MPI_User_function *func; bool is_commute; } s_smpi_mpi_op_t; #define MAX_OP(a, b) (b) = (a) < (b) ? (b) : (a) #define MIN_OP(a, b) (b) = (a) < (b) ? (a) : (b) #define SUM_OP(a, b) (b) += (a) #define PROD_OP(a, b) (b) *= (a) #define LAND_OP(a, b) (b) = (a) && (b) #define LOR_OP(a, b) (b) = (a) || (b) #define LXOR_OP(a, b) (b) = (!(a) && (b)) || ((a) && !(b)) #define BAND_OP(a, b) (b) &= (a) #define BOR_OP(a, b) (b) |= (a) #define BXOR_OP(a, b) (b) ^= (a) #define MAXLOC_OP(a, b) (b) = (a.value) < (b.value) ? (b) : (a) #define MINLOC_OP(a, b) (b) = (a.value) < (b.value) ? (a) : (b) #define APPLY_FUNC(a, b, length, type, func) \ { \ int i; \ type* x = (type*)(a); \ type* y = (type*)(b); \ for(i = 0; i < *(length); i++) { \ func(x[i], y[i]); \ } \ } static void max_func(void *a, void *b, int *length, MPI_Datatype * datatype) { if (*datatype == MPI_CHAR) { APPLY_FUNC(a, b, length, char, MAX_OP) } else if (*datatype == MPI_SHORT) { APPLY_FUNC(a, b, length, short, MAX_OP) } else if (*datatype == MPI_INT) { APPLY_FUNC(a, b, length, int, MAX_OP) } else if (*datatype == MPI_LONG) { APPLY_FUNC(a, b, length, long, MAX_OP) } else if (*datatype == MPI_UNSIGNED_SHORT) { APPLY_FUNC(a, b, length, unsigned short, MAX_OP) } else if (*datatype == MPI_UNSIGNED) { APPLY_FUNC(a, b, length, unsigned int, MAX_OP) } else if (*datatype == MPI_UNSIGNED_LONG) { APPLY_FUNC(a, b, length, unsigned long, MAX_OP) } else if (*datatype == MPI_UNSIGNED_CHAR) { APPLY_FUNC(a, b, length, unsigned char, MAX_OP) } else if (*datatype == MPI_FLOAT) { APPLY_FUNC(a, b, length, float, MAX_OP) } else if (*datatype == MPI_DOUBLE) { APPLY_FUNC(a, b, length, double, MAX_OP) } else if (*datatype == MPI_LONG_DOUBLE) { APPLY_FUNC(a, b, length, long double, MAX_OP) } else { xbt_die("Failed to apply MAX_OP to type %s", (*datatype)->name); } } static void min_func(void *a, void *b, int *length, MPI_Datatype * datatype) { if (*datatype == MPI_CHAR) { APPLY_FUNC(a, b, length, char, MIN_OP) } else if (*datatype == MPI_SHORT) { APPLY_FUNC(a, b, length, short, MIN_OP) } else if (*datatype == MPI_INT) { APPLY_FUNC(a, b, length, int, MIN_OP) } else if (*datatype == MPI_LONG) { APPLY_FUNC(a, b, length, long, MIN_OP) } else if (*datatype == MPI_UNSIGNED_SHORT) { APPLY_FUNC(a, b, length, unsigned short, MIN_OP) } else if (*datatype == MPI_UNSIGNED) { APPLY_FUNC(a, b, length, unsigned int, MIN_OP) } else if (*datatype == MPI_UNSIGNED_LONG) { APPLY_FUNC(a, b, length, unsigned long, MIN_OP) } else if (*datatype == MPI_UNSIGNED_CHAR) { APPLY_FUNC(a, b, length, unsigned char, MIN_OP) } else if (*datatype == MPI_FLOAT) { APPLY_FUNC(a, b, length, float, MIN_OP) } else if (*datatype == MPI_DOUBLE) { APPLY_FUNC(a, b, length, double, MIN_OP) } else if (*datatype == MPI_LONG_DOUBLE) { APPLY_FUNC(a, b, length, long double, MIN_OP) } else { xbt_die("Failed to apply MIN_OP to type %s", (*datatype)->name); } } static void sum_func(void *a, void *b, int *length, MPI_Datatype * datatype) { if (*datatype == MPI_CHAR) { APPLY_FUNC(a, b, length, char, SUM_OP) } else if (*datatype == MPI_SHORT) { APPLY_FUNC(a, b, length, short, SUM_OP) } else if (*datatype == MPI_INT) { APPLY_FUNC(a, b, length, int, SUM_OP) } else if (*datatype == MPI_LONG) { APPLY_FUNC(a, b, length, long, SUM_OP) } else if (*datatype == MPI_UNSIGNED_SHORT) { APPLY_FUNC(a, b, length, unsigned short, SUM_OP) } else if (*datatype == MPI_UNSIGNED) { APPLY_FUNC(a, b, length, unsigned int, SUM_OP) } else if (*datatype == MPI_UNSIGNED_LONG) { APPLY_FUNC(a, b, length, unsigned long, SUM_OP) } else if (*datatype == MPI_UNSIGNED_CHAR) { APPLY_FUNC(a, b, length, unsigned char, SUM_OP) } else if (*datatype == MPI_FLOAT) { APPLY_FUNC(a, b, length, float, SUM_OP) } else if (*datatype == MPI_DOUBLE) { APPLY_FUNC(a, b, length, double, SUM_OP) } else if (*datatype == MPI_LONG_DOUBLE) { APPLY_FUNC(a, b, length, long double, SUM_OP) } else if (*datatype == MPI_C_FLOAT_COMPLEX) { APPLY_FUNC(a, b, length, float _Complex, SUM_OP) } else if (*datatype == MPI_C_DOUBLE_COMPLEX) { APPLY_FUNC(a, b, length, double _Complex, SUM_OP) } else if (*datatype == MPI_C_LONG_DOUBLE_COMPLEX) { APPLY_FUNC(a, b, length, long double _Complex, SUM_OP) } else { xbt_die("Failed to apply SUM_OP to type %s", (*datatype)->name); } } static void prod_func(void *a, void *b, int *length, MPI_Datatype * datatype) { if (*datatype == MPI_CHAR) { APPLY_FUNC(a, b, length, char, PROD_OP) } else if (*datatype == MPI_SHORT) { APPLY_FUNC(a, b, length, short, PROD_OP) } else if (*datatype == MPI_INT) { APPLY_FUNC(a, b, length, int, PROD_OP) } else if (*datatype == MPI_LONG) { APPLY_FUNC(a, b, length, long, PROD_OP) } else if (*datatype == MPI_UNSIGNED_SHORT) { APPLY_FUNC(a, b, length, unsigned short, PROD_OP) } else if (*datatype == MPI_UNSIGNED) { APPLY_FUNC(a, b, length, unsigned int, PROD_OP) } else if (*datatype == MPI_UNSIGNED_LONG) { APPLY_FUNC(a, b, length, unsigned long, PROD_OP) } else if (*datatype == MPI_UNSIGNED_CHAR) { APPLY_FUNC(a, b, length, unsigned char, PROD_OP) } else if (*datatype == MPI_FLOAT) { APPLY_FUNC(a, b, length, float, PROD_OP) } else if (*datatype == MPI_DOUBLE) { APPLY_FUNC(a, b, length, double, PROD_OP) } else if (*datatype == MPI_LONG_DOUBLE) { APPLY_FUNC(a, b, length, long double, PROD_OP) } else if (*datatype == MPI_C_FLOAT_COMPLEX) { APPLY_FUNC(a, b, length, float _Complex, PROD_OP) } else if (*datatype == MPI_C_DOUBLE_COMPLEX) { APPLY_FUNC(a, b, length, double _Complex, PROD_OP) } else if (*datatype == MPI_C_LONG_DOUBLE_COMPLEX) { APPLY_FUNC(a, b, length, long double _Complex, PROD_OP) } else { xbt_die("Failed to apply PROD_OP to type %s", (*datatype)->name); } } static void land_func(void *a, void *b, int *length, MPI_Datatype * datatype) { if (*datatype == MPI_CHAR) { APPLY_FUNC(a, b, length, char, LAND_OP) } else if (*datatype == MPI_SHORT) { APPLY_FUNC(a, b, length, short, LAND_OP) } else if (*datatype == MPI_INT) { APPLY_FUNC(a, b, length, int, LAND_OP) } else if (*datatype == MPI_LONG) { APPLY_FUNC(a, b, length, long, LAND_OP) } else if (*datatype == MPI_UNSIGNED_SHORT) { APPLY_FUNC(a, b, length, unsigned short, LAND_OP) } else if (*datatype == MPI_UNSIGNED) { APPLY_FUNC(a, b, length, unsigned int, LAND_OP) } else if (*datatype == MPI_UNSIGNED_LONG) { APPLY_FUNC(a, b, length, unsigned long, LAND_OP) } else if (*datatype == MPI_UNSIGNED_CHAR) { APPLY_FUNC(a, b, length, unsigned char, LAND_OP) } else if (*datatype == MPI_C_BOOL) { APPLY_FUNC(a, b, length, bool, LAND_OP) } else { xbt_die("Failed to apply LAND_OP to type %s", (*datatype)->name); } } static void lor_func(void *a, void *b, int *length, MPI_Datatype * datatype) { if (*datatype == MPI_CHAR) { APPLY_FUNC(a, b, length, char, LOR_OP) } else if (*datatype == MPI_SHORT) { APPLY_FUNC(a, b, length, short, LOR_OP) } else if (*datatype == MPI_INT) { APPLY_FUNC(a, b, length, int, LOR_OP) } else if (*datatype == MPI_LONG) { APPLY_FUNC(a, b, length, long, LOR_OP) } else if (*datatype == MPI_UNSIGNED_SHORT) { APPLY_FUNC(a, b, length, unsigned short, LOR_OP) } else if (*datatype == MPI_UNSIGNED) { APPLY_FUNC(a, b, length, unsigned int, LOR_OP) } else if (*datatype == MPI_UNSIGNED_LONG) { APPLY_FUNC(a, b, length, unsigned long, LOR_OP) } else if (*datatype == MPI_UNSIGNED_CHAR) { APPLY_FUNC(a, b, length, unsigned char, LOR_OP) } else if (*datatype == MPI_C_BOOL) { APPLY_FUNC(a, b, length, bool, LOR_OP) } else { xbt_die("Failed to apply LOR_OP to type %s", (*datatype)->name); } } static void lxor_func(void *a, void *b, int *length, MPI_Datatype * datatype) { if (*datatype == MPI_CHAR) { APPLY_FUNC(a, b, length, char, LXOR_OP) } else if (*datatype == MPI_SHORT) { APPLY_FUNC(a, b, length, short, LXOR_OP) } else if (*datatype == MPI_INT) { APPLY_FUNC(a, b, length, int, LXOR_OP) } else if (*datatype == MPI_LONG) { APPLY_FUNC(a, b, length, long, LXOR_OP) } else if (*datatype == MPI_UNSIGNED_SHORT) { APPLY_FUNC(a, b, length, unsigned short, LXOR_OP) } else if (*datatype == MPI_UNSIGNED) { APPLY_FUNC(a, b, length, unsigned int, LXOR_OP) } else if (*datatype == MPI_UNSIGNED_LONG) { APPLY_FUNC(a, b, length, unsigned long, LXOR_OP) } else if (*datatype == MPI_UNSIGNED_CHAR) { APPLY_FUNC(a, b, length, unsigned char, LXOR_OP) } else if (*datatype == MPI_C_BOOL) { APPLY_FUNC(a, b, length, bool, LXOR_OP) } else { xbt_die("Failed to apply LXOR_OP to type %s", (*datatype)->name); } } static void band_func(void *a, void *b, int *length, MPI_Datatype * datatype) { if (*datatype == MPI_CHAR) { APPLY_FUNC(a, b, length, char, BAND_OP) }else if (*datatype == MPI_SHORT) { APPLY_FUNC(a, b, length, short, BAND_OP) } else if (*datatype == MPI_INT) { APPLY_FUNC(a, b, length, int, BAND_OP) } else if (*datatype == MPI_LONG) { APPLY_FUNC(a, b, length, long, BAND_OP) } else if (*datatype == MPI_UNSIGNED_SHORT) { APPLY_FUNC(a, b, length, unsigned short, BAND_OP) } else if (*datatype == MPI_UNSIGNED) { APPLY_FUNC(a, b, length, unsigned int, BAND_OP) } else if (*datatype == MPI_UNSIGNED_LONG) { APPLY_FUNC(a, b, length, unsigned long, BAND_OP) } else if (*datatype == MPI_UNSIGNED_CHAR) { APPLY_FUNC(a, b, length, unsigned char, BAND_OP) } else if (*datatype == MPI_BYTE) { APPLY_FUNC(a, b, length, uint8_t, BAND_OP) } else { xbt_die("Failed to apply BAND_OP to type %s", (*datatype)->name); } } static void bor_func(void *a, void *b, int *length, MPI_Datatype * datatype) { if (*datatype == MPI_CHAR) { APPLY_FUNC(a, b, length, char, BOR_OP) } else if (*datatype == MPI_SHORT) { APPLY_FUNC(a, b, length, short, BOR_OP) } else if (*datatype == MPI_INT) { APPLY_FUNC(a, b, length, int, BOR_OP) } else if (*datatype == MPI_LONG) { APPLY_FUNC(a, b, length, long, BOR_OP) } else if (*datatype == MPI_UNSIGNED_SHORT) { APPLY_FUNC(a, b, length, unsigned short, BOR_OP) } else if (*datatype == MPI_UNSIGNED) { APPLY_FUNC(a, b, length, unsigned int, BOR_OP) } else if (*datatype == MPI_UNSIGNED_LONG) { APPLY_FUNC(a, b, length, unsigned long, BOR_OP) } else if (*datatype == MPI_UNSIGNED_CHAR) { APPLY_FUNC(a, b, length, unsigned char, BOR_OP) } else if (*datatype == MPI_BYTE) { APPLY_FUNC(a, b, length, uint8_t, BOR_OP) } else { xbt_die("Failed to apply BOR_OP to type %s", (*datatype)->name); } } static void bxor_func(void *a, void *b, int *length, MPI_Datatype * datatype) { if (*datatype == MPI_CHAR) { APPLY_FUNC(a, b, length, char, BXOR_OP) } else if (*datatype == MPI_SHORT) { APPLY_FUNC(a, b, length, short, BXOR_OP) } else if (*datatype == MPI_INT) { APPLY_FUNC(a, b, length, int, BXOR_OP) } else if (*datatype == MPI_LONG) { APPLY_FUNC(a, b, length, long, BXOR_OP) } else if (*datatype == MPI_UNSIGNED_SHORT) { APPLY_FUNC(a, b, length, unsigned short, BXOR_OP) } else if (*datatype == MPI_UNSIGNED) { APPLY_FUNC(a, b, length, unsigned int, BXOR_OP) } else if (*datatype == MPI_UNSIGNED_LONG) { APPLY_FUNC(a, b, length, unsigned long, BXOR_OP) } else if (*datatype == MPI_UNSIGNED_CHAR) { APPLY_FUNC(a, b, length, unsigned char, BXOR_OP) } else if (*datatype == MPI_BYTE) { APPLY_FUNC(a, b, length, uint8_t, BXOR_OP) } else { xbt_die("Failed to apply BXOR_OP to type %s", (*datatype)->name); } } static void minloc_func(void *a, void *b, int *length, MPI_Datatype * datatype) { if (*datatype == MPI_FLOAT_INT) { APPLY_FUNC(a, b, length, float_int, MINLOC_OP) } else if (*datatype == MPI_LONG_INT) { APPLY_FUNC(a, b, length, long_int, MINLOC_OP) } else if (*datatype == MPI_DOUBLE_INT) { APPLY_FUNC(a, b, length, double_int, MINLOC_OP) } else if (*datatype == MPI_SHORT_INT) { APPLY_FUNC(a, b, length, short_int, MINLOC_OP) } else if (*datatype == MPI_2LONG) { APPLY_FUNC(a, b, length, long_long, MINLOC_OP) } else if (*datatype == MPI_2INT) { APPLY_FUNC(a, b, length, int_int, MINLOC_OP) } else if (*datatype == MPI_LONG_DOUBLE_INT) { APPLY_FUNC(a, b, length, long_double_int, MINLOC_OP) } else if (*datatype == MPI_2FLOAT) { APPLY_FUNC(a, b, length, float_float, MINLOC_OP) } else if (*datatype == MPI_2DOUBLE) { APPLY_FUNC(a, b, length, double_double, MINLOC_OP) } else { xbt_die("Failed to apply MINLOC_OP to type %s", (*datatype)->name); } } static void maxloc_func(void *a, void *b, int *length, MPI_Datatype * datatype) { if (*datatype == MPI_FLOAT_INT) { APPLY_FUNC(a, b, length, float_int, MAXLOC_OP) } else if (*datatype == MPI_LONG_INT) { APPLY_FUNC(a, b, length, long_int, MAXLOC_OP) } else if (*datatype == MPI_DOUBLE_INT) { APPLY_FUNC(a, b, length, double_int, MAXLOC_OP) } else if (*datatype == MPI_SHORT_INT) { APPLY_FUNC(a, b, length, short_int, MAXLOC_OP) } else if (*datatype == MPI_2LONG) { APPLY_FUNC(a, b, length, long_long, MAXLOC_OP) } else if (*datatype == MPI_2INT) { APPLY_FUNC(a, b, length, int_int, MAXLOC_OP) } else if (*datatype == MPI_LONG_DOUBLE_INT) { APPLY_FUNC(a, b, length, long_double_int, MAXLOC_OP) } else if (*datatype == MPI_2FLOAT) { APPLY_FUNC(a, b, length, float_float, MAXLOC_OP) } else if (*datatype == MPI_2DOUBLE) { APPLY_FUNC(a, b, length, double_double, MAXLOC_OP) } else { xbt_die("Failed to apply MAXLOC_OP to type %s", (*datatype)->name); } } static void replace_func(void *a, void *b, int *length, MPI_Datatype * datatype) { memcpy(b, a, *length * smpi_datatype_size(*datatype)); } #define CREATE_MPI_OP(name, func) \ static s_smpi_mpi_op_t mpi_##name = { &(func) /* func */, true }; \ MPI_Op name = &mpi_##name; CREATE_MPI_OP(MPI_MAX, max_func); CREATE_MPI_OP(MPI_MIN, min_func); CREATE_MPI_OP(MPI_SUM, sum_func); CREATE_MPI_OP(MPI_PROD, prod_func); CREATE_MPI_OP(MPI_LAND, land_func); CREATE_MPI_OP(MPI_LOR, lor_func); CREATE_MPI_OP(MPI_LXOR, lxor_func); CREATE_MPI_OP(MPI_BAND, band_func); CREATE_MPI_OP(MPI_BOR, bor_func); CREATE_MPI_OP(MPI_BXOR, bxor_func); CREATE_MPI_OP(MPI_MAXLOC, maxloc_func); CREATE_MPI_OP(MPI_MINLOC, minloc_func); CREATE_MPI_OP(MPI_REPLACE, replace_func); MPI_Op smpi_op_new(MPI_User_function * function, bool commute) { MPI_Op op; op = xbt_new(s_smpi_mpi_op_t, 1); op->func = function; op-> is_commute = commute; return op; } bool smpi_op_is_commute(MPI_Op op) { return (op==MPI_OP_NULL) ? true : op-> is_commute; } void smpi_op_destroy(MPI_Op op) { xbt_free(op); } void smpi_op_apply(MPI_Op op, const void *invec, void *inoutvec, int *len, MPI_Datatype * datatype) { if(op==MPI_OP_NULL) return; if(smpi_privatize_global_variables){//we need to switch as the called function may silently touch global variables XBT_DEBUG("Applying operation, switch to the right data frame "); smpi_switch_data_segment(smpi_process_index()); } if(!smpi_process_get_replaying()) op->func(const_cast(invec), inoutvec, len, datatype); } int smpi_type_attr_delete(MPI_Datatype type, int keyval){ smpi_type_key_elem elem = static_cast(xbt_dict_get_or_null_ext(smpi_type_keyvals, reinterpret_cast(&keyval), sizeof(int))); if(elem==nullptr) return MPI_ERR_ARG; if(elem->delete_fn!=MPI_NULL_DELETE_FN){ void * value = nullptr; int flag; if(smpi_type_attr_get(type, keyval, &value, &flag)==MPI_SUCCESS){ int ret = elem->delete_fn(type, keyval, value, &flag); if(ret!=MPI_SUCCESS) return ret; } } if(type->attributes==nullptr) return MPI_ERR_ARG; xbt_dict_remove_ext(type->attributes, reinterpret_cast(&keyval), sizeof(int)); return MPI_SUCCESS; } int smpi_type_attr_get(MPI_Datatype type, int keyval, void* attr_value, int* flag){ smpi_type_key_elem elem = static_cast(xbt_dict_get_or_null_ext(smpi_type_keyvals, reinterpret_cast(&keyval), sizeof(int))); if(elem==nullptr) return MPI_ERR_ARG; if(type->attributes==nullptr){ *flag=0; return MPI_SUCCESS; } try { *static_cast(attr_value) = xbt_dict_get_ext(type->attributes, reinterpret_cast(&keyval), sizeof(int)); *flag=1; } catch (xbt_ex& ex) { *flag=0; } return MPI_SUCCESS; } int smpi_type_attr_put(MPI_Datatype type, int keyval, void* attr_value){ if(smpi_type_keyvals==nullptr) smpi_type_keyvals = xbt_dict_new_homogeneous(nullptr); smpi_type_key_elem elem = static_cast(xbt_dict_get_or_null_ext(smpi_type_keyvals, reinterpret_cast(&keyval), sizeof(int))); if(elem==nullptr) return MPI_ERR_ARG; int flag; void* value = nullptr; smpi_type_attr_get(type, keyval, &value, &flag); if(flag!=0 && elem->delete_fn!=MPI_NULL_DELETE_FN){ int ret = elem->delete_fn(type, keyval, value, &flag); if(ret!=MPI_SUCCESS) return ret; } if(type->attributes==nullptr) type->attributes = xbt_dict_new_homogeneous(nullptr); xbt_dict_set_ext(type->attributes, reinterpret_cast(&keyval), sizeof(int), attr_value, nullptr); return MPI_SUCCESS; } int smpi_type_keyval_create(MPI_Type_copy_attr_function* copy_fn, MPI_Type_delete_attr_function* delete_fn, int* keyval, void* extra_state){ if(smpi_type_keyvals==nullptr) smpi_type_keyvals = xbt_dict_new_homogeneous(nullptr); smpi_type_key_elem value = (smpi_type_key_elem) xbt_new0(s_smpi_mpi_type_key_elem_t,1); value->copy_fn=copy_fn; value->delete_fn=delete_fn; *keyval = type_keyval_id; xbt_dict_set_ext(smpi_type_keyvals,reinterpret_cast(keyval), sizeof(int),reinterpret_cast(value), nullptr); type_keyval_id++; return MPI_SUCCESS; } int smpi_type_keyval_free(int* keyval){ smpi_type_key_elem elem = static_cast(xbt_dict_get_or_null_ext(smpi_type_keyvals, reinterpret_cast(keyval), sizeof(int))); if(elem==0){ return MPI_ERR_ARG; } xbt_dict_remove_ext(smpi_type_keyvals, reinterpret_cast(keyval), sizeof(int)); xbt_free(elem); return MPI_SUCCESS; } int smpi_mpi_pack(void* inbuf, int incount, MPI_Datatype type, void* outbuf, int outcount, int* position,MPI_Comm comm){ size_t size = smpi_datatype_size(type); if (outcount - *position < incount*static_cast(size)) return MPI_ERR_BUFFER; smpi_datatype_copy(inbuf, incount, type, static_cast(outbuf) + *position, outcount, MPI_CHAR); *position += incount * size; return MPI_SUCCESS; } int smpi_mpi_unpack(void* inbuf, int insize, int* position, void* outbuf, int outcount, MPI_Datatype type,MPI_Comm comm){ int size = static_cast(smpi_datatype_size(type)); if (outcount*size> insize) return MPI_ERR_BUFFER; smpi_datatype_copy(static_cast(inbuf) + *position, insize, MPI_CHAR, outbuf, outcount, type); *position += outcount * size; return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/smpi_global.cpp000644 001750 001750 00000072541 13031031272 021634 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.h" #include "private.hpp" #include "smpi_mpi_dt_private.h" #include "mc/mc.h" #include "src/mc/mc_record.h" #include "xbt/replay.h" #include "surf/surf.h" #include "src/simix/smx_private.h" #include "simgrid/sg_config.h" #include "src/mc/mc_replay.h" #include "src/msg/msg_private.h" #include "src/kernel/activity/SynchroComm.hpp" #include /* DBL_MAX */ #include #include #include #include #include #include #include XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_kernel, smpi, "Logging specific to SMPI (kernel)"); #include #include /* trim_right / trim_left */ #if HAVE_PAPI #include "papi.h" const char* papi_default_config_name = "default"; struct papi_process_data { papi_counter_t counter_data; int event_set; }; #endif std::unordered_map location2speedup; typedef struct s_smpi_process_data { double simulated; int *argc; char ***argv; smx_mailbox_t mailbox; smx_mailbox_t mailbox_small; xbt_mutex_t mailboxes_mutex; xbt_os_timer_t timer; MPI_Comm comm_self; MPI_Comm comm_intra; MPI_Comm* comm_world; void *data; /* user data */ int index; char state; int sampling; /* inside an SMPI_SAMPLE_ block? */ char* instance_id; bool replaying; /* is the process replaying a trace */ msg_bar_t finalization_barrier; int return_value; smpi_trace_call_location_t trace_call_loc; #if HAVE_PAPI /** Contains hardware data as read by PAPI **/ int papi_event_set; papi_counter_t papi_counter_data; #endif } s_smpi_process_data_t; static smpi_process_data_t *process_data = nullptr; int process_count = 0; int smpi_universe_size = 0; int* index_to_process_data = nullptr; extern double smpi_total_benched_time; extern xbt_dict_t smpi_type_keyvals; extern xbt_dict_t smpi_comm_keyvals; xbt_os_timer_t global_timer; MPI_Comm MPI_COMM_WORLD = MPI_COMM_UNINITIALIZED; MPI_Errhandler *MPI_ERRORS_RETURN = nullptr; MPI_Errhandler *MPI_ERRORS_ARE_FATAL = nullptr; MPI_Errhandler *MPI_ERRHANDLER_NULL = nullptr; #define MAILBOX_NAME_MAXLEN (5 + sizeof(int) * 2 + 1) static char *get_mailbox_name(char *str, int index) { snprintf(str, MAILBOX_NAME_MAXLEN, "SMPI-%0*x", static_cast (sizeof(int) * 2), index); return str; } static char *get_mailbox_name_small(char *str, int index) { snprintf(str, MAILBOX_NAME_MAXLEN, "small%0*x", static_cast (sizeof(int) * 2), index); return str; } void smpi_process_init(int *argc, char ***argv) { if (process_data == nullptr){ printf("SimGrid was not initialized properly before entering MPI_Init. Aborting, please check compilation process and use smpirun\n"); exit(1); } if (argc != nullptr && argv != nullptr) { smx_actor_t proc = SIMIX_process_self(); proc->context->set_cleanup(&MSG_process_cleanup_from_SIMIX); char* instance_id = (*argv)[1]; int rank = xbt_str_parse_int((*argv)[2], "Invalid rank: %s"); int index = smpi_process_index_of_smx_process(proc); if(index_to_process_data == nullptr){ index_to_process_data=static_cast(xbt_malloc(SIMIX_process_count()*sizeof(int))); } if(smpi_privatize_global_variables){ /* Now using segment index of the process */ index = proc->segment_index; /* Done at the process's creation */ SMPI_switch_data_segment(index); } MPI_Comm* temp_comm_world; msg_bar_t temp_bar; smpi_deployment_register_process(instance_id, rank, index, &temp_comm_world, &temp_bar); smpi_process_data_t data = smpi_process_remote_data(index); data->comm_world = temp_comm_world; if(temp_bar != nullptr) data->finalization_barrier = temp_bar; data->index = index; data->instance_id = instance_id; data->replaying = false; simdata_process_t simdata = static_cast(simcall_process_get_data(proc)); simdata->data = data; if (*argc > 3) { memmove(&(*argv)[0], &(*argv)[2], sizeof(char *) * (*argc - 2)); (*argv)[(*argc) - 1] = nullptr; (*argv)[(*argc) - 2] = nullptr; } (*argc)-=2; data->argc = argc; data->argv = argv; // set the process attached to the mailbox simcall_mbox_set_receiver(data->mailbox_small, proc); XBT_DEBUG("<%d> New process in the game: %p", index, proc); } xbt_assert(smpi_process_data(), "smpi_process_data() returned nullptr. You probably gave a nullptr parameter to MPI_Init. " "Although it's required by MPI-2, this is currently not supported by SMPI."); } void smpi_process_destroy() { int index = smpi_process_index(); if(smpi_privatize_global_variables){ smpi_switch_data_segment(index); } process_data[index_to_process_data[index]]->state = SMPI_FINALIZED; XBT_DEBUG("<%d> Process left the game", index); } /** @brief Prepares the current process for termination. */ void smpi_process_finalize() { // This leads to an explosion of the search graph which cannot be reduced: if(MC_is_active() || MC_record_replay_is_active()) return; int index = smpi_process_index(); // wait for all pending asynchronous comms to finish MSG_barrier_wait(process_data[index_to_process_data[index]]->finalization_barrier); } /** @brief Check if a process is finalized */ int smpi_process_finalized() { int index = smpi_process_index(); if (index != MPI_UNDEFINED) return (process_data[index_to_process_data[index]]->state == SMPI_FINALIZED); else return 0; } /** @brief Check if a process is initialized */ int smpi_process_initialized() { if (index_to_process_data == nullptr){ return false; } else{ int index = smpi_process_index(); return ((index != MPI_UNDEFINED) && (process_data[index_to_process_data[index]]->state == SMPI_INITIALIZED)); } } /** @brief Mark a process as initialized (=MPI_Init called) */ void smpi_process_mark_as_initialized() { int index = smpi_process_index(); if ((index != MPI_UNDEFINED) && (process_data[index_to_process_data[index]]->state != SMPI_FINALIZED)) process_data[index_to_process_data[index]]->state = SMPI_INITIALIZED; } void smpi_process_set_replaying(bool value){ int index = smpi_process_index(); if ((index != MPI_UNDEFINED) && (process_data[index_to_process_data[index]]->state != SMPI_FINALIZED)) process_data[index_to_process_data[index]]->replaying = value; } bool smpi_process_get_replaying(){ int index = smpi_process_index(); if (index != MPI_UNDEFINED) return process_data[index_to_process_data[index]]->replaying; else return (_xbt_replay_is_active() != 0); } int smpi_global_size() { char *value = getenv("SMPI_GLOBAL_SIZE"); xbt_assert(value,"Please set env var SMPI_GLOBAL_SIZE to the expected number of processes."); return xbt_str_parse_int(value, "SMPI_GLOBAL_SIZE contains a non-numerical value: %s"); } smpi_process_data_t smpi_process_data() { simdata_process_t simdata = static_cast(SIMIX_process_self_get_data()); return static_cast(simdata->data); } smpi_process_data_t smpi_process_remote_data(int index) { return process_data[index_to_process_data[index]]; } void smpi_process_set_user_data(void *data) { smpi_process_data_t process_data = smpi_process_data(); process_data->data = data; } void *smpi_process_get_user_data() { smpi_process_data_t process_data = smpi_process_data(); return process_data->data; } int smpi_process_count() { return process_count; } /** * \brief Returns a structure that stores the location (filename + linenumber) * of the last calls to MPI_* functions. * * \see smpi_trace_set_call_location */ smpi_trace_call_location_t* smpi_process_get_call_location() { smpi_process_data_t process_data = smpi_process_data(); return &process_data->trace_call_loc; } int smpi_process_index() { smpi_process_data_t data = smpi_process_data(); //return -1 if not initialized return data != nullptr ? data->index : MPI_UNDEFINED; } MPI_Comm smpi_process_comm_world() { smpi_process_data_t data = smpi_process_data(); //return MPI_COMM_NULL if not initialized return data != nullptr ? *data->comm_world : MPI_COMM_NULL; } smx_mailbox_t smpi_process_mailbox() { smpi_process_data_t data = smpi_process_data(); return data->mailbox; } smx_mailbox_t smpi_process_mailbox_small() { smpi_process_data_t data = smpi_process_data(); return data->mailbox_small; } xbt_mutex_t smpi_process_mailboxes_mutex() { smpi_process_data_t data = smpi_process_data(); return data->mailboxes_mutex; } smx_mailbox_t smpi_process_remote_mailbox(int index) { smpi_process_data_t data = smpi_process_remote_data(index); return data->mailbox; } smx_mailbox_t smpi_process_remote_mailbox_small(int index) { smpi_process_data_t data = smpi_process_remote_data(index); return data->mailbox_small; } xbt_mutex_t smpi_process_remote_mailboxes_mutex(int index) { smpi_process_data_t data = smpi_process_remote_data(index); return data->mailboxes_mutex; } #if HAVE_PAPI int smpi_process_papi_event_set(void) { smpi_process_data_t data = smpi_process_data(); return data->papi_event_set; } papi_counter_t& smpi_process_papi_counters(void) { smpi_process_data_t data = smpi_process_data(); return data->papi_counter_data; } #endif xbt_os_timer_t smpi_process_timer() { smpi_process_data_t data = smpi_process_data(); return data->timer; } void smpi_process_simulated_start() { smpi_process_data_t data = smpi_process_data(); data->simulated = SIMIX_get_clock(); } double smpi_process_simulated_elapsed() { smpi_process_data_t data = smpi_process_data(); return SIMIX_get_clock() - data->simulated; } MPI_Comm smpi_process_comm_self() { smpi_process_data_t data = smpi_process_data(); if(data->comm_self==MPI_COMM_NULL){ MPI_Group group = smpi_group_new(1); data->comm_self = smpi_comm_new(group, nullptr); smpi_group_set_mapping(group, smpi_process_index(), 0); } return data->comm_self; } MPI_Comm smpi_process_get_comm_intra() { smpi_process_data_t data = smpi_process_data(); return data->comm_intra; } void smpi_process_set_comm_intra(MPI_Comm comm) { smpi_process_data_t data = smpi_process_data(); data->comm_intra = comm; } void smpi_process_set_sampling(int s) { smpi_process_data_t data = smpi_process_data(); data->sampling = s; } int smpi_process_get_sampling() { smpi_process_data_t data = smpi_process_data(); return data->sampling; } void print_request(const char *message, MPI_Request request) { XBT_VERB("%s request %p [buf = %p, size = %zu, src = %d, dst = %d, tag = %d, flags = %x]", message, request, request->buf, request->size, request->src, request->dst, request->tag, request->flags); } void smpi_comm_copy_buffer_callback(smx_activity_t synchro, void *buff, size_t buff_size) { XBT_DEBUG("Copy the data over"); void* tmpbuff=buff; simgrid::kernel::activity::Comm *comm = dynamic_cast(synchro); if((smpi_privatize_global_variables) && (static_cast(buff) >= smpi_start_data_exe) && (static_cast(buff) < smpi_start_data_exe + smpi_size_data_exe ) ){ XBT_DEBUG("Privatization : We are copying from a zone inside global memory... Saving data to temp buffer !"); smpi_switch_data_segment((static_cast((static_cast(SIMIX_process_get_data(comm->src_proc))->data))->index)); tmpbuff = static_cast(xbt_malloc(buff_size)); memcpy(tmpbuff, buff, buff_size); } if((smpi_privatize_global_variables) && ((char*)comm->dst_buff >= smpi_start_data_exe) && ((char*)comm->dst_buff < smpi_start_data_exe + smpi_size_data_exe )){ XBT_DEBUG("Privatization : We are copying to a zone inside global memory - Switch data segment"); smpi_switch_data_segment((static_cast((static_cast(SIMIX_process_get_data(comm->dst_proc))->data))->index)); } memcpy(comm->dst_buff, tmpbuff, buff_size); if (comm->detached) { // if this is a detached send, the source buffer was duplicated by SMPI // sender to make the original buffer available to the application ASAP xbt_free(buff); //It seems that the request is used after the call there this should be free somewhere else but where??? //xbt_free(comm->comm.src_data);// inside SMPI the request is kept inside the user data and should be free comm->src_buff = nullptr; } if(tmpbuff!=buff)xbt_free(tmpbuff); } void smpi_comm_null_copy_buffer_callback(smx_activity_t comm, void *buff, size_t buff_size) { return; } static void smpi_check_options(){ //check correctness of MPI parameters xbt_assert(xbt_cfg_get_int("smpi/async-small-thresh") <= xbt_cfg_get_int("smpi/send-is-detached-thresh")); if (xbt_cfg_is_default_value("smpi/host-speed")) { XBT_INFO("You did not set the power of the host running the simulation. " "The timings will certainly not be accurate. " "Use the option \"--cfg=smpi/host-speed:\" to set its value." "Check http://simgrid.org/simgrid/latest/doc/options.html#options_smpi_bench for more information."); } xbt_assert(xbt_cfg_get_double("smpi/cpu-threshold") >=0, "The 'smpi/cpu-threshold' option cannot have negative values [anymore]. If you want to discard " "the simulation of any computation, please use 'smpi/simulate-computation:no' instead."); } int smpi_enabled() { return process_data != nullptr; } void smpi_global_init() { int i; MPI_Group group; char name[MAILBOX_NAME_MAXLEN]; int smpirun=0; if (!MC_is_active()) { global_timer = xbt_os_timer_new(); xbt_os_walltimer_start(global_timer); } if (xbt_cfg_get_string("smpi/comp-adjustment-file")[0] != '\0') { std::string filename {xbt_cfg_get_string("smpi/comp-adjustment-file")}; std::ifstream fstream(filename); if (!fstream.is_open()) { xbt_die("Could not open file %s. Does it exist?", filename.c_str()); } std::string line; typedef boost::tokenizer< boost::escaped_list_separator> Tokenizer; std::getline(fstream, line); // Skip the header line while (std::getline(fstream, line)) { Tokenizer tok(line); Tokenizer::iterator it = tok.begin(); Tokenizer::iterator end = std::next(tok.begin()); std::string location = *it; boost::trim(location); location2speedup.insert(std::pair(location, std::stod(*end))); } } #if HAVE_PAPI // This map holds for each computation unit (such as "default" or "process1" etc.) // the configuration as given by the user (counter data as a pair of (counter_name, counter_counter)) // and the (computed) event_set. std::map units2papi_setup; if (xbt_cfg_get_string("smpi/papi-events")[0] != '\0') { if (PAPI_library_init(PAPI_VER_CURRENT) != PAPI_VER_CURRENT) XBT_ERROR("Could not initialize PAPI library; is it correctly installed and linked?" " Expected version is %i", PAPI_VER_CURRENT); typedef boost::tokenizer> Tokenizer; boost::char_separator separator_units(";"); std::string str = std::string(xbt_cfg_get_string("smpi/papi-events")); Tokenizer tokens(str, separator_units); // Iterate over all the computational units. This could be // processes, hosts, threads, ranks... You name it. I'm not exactly // sure what we will support eventually, so I'll leave it at the // general term "units". for (auto& unit_it : tokens) { boost::char_separator separator_events(":"); Tokenizer event_tokens(unit_it, separator_events); int event_set = PAPI_NULL; if (PAPI_create_eventset(&event_set) != PAPI_OK) { // TODO: Should this let the whole simulation die? XBT_CRITICAL("Could not create PAPI event set during init."); } // NOTE: We cannot use a map here, as we must obey the order of the counters // This is important for PAPI: We need to map the values of counters back // to the event_names (so, when PAPI_read() has finished)! papi_counter_t counters2values; // Iterate over all counters that were specified for this specific // unit. // Note that we need to remove the name of the unit // (that could also be the "default" value), which always comes first. // Hence, we start at ++(events.begin())! for (Tokenizer::iterator events_it = ++(event_tokens.begin()); events_it != event_tokens.end(); events_it++) { int event_code = PAPI_NULL; char* event_name = const_cast((*events_it).c_str()); if (PAPI_event_name_to_code(event_name, &event_code) == PAPI_OK) { if (PAPI_add_event(event_set, event_code) != PAPI_OK) { XBT_ERROR("Could not add PAPI event '%s'. Skipping.", event_name); continue; } else { XBT_DEBUG("Successfully added PAPI event '%s' to the event set.", event_name); } } else { XBT_CRITICAL("Could not find PAPI event '%s'. Skipping.", event_name); continue; } counters2values.push_back( // We cannot just pass *events_it, as this is of type const basic_string std::make_pair(std::string(*events_it), 0)); } std::string unit_name = *(event_tokens.begin()); papi_process_data config = {.counter_data = std::move(counters2values), .event_set = event_set}; units2papi_setup.insert(std::make_pair(unit_name, std::move(config))); } } #endif if (process_count == 0){ process_count = SIMIX_process_count(); smpirun=1; } smpi_universe_size = process_count; process_data = new smpi_process_data_t[process_count]; for (i = 0; i < process_count; i++) { process_data[i] = new s_smpi_process_data_t; process_data[i]->argc = nullptr; process_data[i]->argv = nullptr; process_data[i]->mailbox = simcall_mbox_create(get_mailbox_name(name, i)); process_data[i]->mailbox_small = simcall_mbox_create(get_mailbox_name_small(name, i)); process_data[i]->mailboxes_mutex = xbt_mutex_init(); process_data[i]->timer = xbt_os_timer_new(); if (MC_is_active()) MC_ignore_heap(process_data[i]->timer, xbt_os_timer_size()); process_data[i]->comm_self = MPI_COMM_NULL; process_data[i]->comm_intra = MPI_COMM_NULL; process_data[i]->comm_world = nullptr; process_data[i]->state = SMPI_UNINITIALIZED; process_data[i]->sampling = 0; process_data[i]->finalization_barrier = nullptr; process_data[i]->return_value = 0; #if HAVE_PAPI if (xbt_cfg_get_string("smpi/papi-events")[0] != '\0') { // TODO: Implement host/process/thread based counters. This implementation // just always takes the values passed via "default", like this: // "default:COUNTER1:COUNTER2:COUNTER3;". auto it = units2papi_setup.find(papi_default_config_name); if (it != units2papi_setup.end()) { process_data[i]->papi_event_set = it->second.event_set; process_data[i]->papi_counter_data = it->second.counter_data; XBT_DEBUG("Setting PAPI set for process %i", i); } else { process_data[i]->papi_event_set = PAPI_NULL; XBT_DEBUG("No PAPI set for process %i", i); } } #endif } //if the process was launched through smpirun script we generate a global mpi_comm_world //if not, we let MPI_COMM_NULL, and the comm world will be private to each mpi instance if(smpirun){ group = smpi_group_new(process_count); MPI_COMM_WORLD = smpi_comm_new(group, nullptr); MPI_Attr_put(MPI_COMM_WORLD, MPI_UNIVERSE_SIZE, reinterpret_cast(process_count)); msg_bar_t bar = MSG_barrier_init(process_count); for (i = 0; i < process_count; i++) { smpi_group_set_mapping(group, i, i); process_data[i]->finalization_barrier = bar; } } } void smpi_global_destroy() { int count = smpi_process_count(); smpi_bench_destroy(); if (MPI_COMM_WORLD != MPI_COMM_UNINITIALIZED){ while (smpi_group_unuse(smpi_comm_group(MPI_COMM_WORLD)) > 0); MSG_barrier_destroy(process_data[0]->finalization_barrier); }else{ smpi_deployment_cleanup_instances(); } for (int i = 0; i < count; i++) { if(process_data[i]->comm_self!=MPI_COMM_NULL){ smpi_comm_destroy(process_data[i]->comm_self); } if(process_data[i]->comm_intra!=MPI_COMM_NULL){ smpi_comm_destroy(process_data[i]->comm_intra); } xbt_os_timer_free(process_data[i]->timer); xbt_mutex_destroy(process_data[i]->mailboxes_mutex); delete process_data[i]; } delete[] process_data; process_data = nullptr; if (MPI_COMM_WORLD != MPI_COMM_UNINITIALIZED){ smpi_comm_cleanup_smp(MPI_COMM_WORLD); smpi_comm_cleanup_attributes(MPI_COMM_WORLD); if(smpi_coll_cleanup_callback!=nullptr) smpi_coll_cleanup_callback(); xbt_free(MPI_COMM_WORLD); } MPI_COMM_WORLD = MPI_COMM_NULL; if (!MC_is_active()) { xbt_os_timer_free(global_timer); } xbt_free(index_to_process_data); if(smpi_type_keyvals!=nullptr) xbt_dict_free(&smpi_type_keyvals); if(smpi_comm_keyvals!=nullptr) xbt_dict_free(&smpi_comm_keyvals); if(smpi_privatize_global_variables) smpi_destroy_global_memory_segments(); smpi_free_static(); } #ifndef WIN32 void __attribute__ ((weak)) user_main_() { xbt_die("Should not be in this smpi_simulated_main"); return; } int __attribute__ ((weak)) smpi_simulated_main_(int argc, char **argv) { smpi_process_init(&argc, &argv); user_main_(); return 0; } inline static int smpi_main_wrapper(int argc, char **argv){ int ret = smpi_simulated_main_(argc,argv); if(ret !=0){ XBT_WARN("SMPI process did not return 0. Return value : %d", ret); smpi_process_data()->return_value=ret; } return 0; } int __attribute__ ((weak)) main(int argc, char **argv) { return smpi_main(smpi_main_wrapper, argc, argv); } #endif extern "C" { static void smpi_init_logs(){ /* Connect log categories. See xbt/log.c */ XBT_LOG_CONNECT(smpi); /* Keep this line as soon as possible in this function: xbt_log_appender_file.c depends on it DO NOT connect this in XBT or so, or it will be useless to xbt_log_appender_file.c */ XBT_LOG_CONNECT(instr_smpi); XBT_LOG_CONNECT(smpi_base); XBT_LOG_CONNECT(smpi_bench); XBT_LOG_CONNECT(smpi_coll); XBT_LOG_CONNECT(smpi_colls); XBT_LOG_CONNECT(smpi_comm); XBT_LOG_CONNECT(smpi_dvfs); XBT_LOG_CONNECT(smpi_group); XBT_LOG_CONNECT(smpi_kernel); XBT_LOG_CONNECT(smpi_mpi); XBT_LOG_CONNECT(smpi_mpi_dt); XBT_LOG_CONNECT(smpi_pmpi); XBT_LOG_CONNECT(smpi_replay); XBT_LOG_CONNECT(smpi_rma); } } static void smpi_init_options(){ int gather_id = find_coll_description(mpi_coll_gather_description, xbt_cfg_get_string("smpi/gather"),"gather"); mpi_coll_gather_fun = reinterpret_cast (mpi_coll_gather_description[gather_id].coll); int allgather_id = find_coll_description(mpi_coll_allgather_description, xbt_cfg_get_string("smpi/allgather"),"allgather"); mpi_coll_allgather_fun = reinterpret_cast (mpi_coll_allgather_description[allgather_id].coll); int allgatherv_id = find_coll_description(mpi_coll_allgatherv_description, xbt_cfg_get_string("smpi/allgatherv"),"allgatherv"); mpi_coll_allgatherv_fun = reinterpret_cast (mpi_coll_allgatherv_description[allgatherv_id].coll); int allreduce_id = find_coll_description(mpi_coll_allreduce_description, xbt_cfg_get_string("smpi/allreduce"),"allreduce"); mpi_coll_allreduce_fun = reinterpret_cast (mpi_coll_allreduce_description[allreduce_id].coll); int alltoall_id = find_coll_description(mpi_coll_alltoall_description, xbt_cfg_get_string("smpi/alltoall"),"alltoall"); mpi_coll_alltoall_fun = reinterpret_cast (mpi_coll_alltoall_description[alltoall_id].coll); int alltoallv_id = find_coll_description(mpi_coll_alltoallv_description, xbt_cfg_get_string("smpi/alltoallv"),"alltoallv"); mpi_coll_alltoallv_fun = reinterpret_cast (mpi_coll_alltoallv_description[alltoallv_id].coll); int bcast_id = find_coll_description(mpi_coll_bcast_description, xbt_cfg_get_string("smpi/bcast"),"bcast"); mpi_coll_bcast_fun = reinterpret_cast (mpi_coll_bcast_description[bcast_id].coll); int reduce_id = find_coll_description(mpi_coll_reduce_description, xbt_cfg_get_string("smpi/reduce"),"reduce"); mpi_coll_reduce_fun = reinterpret_cast (mpi_coll_reduce_description[reduce_id].coll); int reduce_scatter_id = find_coll_description(mpi_coll_reduce_scatter_description, xbt_cfg_get_string("smpi/reduce-scatter"),"reduce_scatter"); mpi_coll_reduce_scatter_fun = reinterpret_cast (mpi_coll_reduce_scatter_description[reduce_scatter_id].coll); int scatter_id = find_coll_description(mpi_coll_scatter_description, xbt_cfg_get_string("smpi/scatter"),"scatter"); mpi_coll_scatter_fun = reinterpret_cast (mpi_coll_scatter_description[scatter_id].coll); int barrier_id = find_coll_description(mpi_coll_barrier_description, xbt_cfg_get_string("smpi/barrier"),"barrier"); mpi_coll_barrier_fun = reinterpret_cast (mpi_coll_barrier_description[barrier_id].coll); smpi_coll_cleanup_callback=nullptr; smpi_cpu_threshold = xbt_cfg_get_double("smpi/cpu-threshold"); smpi_host_speed = xbt_cfg_get_double("smpi/host-speed"); smpi_privatize_global_variables = xbt_cfg_get_boolean("smpi/privatize-global-variables"); if (smpi_cpu_threshold < 0) smpi_cpu_threshold = DBL_MAX; } int smpi_main(int (*realmain) (int argc, char *argv[]), int argc, char *argv[]) { srand(SMPI_RAND_SEED); if (getenv("SMPI_PRETEND_CC") != nullptr) { /* Hack to ensure that smpicc can pretend to be a simple compiler. Particularly handy to pass it to the * configuration tools */ return 0; } smpi_init_logs(); TRACE_global_init(&argc, argv); TRACE_add_start_function(TRACE_smpi_alloc); TRACE_add_end_function(TRACE_smpi_release); SIMIX_global_init(&argc, argv); MSG_init(&argc,argv); SMPI_switch_data_segment = smpi_switch_data_segment; smpi_init_options(); // parse the platform file: get the host list SIMIX_create_environment(argv[1]); SIMIX_comm_set_copy_data_callback(&smpi_comm_copy_buffer_callback); SIMIX_function_register_default(realmain); SIMIX_launch_application(argv[2]); smpi_global_init(); smpi_check_options(); if(smpi_privatize_global_variables) smpi_initialize_global_memory_segments(); /* Clean IO before the run */ fflush(stdout); fflush(stderr); if (MC_is_active()) { MC_run(); } else { SIMIX_run(); xbt_os_walltimer_stop(global_timer); if (xbt_cfg_get_boolean("smpi/display-timing")){ double global_time = xbt_os_timer_elapsed(global_timer); XBT_INFO("Simulated time: %g seconds. \n\n" "The simulation took %g seconds (after parsing and platform setup)\n" "%g seconds were actual computation of the application", SIMIX_get_clock(), global_time , smpi_total_benched_time); if (smpi_total_benched_time/global_time>=0.75) XBT_INFO("More than 75%% of the time was spent inside the application code.\n" "You may want to use sampling functions or trace replay to reduce this."); } } int count = smpi_process_count(); int i, ret=0; for (i = 0; i < count; i++) { if(process_data[i]->return_value!=0){ ret=process_data[i]->return_value;//return first non 0 value break; } } smpi_global_destroy(); TRACE_end(); return ret; } // This function can be called from extern file, to initialize logs, options, and processes of smpi // without the need of smpirun void SMPI_init(){ smpi_init_logs(); smpi_init_options(); smpi_global_init(); smpi_check_options(); if (TRACE_is_enabled() && TRACE_is_configured()) TRACE_smpi_alloc(); if(smpi_privatize_global_variables) smpi_initialize_global_memory_segments(); } void SMPI_finalize(){ smpi_global_destroy(); } SimGrid-3.14.159/src/smpi/smpi_bench.cpp000644 001750 001750 00000070055 13031031272 021451 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007, 2009-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include "src/internal_config.h" #include "private.h" #include "private.hpp" #include #include "xbt/dict.h" #include "xbt/sysdep.h" #include "xbt/ex.h" #include "surf/surf.h" #include "simgrid/sg_config.h" #include "simgrid/modelchecker.h" #include "src/mc/mc_replay.h" #include #ifndef WIN32 #include #endif #include #include #include #include // sqrt #include #include #include #if HAVE_PAPI #include #endif #ifndef MAP_ANONYMOUS #define MAP_ANONYMOUS MAP_ANON #endif XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_bench, smpi, "Logging specific to SMPI (benchmarking)"); /* Shared allocations are handled through shared memory segments. * Associated data and metadata are used as follows: * * mmap #1 * `allocs' dict ---- -. * ---------- shared_data_t shared_metadata_t / | | | * .->| | ---> -------------------- <--. ----------------- | | | | * | ---------- | fd of | | | size of mmap | --| | | | * | | count (2) | |-- | data | \ | | | * `----------------- | | | ----------------- ---- | * -------------------- | ^ | * | | | * | | `allocs_metadata' dict | * | | ---------------------- | * | `-- | |<-' * | .-- | |<-. * | | ---------------------- | * | | | * | | | * | | | * | | mmap #2 | * | v ---- -' * | shared_metadata_t / | | * | ----------------- | | | * | | size of mmap | --| | | * `-- | data | | | | * ----------------- | | | * \ | | * ---- */ #define PTR_STRLEN (2 + 2 * sizeof(void*) + 1) xbt_dict_t samples = nullptr; /* Allocated on first use */ xbt_dict_t calls = nullptr; /* Allocated on first use */ double smpi_cpu_threshold; double smpi_host_speed; int smpi_loaded_page = -1; char* smpi_start_data_exe = nullptr; int smpi_size_data_exe = 0; bool smpi_privatize_global_variables; double smpi_total_benched_time = 0; smpi_privatisation_region_t smpi_privatisation_regions; namespace { /** Some location in the source code * * This information is used by SMPI_SHARED_MALLOC to allocate some shared memory for all simulated processes. */ class smpi_source_location { public: smpi_source_location(const char* filename, int line) : filename(xbt_strdup(filename)), filename_length(strlen(filename)), line(line) {} /** Pointer to a static string containing the file name */ char* filename = nullptr; int filename_length = 0; int line = 0; bool operator==(smpi_source_location const& that) const { return filename_length == that.filename_length && line == that.line && std::memcmp(filename, that.filename, filename_length) == 0; } bool operator!=(smpi_source_location const& that) const { return !(*this == that); } }; } namespace std { template<> class hash { public: typedef smpi_source_location argument_type; typedef std::size_t result_type; result_type operator()(smpi_source_location const& loc) const { return xbt_str_hash_ext(loc.filename, loc.filename_length) ^ xbt_str_hash_ext((const char*) &loc.line, sizeof(loc.line)); } }; } namespace { typedef struct { int fd = -1; int count = 0; } shared_data_t; std::unordered_map allocs; typedef std::unordered_map::value_type shared_data_key_type; typedef struct { size_t size; shared_data_key_type* data; } shared_metadata_t; std::unordered_map allocs_metadata; } static size_t shm_size(int fd) { struct stat st; if(fstat(fd, &st) < 0) { xbt_die("Could not stat fd %d: %s", fd, strerror(errno)); } return static_cast(st.st_size); } #ifndef WIN32 static void* shm_map(int fd, size_t size, shared_data_key_type* data) { void* mem; char loc[PTR_STRLEN]; shared_metadata_t meta; if(size > shm_size(fd) && (ftruncate(fd, static_cast(size)) < 0)) { xbt_die("Could not truncate fd %d to %zu: %s", fd, size, strerror(errno)); } mem = mmap(nullptr, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); if(mem == MAP_FAILED) { xbt_die("Could not map fd %d with size %zu: %s.\n If you are running a lot of processes, you may be exceeding the amount of mappings allowed per process. \n On linux systems, this value can be set by using sudo sysctl -w vm.max_map_count=newvalue .\n Default value is 65536", fd, size, strerror(errno)); } snprintf(loc, PTR_STRLEN, "%p", mem); meta.size = size; meta.data = data; allocs_metadata[mem] = meta; XBT_DEBUG("MMAP %zu to %p", size, mem); return mem; } #endif void smpi_bench_destroy() { allocs.clear(); allocs_metadata.clear(); xbt_dict_free(&samples); xbt_dict_free(&calls); } extern "C" XBT_PUBLIC(void) smpi_execute_flops_(double *flops); void smpi_execute_flops_(double *flops) { smpi_execute_flops(*flops); } extern "C" XBT_PUBLIC(void) smpi_execute_(double *duration); void smpi_execute_(double *duration) { smpi_execute(*duration); } void smpi_execute_flops(double flops) { XBT_DEBUG("Handle real computation time: %f flops", flops); smx_activity_t action = simcall_execution_start("computation", flops, 1, 0); simcall_set_category (action, TRACE_internal_smpi_get_category()); simcall_execution_wait(action); smpi_switch_data_segment(smpi_process_index()); } void smpi_execute(double duration) { if (duration >= smpi_cpu_threshold) { XBT_DEBUG("Sleep for %g to handle real computation time", duration); double flops = duration * smpi_host_speed; int rank = smpi_process_index(); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type=TRACING_COMPUTING; extra->comp_size=flops; TRACE_smpi_computing_in(rank, extra); smpi_execute_flops(flops); TRACE_smpi_computing_out(rank); } else { XBT_DEBUG("Real computation took %g while option smpi/cpu_threshold is set to %g => ignore it", duration, smpi_cpu_threshold); } } void smpi_bench_begin() { if (smpi_privatize_global_variables) { smpi_switch_data_segment(smpi_process_index()); } if (MC_is_active() || MC_record_replay_is_active()) return; #if HAVE_PAPI if (xbt_cfg_get_string("smpi/papi-events")[0] != '\0') { int event_set = smpi_process_papi_event_set(); // PAPI_start sets everything to 0! See man(3) PAPI_start if (PAPI_LOW_LEVEL_INITED == PAPI_is_initialized()) { if (PAPI_start(event_set) != PAPI_OK) { // TODO This needs some proper handling. XBT_CRITICAL("Could not start PAPI counters.\n"); xbt_die("Error."); } } } #endif xbt_os_threadtimer_start(smpi_process_timer()); } void smpi_bench_end() { if (MC_is_active() || MC_record_replay_is_active()) return; double speedup = 1; xbt_os_timer_t timer = smpi_process_timer(); xbt_os_threadtimer_stop(timer); #if HAVE_PAPI /** * An MPI function has been called and now is the right time to update * our PAPI counters for this process. */ if (xbt_cfg_get_string("smpi/papi-events")[0] != '\0') { papi_counter_t& counter_data = smpi_process_papi_counters(); int event_set = smpi_process_papi_event_set(); std::vector event_values = std::vector(counter_data.size()); if (PAPI_stop(event_set, &event_values[0]) != PAPI_OK) { // Error XBT_CRITICAL("Could not stop PAPI counters.\n"); xbt_die("Error."); } else { for (unsigned int i = 0; i < counter_data.size(); i++) { counter_data[i].second += event_values[i]; // XBT_DEBUG("[%i] PAPI: Counter %s: Value is now %lli (got increment by %lli\n", smpi_process_index(), // counter_data[i].first.c_str(), counter_data[i].second, event_values[i]); } } } #endif if (smpi_process_get_sampling()) { XBT_CRITICAL("Cannot do recursive benchmarks."); XBT_CRITICAL("Are you trying to make a call to MPI within a SMPI_SAMPLE_ block?"); xbt_backtrace_display_current(); xbt_die("Aborting."); } if (xbt_cfg_get_string("smpi/comp-adjustment-file")[0] != '\0') { // Maybe we need to artificially speed up or slow // down our computation based on our statistical analysis. smpi_trace_call_location_t* loc = smpi_process_get_call_location(); std::string key = loc->get_composed_key(); std::unordered_map::const_iterator it = location2speedup.find(key); if (it != location2speedup.end()) { speedup = it->second; } } // Simulate the benchmarked computation unless disabled via command-line argument if (xbt_cfg_get_boolean("smpi/simulate-computation")) { smpi_execute(xbt_os_timer_elapsed(timer)/speedup); } #if HAVE_PAPI if (xbt_cfg_get_string("smpi/papi-events")[0] != '\0' && TRACE_smpi_is_enabled()) { char container_name[INSTR_DEFAULT_STR_SIZE]; smpi_container(smpi_process_index(), container_name, INSTR_DEFAULT_STR_SIZE); container_t container = PJ_container_get(container_name); papi_counter_t& counter_data = smpi_process_papi_counters(); for (auto& pair : counter_data) { new_pajeSetVariable(surf_get_clock(), container, PJ_type_get(/* countername */ pair.first.c_str(), container->type), pair.second); } } #endif smpi_total_benched_time += xbt_os_timer_elapsed(timer); } /* Private sleep function used by smpi_sleep() and smpi_usleep() */ static unsigned int private_sleep(double secs) { smpi_bench_end(); XBT_DEBUG("Sleep for: %lf secs", secs); int rank = smpi_comm_rank(MPI_COMM_WORLD); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type=TRACING_SLEEPING; extra->sleep_duration=secs; TRACE_smpi_sleeping_in(rank, extra); simcall_process_sleep(secs); TRACE_smpi_sleeping_out(rank); smpi_bench_begin(); return 0; } unsigned int smpi_sleep(unsigned int secs) { return private_sleep(static_cast(secs)); } int smpi_usleep(useconds_t usecs) { return static_cast(private_sleep(static_cast(usecs) / 1000000.0)); } #if _POSIX_TIMERS > 0 int smpi_nanosleep(const struct timespec *tp, struct timespec * t) { return static_cast(private_sleep(static_cast(tp->tv_sec + tp->tv_nsec / 1000000000.0))); } #endif int smpi_gettimeofday(struct timeval *tv, void* tz) { double now; smpi_bench_end(); now = SIMIX_get_clock(); if (tv) { tv->tv_sec = static_cast(now); #ifdef WIN32 tv->tv_usec = static_cast((now - tv->tv_sec) * 1e6); #else tv->tv_usec = static_cast((now - tv->tv_sec) * 1e6); #endif } smpi_bench_begin(); return 0; } #if _POSIX_TIMERS > 0 int smpi_clock_gettime(clockid_t clk_id, struct timespec *tp) { //there is only one time in SMPI, so clk_id is ignored. double now; smpi_bench_end(); now = SIMIX_get_clock(); if (tp) { tp->tv_sec = static_cast(now); tp->tv_nsec = static_cast((now - tp->tv_sec) * 1e9); } smpi_bench_begin(); return 0; } #endif extern double sg_surf_precision; unsigned long long smpi_rastro_resolution () { smpi_bench_end(); double resolution = (1/sg_surf_precision); smpi_bench_begin(); return static_cast(resolution); } unsigned long long smpi_rastro_timestamp () { smpi_bench_end(); double now = SIMIX_get_clock(); unsigned long long sec = (unsigned long long)now; unsigned long long pre = (now - sec) * smpi_rastro_resolution(); smpi_bench_begin(); return static_cast(sec) * smpi_rastro_resolution() + pre; } /* ****************************** Functions related to the SMPI_SAMPLE_ macros ************************************/ typedef struct { double threshold; /* maximal stderr requested (if positive) */ double relstderr; /* observed stderr so far */ double mean; /* mean of benched times, to be used if the block is disabled */ double sum; /* sum of benched times (to compute the mean and stderr) */ double sum_pow2; /* sum of the square of the benched times (to compute the stderr) */ int iters; /* amount of requested iterations */ int count; /* amount of iterations done so far */ int benching; /* 1: we are benchmarking; 0: we have enough data, no bench anymore */ } local_data_t; static char *sample_location(int global, const char *file, int line) { if (global) { return bprintf("%s:%d", file, line); } else { return bprintf("%s:%d:%d", file, line, smpi_process_index()); } } static int sample_enough_benchs(local_data_t *data) { int res = data->count >= data->iters; if (data->threshold>0.0) { if (data->count <2) res = 0; // not enough data if (data->relstderr > data->threshold) res = 0; // stderr too high yet } XBT_DEBUG("%s (count:%d iter:%d stderr:%f thres:%f mean:%fs)", (res?"enough benchs":"need more data"), data->count, data->iters, data->relstderr, data->threshold, data->mean); return res; } void smpi_sample_1(int global, const char *file, int line, int iters, double threshold) { char *loc = sample_location(global, file, line); local_data_t *data; smpi_bench_end(); /* Take time from previous, unrelated computation into account */ smpi_process_set_sampling(1); if (samples==nullptr) samples = xbt_dict_new_homogeneous(free); data = static_cast(xbt_dict_get_or_null(samples, loc)); if (data==nullptr) { xbt_assert(threshold>0 || iters>0, "You should provide either a positive amount of iterations to bench, or a positive maximal stderr (or both)"); data = static_cast( xbt_new(local_data_t, 1)); data->count = 0; data->sum = 0.0; data->sum_pow2 = 0.0; data->iters = iters; data->threshold = threshold; data->benching = 1; // If we have no data, we need at least one data->mean = 0; xbt_dict_set(samples, loc, data, nullptr); XBT_DEBUG("XXXXX First time ever on benched nest %s.",loc); } else { if (data->iters != iters || data->threshold != threshold) { XBT_ERROR("Asked to bench block %s with different settings %d, %f is not %d, %f. " "How did you manage to give two numbers at the same line??", loc, data->iters, data->threshold, iters,threshold); THROW_IMPOSSIBLE; } // if we already have some data, check whether sample_2 should get one more bench or whether it should emulate // the computation instead data->benching = (sample_enough_benchs(data) == 0); XBT_DEBUG("XXXX Re-entering the benched nest %s. %s",loc, (data->benching?"more benching needed":"we have enough data, skip computes")); } xbt_free(loc); } int smpi_sample_2(int global, const char *file, int line) { char *loc = sample_location(global, file, line); local_data_t *data; int res; xbt_assert(samples, "Y U NO use SMPI_SAMPLE_* macros? Stop messing directly with smpi_sample_* functions!"); data = static_cast(xbt_dict_get(samples, loc)); XBT_DEBUG("sample2 %s",loc); xbt_free(loc); if (data->benching==1) { // we need to run a new bench XBT_DEBUG("benchmarking: count:%d iter:%d stderr:%f thres:%f; mean:%f", data->count, data->iters, data->relstderr, data->threshold, data->mean); res = 1; } else { // Enough data, no more bench (either we got enough data from previous visits to this benched nest, or we just //ran one bench and need to bail out now that our job is done). Just sleep instead XBT_DEBUG("No benchmark (either no need, or just ran one): count >= iter (%d >= %d) or stderrcount, data->iters, data->relstderr, data->threshold, data->mean); smpi_execute(data->mean); smpi_process_set_sampling(0); res = 0; // prepare to capture future, unrelated computations } smpi_bench_begin(); return res; } void smpi_sample_3(int global, const char *file, int line) { char *loc = sample_location(global, file, line); local_data_t *data; xbt_assert(samples, "Y U NO use SMPI_SAMPLE_* macros? Stop messing directly with smpi_sample_* functions!"); data = static_cast(xbt_dict_get(samples, loc)); XBT_DEBUG("sample3 %s",loc); xbt_free(loc); if (data->benching==0) { THROW_IMPOSSIBLE; } // ok, benchmarking this loop is over xbt_os_threadtimer_stop(smpi_process_timer()); // update the stats double sample, n; data->count++; sample = xbt_os_timer_elapsed(smpi_process_timer()); data->sum += sample; data->sum_pow2 += sample * sample; n = static_cast(data->count); data->mean = data->sum / n; data->relstderr = sqrt((data->sum_pow2 / n - data->mean * data->mean) / n) / data->mean; if (sample_enough_benchs(data)==0) { data->mean = sample; // Still in benching process; We want sample_2 to simulate the exact time of this loop // occurrence before leaving, not the mean over the history } XBT_DEBUG("Average mean after %d steps is %f, relative standard error is %f (sample was %f)", data->count, data->mean, data->relstderr, sample); // That's enough for now, prevent sample_2 to run the same code over and over data->benching = 0; } #ifndef WIN32 void *smpi_shared_malloc(size_t size, const char *file, int line) { void* mem; if (size > 0 && xbt_cfg_get_boolean("smpi/use-shared-malloc")){ int fd; smpi_source_location loc(file, line); auto res = allocs.insert(std::make_pair(loc, shared_data_t())); auto data = res.first; if (res.second) { // The insertion did not take place. // Generate a shared memory name from the address of the shared_data: char shmname[32]; // cannot be longer than PSHMNAMLEN = 31 on Mac OS X (shm_open raises ENAMETOOLONG otherwise) snprintf(shmname, 31, "/shmalloc%p", &*data); fd = shm_open(shmname, O_RDWR | O_CREAT | O_EXCL, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH); if (fd < 0) { if(errno==EEXIST) xbt_die("Please cleanup /dev/shm/%s", shmname); else xbt_die("An unhandled error occurred while opening %s. shm_open: %s", shmname, strerror(errno)); } data->second.fd = fd; data->second.count = 1; mem = shm_map(fd, size, &*data); if (shm_unlink(shmname) < 0) { XBT_WARN("Could not early unlink %s. shm_unlink: %s", shmname, strerror(errno)); } XBT_DEBUG("Mapping %s at %p through %d", shmname, mem, fd); } else { mem = shm_map(data->second.fd, size, &*data); data->second.count++; } XBT_DEBUG("Shared malloc %zu in %p (metadata at %p)", size, mem, &*data); } else { mem = xbt_malloc(size); XBT_DEBUG("Classic malloc %zu in %p", size, mem); } return mem; } void smpi_shared_free(void *ptr) { char loc[PTR_STRLEN]; if (xbt_cfg_get_boolean("smpi/use-shared-malloc")){ snprintf(loc, PTR_STRLEN, "%p", ptr); auto meta = allocs_metadata.find(ptr); if (meta == allocs_metadata.end()) { XBT_WARN("Cannot free: %p was not shared-allocated by SMPI - maybe its size was 0?", ptr); return; } shared_data_t* data = &meta->second.data->second; if (munmap(ptr, meta->second.size) < 0) { XBT_WARN("Unmapping of fd %d failed: %s", data->fd, strerror(errno)); } data->count--; if (data->count <= 0) { close(data->fd); allocs.erase(allocs.find(meta->second.data->first)); XBT_DEBUG("Shared free - with removal - of %p", ptr); }else{ XBT_DEBUG("Shared free - no removal - of %p, count = %d", ptr, data->count); } }else{ XBT_DEBUG("Classic free of %p", ptr); xbt_free(ptr); } } #endif int smpi_shared_known_call(const char* func, const char* input) { char* loc = bprintf("%s:%s", func, input); int known = 0; if (calls==nullptr) { calls = xbt_dict_new_homogeneous(nullptr); } try { xbt_dict_get(calls, loc); /* Succeed or throw */ known = 1; xbt_free(loc); } catch (xbt_ex& ex) { xbt_free(loc); if (ex.category != not_found_error) throw; } catch(...) { xbt_free(loc); throw; } return known; } void* smpi_shared_get_call(const char* func, const char* input) { char* loc = bprintf("%s:%s", func, input); void* data; if(calls==nullptr) { calls = xbt_dict_new_homogeneous(nullptr); } data = xbt_dict_get(calls, loc); xbt_free(loc); return data; } void* smpi_shared_set_call(const char* func, const char* input, void* data) { char* loc = bprintf("%s:%s", func, input); if(calls==0) { calls = xbt_dict_new_homogeneous(nullptr); } xbt_dict_set(calls, loc, data, nullptr); xbt_free(loc); return data; } /** Map a given SMPI privatization segment (make a SMPI process active) */ void smpi_switch_data_segment(int dest) { if (smpi_loaded_page == dest)//no need to switch, we've already loaded the one we want return; // So the job: smpi_really_switch_data_segment(dest); } /** Map a given SMPI privatization segment (make a SMPI process active) even if SMPI thinks it is already active * * When doing a state restoration, the state of the restored variables might not be consistent with the state of the * virtual memory. In this case, we to change the data segment. */ void smpi_really_switch_data_segment(int dest) { if(smpi_size_data_exe == 0)//no need to switch return; #if HAVE_PRIVATIZATION if(smpi_loaded_page==-1){//initial switch, do the copy from the real page here for (int i=0; i< smpi_process_count(); i++){ memcpy(smpi_privatisation_regions[i].address, TOPAGE(smpi_start_data_exe), smpi_size_data_exe); } } // FIXME, cross-process support (mmap across process when necessary) int current = smpi_privatisation_regions[dest].file_descriptor; XBT_DEBUG("Switching data frame to the one of process %d", dest); void* tmp = mmap (TOPAGE(smpi_start_data_exe), smpi_size_data_exe, PROT_READ | PROT_WRITE, MAP_FIXED | MAP_SHARED, current, 0); if (tmp != TOPAGE(smpi_start_data_exe)) xbt_die("Couldn't map the new region"); smpi_loaded_page = dest; #endif } int smpi_is_privatisation_file(char* file) { return strncmp("/dev/shm/my-buffer-", file, std::strlen("/dev/shm/my-buffer-")) == 0; } void smpi_initialize_global_memory_segments(){ #if !HAVE_PRIVATIZATION smpi_privatize_global_variables=false; xbt_die("You are trying to use privatization on a system that does not support it. Don't."); return; #else smpi_get_executable_global_size(); XBT_DEBUG ("bss+data segment found : size %d starting at %p", smpi_size_data_exe, smpi_start_data_exe ); if (smpi_size_data_exe == 0){//no need to switch smpi_privatize_global_variables=false; return; } smpi_privatisation_regions = static_cast( xbt_malloc(smpi_process_count() * sizeof(struct s_smpi_privatisation_region))); for (int i=0; i< smpi_process_count(); i++){ //create SIMIX_process_count() mappings of this size with the same data inside int file_descriptor; void *address = nullptr; char path[24]; int status; do { snprintf(path, sizeof(path), "/smpi-buffer-%06x", rand()%0xffffff); file_descriptor = shm_open(path, O_RDWR|O_CREAT|O_EXCL, S_IRUSR|S_IWUSR); } while (file_descriptor == -1 && errno == EEXIST); if (file_descriptor < 0) { if (errno==EMFILE) { xbt_die("Impossible to create temporary file for memory mapping: %s\n\ The open() system call failed with the EMFILE error code (too many files). \n\n\ This means that you reached the system limits concerning the amount of files per process. \ This is not a surprise if you are trying to virtualize many processes on top of SMPI. \ Don't panic -- you should simply increase your system limits and try again. \n\n\ First, check what your limits are:\n\ cat /proc/sys/fs/file-max # Gives you the system-wide limit\n\ ulimit -Hn # Gives you the per process hard limit\n\ ulimit -Sn # Gives you the per process soft limit\n\ cat /proc/self/limits # Displays any per-process limitation (including the one given above)\n\n\ If one of these values is less than the amount of MPI processes that you try to run, then you got the explanation of this error. \ Ask the Internet about tutorials on how to increase the files limit such as: https://rtcamp.com/tutorials/linux/increase-open-files-limit/", strerror(errno)); } xbt_die("Impossible to create temporary file for memory mapping: %s", strerror(errno)); } status = ftruncate(file_descriptor, smpi_size_data_exe); if(status) xbt_die("Impossible to set the size of the temporary file for memory mapping"); /* Ask for a free region */ address = mmap (nullptr, smpi_size_data_exe, PROT_READ | PROT_WRITE, MAP_SHARED, file_descriptor, 0); if (address == MAP_FAILED) xbt_die("Couldn't find a free region for memory mapping"); status = shm_unlink(path); if (status) xbt_die("Impossible to unlink temporary file for memory mapping"); //initialize the values memcpy(address, TOPAGE(smpi_start_data_exe), smpi_size_data_exe); //store the address of the mapping for further switches smpi_privatisation_regions[i].file_descriptor = file_descriptor; smpi_privatisation_regions[i].address = address; } #endif } void smpi_destroy_global_memory_segments(){ if (smpi_size_data_exe == 0)//no need to switch return; #if HAVE_PRIVATIZATION int i; for (i=0; i< smpi_process_count(); i++){ if(munmap(smpi_privatisation_regions[i].address, smpi_size_data_exe) < 0) { XBT_WARN("Unmapping of fd %d failed: %s", smpi_privatisation_regions[i].file_descriptor, strerror(errno)); } close(smpi_privatisation_regions[i].file_descriptor); } xbt_free(smpi_privatisation_regions); #endif } extern "C" { /** These functions will be called from the user code **/ smpi_trace_call_location_t* smpi_trace_get_call_location() { return smpi_process_get_call_location(); } void smpi_trace_set_call_location(const char* file, const int line) { smpi_trace_call_location_t* loc = smpi_process_get_call_location(); loc->previous_filename = loc->filename; loc->previous_linenumber = loc->linenumber; loc->filename = file; loc->linenumber = line; } /** * Required for Fortran bindings */ void smpi_trace_set_call_location_(const char* file, int* line) { smpi_trace_set_call_location(file, *line); } /** * Required for Fortran if -fsecond-underscore is activated */ void smpi_trace_set_call_location__(const char* file, int* line) { smpi_trace_set_call_location(file, *line); } } SimGrid-3.14.159/src/smpi/smpi_deployment.cpp000644 001750 001750 00000006136 13031031272 022551 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2004-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.h" #include "simgrid/msg.h" /* barrier */ #include "xbt/dict.h" #include "xbt/log.h" #include "xbt/sysdep.h" static xbt_dict_t smpi_instances = nullptr; extern int process_count; extern int* index_to_process_data; typedef struct s_smpi_mpi_instance{ const char* name; int size; int present_processes; int index; MPI_Comm comm_world; msg_bar_t finalization_barrier; } s_smpi_mpi_instance_t; /** \ingroup smpi_simulation * \brief Registers a running instance of a MPI program. * * FIXME : remove MSG from the loop at some point. * \param name the reference name of the function. * \param code the main mpi function (must have a int ..(int argc, char *argv[])) prototype * \param num_processes the size of the instance we want to deploy */ void SMPI_app_instance_register(const char *name, xbt_main_func_t code, int num_processes) { SIMIX_function_register(name, code); s_smpi_mpi_instance_t* instance = (s_smpi_mpi_instance_t*)xbt_malloc(sizeof(s_smpi_mpi_instance_t)); instance->name = name; instance->size = num_processes; instance->present_processes = 0; instance->index = process_count; instance->comm_world = MPI_COMM_NULL; instance->finalization_barrier = MSG_barrier_init(num_processes); process_count+=num_processes; if(smpi_instances==nullptr){ smpi_instances = xbt_dict_new_homogeneous(xbt_free_f); } xbt_dict_set(smpi_instances, name, (void*)instance, nullptr); return; } //get the index of the process in the process_data array void smpi_deployment_register_process(const char* instance_id, int rank, int index, MPI_Comm** comm, msg_bar_t* bar) { if(smpi_instances==nullptr){//no instance registered, we probably used smpirun. index_to_process_data[index]=index; *bar = nullptr; *comm = nullptr; return; } s_smpi_mpi_instance_t* instance = static_cast(xbt_dict_get_or_null(smpi_instances, instance_id)); xbt_assert(instance, "Error, unknown instance %s", instance_id); if(instance->comm_world == MPI_COMM_NULL){ MPI_Group group = smpi_group_new(instance->size); instance->comm_world = smpi_comm_new(group, nullptr); } instance->present_processes++; index_to_process_data[index]=instance->index+rank; smpi_group_set_mapping(smpi_comm_group(instance->comm_world), index, rank); *bar = instance->finalization_barrier; *comm = &instance->comm_world; return; } void smpi_deployment_cleanup_instances(){ xbt_dict_cursor_t cursor = nullptr; s_smpi_mpi_instance_t* instance = nullptr; char *name = nullptr; xbt_dict_foreach(smpi_instances, cursor, name, instance) { if(instance->comm_world!=MPI_COMM_NULL) while (smpi_group_unuse(smpi_comm_group(instance->comm_world)) > 0); xbt_free(instance->comm_world); MSG_barrier_destroy(instance->finalization_barrier); } xbt_dict_free(&smpi_instances); } SimGrid-3.14.159/src/smpi/smpiff.in000644 001750 001750 00000005466 13031031276 020462 0ustar00mquinsonmquinson000000 000000 #! /bin/sh # Copyright (c) 2012-2015. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. SIMGRID_VERSION="@SIMGRID_VERSION_STRING@" SIMGRID_GITHASH="@SIMGRID_GITHASH@" F77=@SAVED_Fortran_COMPILER@ INCLUDEARGS="@includeflag@" CMAKE_LINKARGS="-L@libdir@" @SMPITOOLS_SH@ list_set FFLAGS "-ff2c" "-fno-second-underscore" list_set LINKARGS "-lsimgrid" "-lm" "-lgfortran" list_set TMPFILES main_name=main cleanup () { eval $(list_get TMPFILES) rm -f "$@" } trap 'cleanup' EXIT filter_and_compile() { list_add TMPFILES "${TMPFILE}" #replace "program main_name by subroutine user\_main (and the end clause as well)" if [ $TRACE_CALL_LOCATION -gt 0 ]; then echo "#include \"@includedir@/smpi/smpi_extended_traces_fortran.h\"" > ${TMPFILE} echo "#line 1 \"${ARG}\"" >> ${TMPFILE} fi sed 's/[[:space:]]\{6\}[[:space:]]*\([eE][nN][dD] \)\{0,1\}[pP][rR][oO][gG][rR][aA][mM][[:space:]]*\([a-zA-Z0-9\-\_]*\)/ \1subroutine user_main /g;s/[[:space:]]*[uU][sS][eE][[:space:]]*[mM][pP][iI]/\include \"mpif\.h\" /g' "${ARG}" >> "${TMPFILE}" SRCFILE="${TMPFILE}" list_add CMDLINE "${SRCFILE}" } TRACE_CALL_LOCATION=0 NEEDS_OUTPUT=1 list_set CMDLINE "${F77}" list_add_not_empty CMDLINE "${FFLAGS}" while [ $# -gt 0 ]; do ARG="$1" shift case "${ARG}" in -c) CMAKE_LINKARGS="" LINKARGS="" list_add CMDLINE "-c" ;; *.f) TMPFILE=$(mymktemp "${ARG}" ".f") ORIGFILE="${ARG%.f}" filter_and_compile ;; *.F) TMPFILE=$(mymktemp "${ARG}" ".F") ORIGFILE="${ARG%.F}" filter_and_compile ;; '-version' | '--version') printf '%b\n' "$SIMGRID_VERSION" exit 0 ;; "-git-version" | "--git-version") printf '%b\n' "$SIMGRID_GITHASH" exit 0 ;; '-compiler-version' | '--compiler-version') ${F77} --version ;; '-trace-call-location') TRACE_CALL_LOCATION=1 # This should be list_add FFLAGS but it's not possible # anymore: FFLAGS was already moved into CMDLINE above. list_add_not_empty CMDLINE "-ffixed-line-length-none" "-cpp" ;; -o) NEEDS_OUTPUT=0 list_add CMDLINE "-o" ;; *) list_add CMDLINE "${ARG}" ;; esac done if [ $NEEDS_OUTPUT -ne 0 ]; then list_add CMDLINE "-o${ORIGFILE}.o" fi list_add_not_empty CMDLINE ${INCLUDEARGS} list_add_not_empty CMDLINE ${CMAKE_LINKARGS} list_add_not_empty CMDLINE "${LINKARGS}" eval $(list_get CMDLINE) "$@" SimGrid-3.14.159/src/smpi/smpi_replay.cpp000644 001750 001750 00000104725 13031031272 021670 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2009-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.h" #include #include #include #include #include #define KEY_SIZE (sizeof(int) * 2 + 1) XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_replay,smpi,"Trace Replay with SMPI"); int communicator_size = 0; static int active_processes = 0; std::unordered_map*> reqq; MPI_Datatype MPI_DEFAULT_TYPE; MPI_Datatype MPI_CURRENT_TYPE; static int sendbuffer_size=0; char* sendbuffer=nullptr; static int recvbuffer_size=0; char* recvbuffer=nullptr; static void log_timed_action (const char *const *action, double clock){ if (XBT_LOG_ISENABLED(smpi_replay, xbt_log_priority_verbose)){ char *name = xbt_str_join_array(action, " "); XBT_VERB("%s %f", name, smpi_process_simulated_elapsed()-clock); xbt_free(name); } } static std::vector* get_reqq_self() { return reqq.at(smpi_process_index()); } static void set_reqq_self(std::vector *mpi_request) { reqq.insert({smpi_process_index(), mpi_request}); } //allocate a single buffer for all sends, growing it if needed void* smpi_get_tmp_sendbuffer(int size) { if (!smpi_process_get_replaying()) return xbt_malloc(size); if (sendbuffer_size(xbt_realloc(sendbuffer,size)); sendbuffer_size=size; } return sendbuffer; } //allocate a single buffer for all recv void* smpi_get_tmp_recvbuffer(int size){ if (!smpi_process_get_replaying()) return xbt_malloc(size); if (recvbuffer_size(xbt_realloc(recvbuffer,size)); recvbuffer_size=size; } return recvbuffer; } void smpi_free_tmp_buffer(void* buf){ if (!smpi_process_get_replaying()) xbt_free(buf); } /* Helper function */ static double parse_double(const char *string) { double value; char *endptr; value = strtod(string, &endptr); if (*endptr != '\0') THROWF(unknown_error, 0, "%s is not a double", string); return value; } static MPI_Datatype decode_datatype(const char *const action) { // Declared datatypes, switch(atoi(action)) { case 0: MPI_CURRENT_TYPE=MPI_DOUBLE; break; case 1: MPI_CURRENT_TYPE=MPI_INT; break; case 2: MPI_CURRENT_TYPE=MPI_CHAR; break; case 3: MPI_CURRENT_TYPE=MPI_SHORT; break; case 4: MPI_CURRENT_TYPE=MPI_LONG; break; case 5: MPI_CURRENT_TYPE=MPI_FLOAT; break; case 6: MPI_CURRENT_TYPE=MPI_BYTE; break; default: MPI_CURRENT_TYPE=MPI_DEFAULT_TYPE; } return MPI_CURRENT_TYPE; } const char* encode_datatype(MPI_Datatype datatype, int* known) { //default type for output is set to MPI_BYTE // MPI_DEFAULT_TYPE is not set for output, use directly MPI_BYTE if(known!=nullptr) *known=1; if (datatype==MPI_BYTE){ return ""; } if(datatype==MPI_DOUBLE) return "0"; if(datatype==MPI_INT) return "1"; if(datatype==MPI_CHAR) return "2"; if(datatype==MPI_SHORT) return "3"; if(datatype==MPI_LONG) return "4"; if(datatype==MPI_FLOAT) return "5"; //tell that the datatype is not handled by replay, and that its size should be measured and replayed as size*MPI_BYTE if(known!=nullptr) *known=0; // default - not implemented. // do not warn here as we pass in this function even for other trace formats return "-1"; } #define CHECK_ACTION_PARAMS(action, mandatory, optional) {\ int i=0;\ while(action[i]!=nullptr)\ i++;\ if(i); } static void action_finalize(const char *const *action) { /* do nothing */ } static void action_comm_size(const char *const *action) { double clock = smpi_process_simulated_elapsed(); communicator_size = parse_double(action[2]); log_timed_action (action, clock); } static void action_comm_split(const char *const *action) { double clock = smpi_process_simulated_elapsed(); log_timed_action (action, clock); } static void action_comm_dup(const char *const *action) { double clock = smpi_process_simulated_elapsed(); log_timed_action (action, clock); } static void action_compute(const char *const *action) { CHECK_ACTION_PARAMS(action, 1, 0) double clock = smpi_process_simulated_elapsed(); double flops= parse_double(action[2]); int rank = smpi_process_index(); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type=TRACING_COMPUTING; extra->comp_size=flops; TRACE_smpi_computing_in(rank, extra); smpi_execute_flops(flops); TRACE_smpi_computing_out(rank); log_timed_action (action, clock); } static void action_send(const char *const *action) { CHECK_ACTION_PARAMS(action, 2, 1) int to = atoi(action[2]); double size=parse_double(action[3]); double clock = smpi_process_simulated_elapsed(); if(action[4]) MPI_CURRENT_TYPE=decode_datatype(action[4]); else MPI_CURRENT_TYPE= MPI_DEFAULT_TYPE; int rank = smpi_process_index(); int dst_traced = smpi_group_rank(smpi_comm_group(MPI_COMM_WORLD), to); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_SEND; extra->send_size = size; extra->src = rank; extra->dst = dst_traced; extra->datatype1 = encode_datatype(MPI_CURRENT_TYPE, nullptr); TRACE_smpi_ptp_in(rank, rank, dst_traced, __FUNCTION__, extra); if (!TRACE_smpi_view_internals()) { TRACE_smpi_send(rank, rank, dst_traced, 0, size*smpi_datatype_size(MPI_CURRENT_TYPE)); } smpi_mpi_send(nullptr, size, MPI_CURRENT_TYPE, to , 0, MPI_COMM_WORLD); log_timed_action (action, clock); TRACE_smpi_ptp_out(rank, rank, dst_traced, __FUNCTION__); } static void action_Isend(const char *const *action) { CHECK_ACTION_PARAMS(action, 2, 1) int to = atoi(action[2]); double size=parse_double(action[3]); double clock = smpi_process_simulated_elapsed(); MPI_Request request; if(action[4]) MPI_CURRENT_TYPE=decode_datatype(action[4]); else MPI_CURRENT_TYPE= MPI_DEFAULT_TYPE; int rank = smpi_process_index(); int dst_traced = smpi_group_rank(smpi_comm_group(MPI_COMM_WORLD), to); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_ISEND; extra->send_size = size; extra->src = rank; extra->dst = dst_traced; extra->datatype1 = encode_datatype(MPI_CURRENT_TYPE, nullptr); TRACE_smpi_ptp_in(rank, rank, dst_traced, __FUNCTION__, extra); if (!TRACE_smpi_view_internals()) { TRACE_smpi_send(rank, rank, dst_traced, 0, size*smpi_datatype_size(MPI_CURRENT_TYPE)); } request = smpi_mpi_isend(nullptr, size, MPI_CURRENT_TYPE, to, 0,MPI_COMM_WORLD); TRACE_smpi_ptp_out(rank, rank, dst_traced, __FUNCTION__); request->send = 1; get_reqq_self()->push_back(request); log_timed_action (action, clock); } static void action_recv(const char *const *action) { CHECK_ACTION_PARAMS(action, 2, 1) int from = atoi(action[2]); double size=parse_double(action[3]); double clock = smpi_process_simulated_elapsed(); MPI_Status status; if(action[4]) MPI_CURRENT_TYPE=decode_datatype(action[4]); else MPI_CURRENT_TYPE= MPI_DEFAULT_TYPE; int rank = smpi_process_index(); int src_traced = smpi_group_rank(smpi_comm_group(MPI_COMM_WORLD), from); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_RECV; extra->send_size = size; extra->src = src_traced; extra->dst = rank; extra->datatype1 = encode_datatype(MPI_CURRENT_TYPE, nullptr); TRACE_smpi_ptp_in(rank, src_traced, rank, __FUNCTION__, extra); //unknown size from the receiver point of view if(size<=0.0){ smpi_mpi_probe(from, 0, MPI_COMM_WORLD, &status); size=status.count; } smpi_mpi_recv(nullptr, size, MPI_CURRENT_TYPE, from, 0, MPI_COMM_WORLD, &status); TRACE_smpi_ptp_out(rank, src_traced, rank, __FUNCTION__); if (!TRACE_smpi_view_internals()) { TRACE_smpi_recv(rank, src_traced, rank, 0); } log_timed_action (action, clock); } static void action_Irecv(const char *const *action) { CHECK_ACTION_PARAMS(action, 2, 1) int from = atoi(action[2]); double size=parse_double(action[3]); double clock = smpi_process_simulated_elapsed(); MPI_Request request; if(action[4]) MPI_CURRENT_TYPE=decode_datatype(action[4]); else MPI_CURRENT_TYPE= MPI_DEFAULT_TYPE; int rank = smpi_process_index(); int src_traced = smpi_group_rank(smpi_comm_group(MPI_COMM_WORLD), from); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_IRECV; extra->send_size = size; extra->src = src_traced; extra->dst = rank; extra->datatype1 = encode_datatype(MPI_CURRENT_TYPE, nullptr); TRACE_smpi_ptp_in(rank, src_traced, rank, __FUNCTION__, extra); MPI_Status status; //unknow size from the receiver pov if(size<=0.0){ smpi_mpi_probe(from, 0, MPI_COMM_WORLD, &status); size=status.count; } request = smpi_mpi_irecv(nullptr, size, MPI_CURRENT_TYPE, from, 0, MPI_COMM_WORLD); TRACE_smpi_ptp_out(rank, src_traced, rank, __FUNCTION__); request->recv = 1; get_reqq_self()->push_back(request); log_timed_action (action, clock); } static void action_test(const char *const *action){ CHECK_ACTION_PARAMS(action, 0, 0) double clock = smpi_process_simulated_elapsed(); MPI_Status status; int flag = true; MPI_Request request = get_reqq_self()->back(); get_reqq_self()->pop_back(); //if request is null here, this may mean that a previous test has succeeded //Different times in traced application and replayed version may lead to this //In this case, ignore the extra calls. if(request!=nullptr){ int rank = smpi_process_index(); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type=TRACING_TEST; TRACE_smpi_testing_in(rank, extra); flag = smpi_mpi_test(&request, &status); XBT_DEBUG("MPI_Test result: %d", flag); /* push back request in vector to be caught by a subsequent wait. if the test did succeed, the request is now nullptr.*/ get_reqq_self()->push_back(request); TRACE_smpi_testing_out(rank); } log_timed_action (action, clock); } static void action_wait(const char *const *action){ CHECK_ACTION_PARAMS(action, 0, 0) double clock = smpi_process_simulated_elapsed(); MPI_Status status; xbt_assert(get_reqq_self()->size(), "action wait not preceded by any irecv or isend: %s", xbt_str_join_array(action," ")); MPI_Request request = get_reqq_self()->back(); get_reqq_self()->pop_back(); if (request==nullptr){ /* Assume that the trace is well formed, meaning the comm might have been caught by a MPI_test. Then just return.*/ return; } int rank = request->comm != MPI_COMM_NULL ? smpi_comm_rank(request->comm) : -1; MPI_Group group = smpi_comm_group(request->comm); int src_traced = smpi_group_rank(group, request->src); int dst_traced = smpi_group_rank(group, request->dst); int is_wait_for_receive = request->recv; instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_WAIT; TRACE_smpi_ptp_in(rank, src_traced, dst_traced, __FUNCTION__, extra); smpi_mpi_wait(&request, &status); TRACE_smpi_ptp_out(rank, src_traced, dst_traced, __FUNCTION__); if (is_wait_for_receive) TRACE_smpi_recv(rank, src_traced, dst_traced, 0); log_timed_action (action, clock); } static void action_waitall(const char *const *action){ CHECK_ACTION_PARAMS(action, 0, 0) double clock = smpi_process_simulated_elapsed(); unsigned int count_requests=get_reqq_self()->size(); if (count_requests>0) { MPI_Status status[count_requests]; int rank_traced = smpi_process_index(); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_WAITALL; extra->send_size=count_requests; TRACE_smpi_ptp_in(rank_traced, -1, -1, __FUNCTION__,extra); int* recvs_snd= xbt_new0(int,count_requests); int* recvs_rcv= xbt_new0(int,count_requests); unsigned int i=0; for (auto req : *(get_reqq_self())){ if (req && req->recv){ recvs_snd[i]=req->src; recvs_rcv[i]=req->dst; }else recvs_snd[i]=-100; i++; } smpi_mpi_waitall(count_requests, &(*get_reqq_self())[0], status); for (i=0; itype = TRACING_BARRIER; TRACE_smpi_collective_in(rank, -1, __FUNCTION__, extra); mpi_coll_barrier_fun(MPI_COMM_WORLD); TRACE_smpi_collective_out(rank, -1, __FUNCTION__); log_timed_action (action, clock); } static void action_bcast(const char *const *action) { CHECK_ACTION_PARAMS(action, 1, 2) double size = parse_double(action[2]); double clock = smpi_process_simulated_elapsed(); int root=0; /* Initialize MPI_CURRENT_TYPE in order to decrease the number of the checks */ MPI_CURRENT_TYPE= MPI_DEFAULT_TYPE; if(action[3]) { root= atoi(action[3]); if(action[4]) { MPI_CURRENT_TYPE=decode_datatype(action[4]); } } int rank = smpi_process_index(); int root_traced = smpi_group_index(smpi_comm_group(MPI_COMM_WORLD), root); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_BCAST; extra->send_size = size; extra->root = root_traced; extra->datatype1 = encode_datatype(MPI_CURRENT_TYPE, nullptr); TRACE_smpi_collective_in(rank, root_traced, __FUNCTION__, extra); void *sendbuf = smpi_get_tmp_sendbuffer(size* smpi_datatype_size(MPI_CURRENT_TYPE)); mpi_coll_bcast_fun(sendbuf, size, MPI_CURRENT_TYPE, root, MPI_COMM_WORLD); TRACE_smpi_collective_out(rank, root_traced, __FUNCTION__); log_timed_action (action, clock); } static void action_reduce(const char *const *action) { CHECK_ACTION_PARAMS(action, 2, 2) double comm_size = parse_double(action[2]); double comp_size = parse_double(action[3]); double clock = smpi_process_simulated_elapsed(); int root=0; MPI_CURRENT_TYPE= MPI_DEFAULT_TYPE; if(action[4]) { root= atoi(action[4]); if(action[5]) { MPI_CURRENT_TYPE=decode_datatype(action[5]); } } int rank = smpi_process_index(); int root_traced = smpi_group_rank(smpi_comm_group(MPI_COMM_WORLD), root); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_REDUCE; extra->send_size = comm_size; extra->comp_size = comp_size; extra->datatype1 = encode_datatype(MPI_CURRENT_TYPE, nullptr); extra->root = root_traced; TRACE_smpi_collective_in(rank, root_traced, __FUNCTION__,extra); void *recvbuf = smpi_get_tmp_sendbuffer(comm_size* smpi_datatype_size(MPI_CURRENT_TYPE)); void *sendbuf = smpi_get_tmp_sendbuffer(comm_size* smpi_datatype_size(MPI_CURRENT_TYPE)); mpi_coll_reduce_fun(sendbuf, recvbuf, comm_size, MPI_CURRENT_TYPE, MPI_OP_NULL, root, MPI_COMM_WORLD); smpi_execute_flops(comp_size); TRACE_smpi_collective_out(rank, root_traced, __FUNCTION__); log_timed_action (action, clock); } static void action_allReduce(const char *const *action) { CHECK_ACTION_PARAMS(action, 2, 1) double comm_size = parse_double(action[2]); double comp_size = parse_double(action[3]); if(action[4]) MPI_CURRENT_TYPE=decode_datatype(action[4]); else MPI_CURRENT_TYPE= MPI_DEFAULT_TYPE; double clock = smpi_process_simulated_elapsed(); int rank = smpi_process_index(); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_ALLREDUCE; extra->send_size = comm_size; extra->comp_size = comp_size; extra->datatype1 = encode_datatype(MPI_CURRENT_TYPE, nullptr); TRACE_smpi_collective_in(rank, -1, __FUNCTION__,extra); void *recvbuf = smpi_get_tmp_sendbuffer(comm_size* smpi_datatype_size(MPI_CURRENT_TYPE)); void *sendbuf = smpi_get_tmp_sendbuffer(comm_size* smpi_datatype_size(MPI_CURRENT_TYPE)); mpi_coll_allreduce_fun(sendbuf, recvbuf, comm_size, MPI_CURRENT_TYPE, MPI_OP_NULL, MPI_COMM_WORLD); smpi_execute_flops(comp_size); TRACE_smpi_collective_out(rank, -1, __FUNCTION__); log_timed_action (action, clock); } static void action_allToAll(const char *const *action) { CHECK_ACTION_PARAMS(action, 2, 2) //two mandatory (send and recv volumes) //two optional (corresponding datatypes) double clock = smpi_process_simulated_elapsed(); int comm_size = smpi_comm_size(MPI_COMM_WORLD); int send_size = parse_double(action[2]); int recv_size = parse_double(action[3]); MPI_Datatype MPI_CURRENT_TYPE2; if(action[4] && action[5]) { MPI_CURRENT_TYPE=decode_datatype(action[4]); MPI_CURRENT_TYPE2=decode_datatype(action[5]); } else{ MPI_CURRENT_TYPE=MPI_DEFAULT_TYPE; MPI_CURRENT_TYPE2=MPI_DEFAULT_TYPE; } void *send = smpi_get_tmp_sendbuffer(send_size*comm_size* smpi_datatype_size(MPI_CURRENT_TYPE)); void *recv = smpi_get_tmp_recvbuffer(recv_size*comm_size* smpi_datatype_size(MPI_CURRENT_TYPE2)); int rank = smpi_process_index(); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_ALLTOALL; extra->send_size = send_size; extra->recv_size = recv_size; extra->datatype1 = encode_datatype(MPI_CURRENT_TYPE, nullptr); extra->datatype2 = encode_datatype(MPI_CURRENT_TYPE2, nullptr); TRACE_smpi_collective_in(rank, -1, __FUNCTION__,extra); mpi_coll_alltoall_fun(send, send_size, MPI_CURRENT_TYPE, recv, recv_size, MPI_CURRENT_TYPE2, MPI_COMM_WORLD); TRACE_smpi_collective_out(rank, -1, __FUNCTION__); log_timed_action (action, clock); } static void action_gather(const char *const *action) { /* The structure of the gather action for the rank 0 (total 4 processes) is the following: 0 gather 68 68 0 0 0 where: 1) 68 is the sendcounts 2) 68 is the recvcounts 3) 0 is the root node 4) 0 is the send datatype id, see decode_datatype() 5) 0 is the recv datatype id, see decode_datatype() */ CHECK_ACTION_PARAMS(action, 2, 3) double clock = smpi_process_simulated_elapsed(); int comm_size = smpi_comm_size(MPI_COMM_WORLD); int send_size = parse_double(action[2]); int recv_size = parse_double(action[3]); MPI_Datatype MPI_CURRENT_TYPE2; if(action[4] && action[5]) { MPI_CURRENT_TYPE=decode_datatype(action[5]); MPI_CURRENT_TYPE2=decode_datatype(action[6]); } else { MPI_CURRENT_TYPE=MPI_DEFAULT_TYPE; MPI_CURRENT_TYPE2=MPI_DEFAULT_TYPE; } void *send = smpi_get_tmp_sendbuffer(send_size* smpi_datatype_size(MPI_CURRENT_TYPE)); void *recv = nullptr; int root=0; if(action[4]) root=atoi(action[4]); int rank = smpi_comm_rank(MPI_COMM_WORLD); if(rank==root) recv = smpi_get_tmp_recvbuffer(recv_size*comm_size* smpi_datatype_size(MPI_CURRENT_TYPE2)); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_GATHER; extra->send_size = send_size; extra->recv_size = recv_size; extra->root = root; extra->datatype1 = encode_datatype(MPI_CURRENT_TYPE, nullptr); extra->datatype2 = encode_datatype(MPI_CURRENT_TYPE2, nullptr); TRACE_smpi_collective_in(smpi_process_index(), root, __FUNCTION__, extra); mpi_coll_gather_fun(send, send_size, MPI_CURRENT_TYPE, recv, recv_size, MPI_CURRENT_TYPE2, root, MPI_COMM_WORLD); TRACE_smpi_collective_out(smpi_process_index(), -1, __FUNCTION__); log_timed_action (action, clock); } static void action_gatherv(const char *const *action) { /* The structure of the gatherv action for the rank 0 (total 4 processes) is the following: 0 gather 68 68 10 10 10 0 0 0 where: 1) 68 is the sendcount 2) 68 10 10 10 is the recvcounts 3) 0 is the root node 4) 0 is the send datatype id, see decode_datatype() 5) 0 is the recv datatype id, see decode_datatype() */ double clock = smpi_process_simulated_elapsed(); int comm_size = smpi_comm_size(MPI_COMM_WORLD); CHECK_ACTION_PARAMS(action, comm_size+1, 2) int send_size = parse_double(action[2]); int *disps = xbt_new0(int, comm_size); int *recvcounts = xbt_new0(int, comm_size); int i=0,recv_sum=0; MPI_Datatype MPI_CURRENT_TYPE2; if(action[4+comm_size] && action[5+comm_size]) { MPI_CURRENT_TYPE=decode_datatype(action[4+comm_size]); MPI_CURRENT_TYPE2=decode_datatype(action[5+comm_size]); } else { MPI_CURRENT_TYPE=MPI_DEFAULT_TYPE; MPI_CURRENT_TYPE2=MPI_DEFAULT_TYPE; } void *send = smpi_get_tmp_sendbuffer(send_size* smpi_datatype_size(MPI_CURRENT_TYPE)); void *recv = nullptr; for(i=0;itype = TRACING_GATHERV; extra->send_size = send_size; extra->recvcounts= xbt_new(int,comm_size); for(i=0; i< comm_size; i++)//copy data to avoid bad free extra->recvcounts[i] = recvcounts[i]; extra->root = root; extra->num_processes = comm_size; extra->datatype1 = encode_datatype(MPI_CURRENT_TYPE, nullptr); extra->datatype2 = encode_datatype(MPI_CURRENT_TYPE2, nullptr); TRACE_smpi_collective_in(smpi_process_index(), root, __FUNCTION__, extra); smpi_mpi_gatherv(send, send_size, MPI_CURRENT_TYPE, recv, recvcounts, disps, MPI_CURRENT_TYPE2, root, MPI_COMM_WORLD); TRACE_smpi_collective_out(smpi_process_index(), -1, __FUNCTION__); log_timed_action (action, clock); xbt_free(recvcounts); xbt_free(disps); } static void action_reducescatter(const char *const *action) { /* The structure of the reducescatter action for the rank 0 (total 4 processes) is the following: 0 reduceScatter 275427 275427 275427 204020 11346849 0 where: 1) The first four values after the name of the action declare the recvcounts array 2) The value 11346849 is the amount of instructions 3) The last value corresponds to the datatype, see decode_datatype(). We analyze a MPI_Reduce_scatter call to one MPI_Reduce and one MPI_Scatterv. */ double clock = smpi_process_simulated_elapsed(); int comm_size = smpi_comm_size(MPI_COMM_WORLD); CHECK_ACTION_PARAMS(action, comm_size+1, 1) int comp_size = parse_double(action[2+comm_size]); int *recvcounts = xbt_new0(int, comm_size); int *disps = xbt_new0(int, comm_size); int i=0; int rank = smpi_process_index(); int size = 0; if(action[3+comm_size]) MPI_CURRENT_TYPE=decode_datatype(action[3+comm_size]); else MPI_CURRENT_TYPE= MPI_DEFAULT_TYPE; for(i=0;itype = TRACING_REDUCE_SCATTER; extra->send_size = 0; extra->recvcounts= xbt_new(int, comm_size); for(i=0; i< comm_size; i++)//copy data to avoid bad free extra->recvcounts[i] = recvcounts[i]; extra->datatype1 = encode_datatype(MPI_CURRENT_TYPE, nullptr); extra->comp_size = comp_size; extra->num_processes = comm_size; TRACE_smpi_collective_in(rank, -1, __FUNCTION__,extra); void *sendbuf = smpi_get_tmp_sendbuffer(size* smpi_datatype_size(MPI_CURRENT_TYPE)); void *recvbuf = smpi_get_tmp_recvbuffer(size* smpi_datatype_size(MPI_CURRENT_TYPE)); mpi_coll_reduce_scatter_fun(sendbuf, recvbuf, recvcounts, MPI_CURRENT_TYPE, MPI_OP_NULL, MPI_COMM_WORLD); smpi_execute_flops(comp_size); TRACE_smpi_collective_out(rank, -1, __FUNCTION__); xbt_free(recvcounts); xbt_free(disps); log_timed_action (action, clock); } static void action_allgather(const char *const *action) { /* The structure of the allgather action for the rank 0 (total 4 processes) is the following: 0 allGather 275427 275427 where: 1) 275427 is the sendcount 2) 275427 is the recvcount 3) No more values mean that the datatype for sent and receive buffer is the default one, see decode_datatype(). */ double clock = smpi_process_simulated_elapsed(); CHECK_ACTION_PARAMS(action, 2, 2) int sendcount=atoi(action[2]); int recvcount=atoi(action[3]); MPI_Datatype MPI_CURRENT_TYPE2; if(action[4] && action[5]) { MPI_CURRENT_TYPE = decode_datatype(action[4]); MPI_CURRENT_TYPE2 = decode_datatype(action[5]); } else { MPI_CURRENT_TYPE = MPI_DEFAULT_TYPE; MPI_CURRENT_TYPE2 = MPI_DEFAULT_TYPE; } void *sendbuf = smpi_get_tmp_sendbuffer(sendcount* smpi_datatype_size(MPI_CURRENT_TYPE)); void *recvbuf = smpi_get_tmp_recvbuffer(recvcount* smpi_datatype_size(MPI_CURRENT_TYPE2)); int rank = smpi_process_index(); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_ALLGATHER; extra->send_size = sendcount; extra->recv_size= recvcount; extra->datatype1 = encode_datatype(MPI_CURRENT_TYPE, nullptr); extra->datatype2 = encode_datatype(MPI_CURRENT_TYPE2, nullptr); extra->num_processes = smpi_comm_size(MPI_COMM_WORLD); TRACE_smpi_collective_in(rank, -1, __FUNCTION__,extra); mpi_coll_allgather_fun(sendbuf, sendcount, MPI_CURRENT_TYPE, recvbuf, recvcount, MPI_CURRENT_TYPE2, MPI_COMM_WORLD); TRACE_smpi_collective_out(rank, -1, __FUNCTION__); log_timed_action (action, clock); } static void action_allgatherv(const char *const *action) { /* The structure of the allgatherv action for the rank 0 (total 4 processes) is the following: 0 allGatherV 275427 275427 275427 275427 204020 where: 1) 275427 is the sendcount 2) The next four elements declare the recvcounts array 3) No more values mean that the datatype for sent and receive buffer is the default one, see decode_datatype(). */ double clock = smpi_process_simulated_elapsed(); int comm_size = smpi_comm_size(MPI_COMM_WORLD); CHECK_ACTION_PARAMS(action, comm_size+1, 2) int i=0; int sendcount=atoi(action[2]); int *recvcounts = xbt_new0(int, comm_size); int *disps = xbt_new0(int, comm_size); int recv_sum=0; MPI_Datatype MPI_CURRENT_TYPE2; if(action[3+comm_size] && action[4+comm_size]) { MPI_CURRENT_TYPE = decode_datatype(action[3+comm_size]); MPI_CURRENT_TYPE2 = decode_datatype(action[4+comm_size]); } else { MPI_CURRENT_TYPE = MPI_DEFAULT_TYPE; MPI_CURRENT_TYPE2 = MPI_DEFAULT_TYPE; } void *sendbuf = smpi_get_tmp_sendbuffer(sendcount* smpi_datatype_size(MPI_CURRENT_TYPE)); for(i=0;itype = TRACING_ALLGATHERV; extra->send_size = sendcount; extra->recvcounts= xbt_new(int, comm_size); for(i=0; i< comm_size; i++)//copy data to avoid bad free extra->recvcounts[i] = recvcounts[i]; extra->datatype1 = encode_datatype(MPI_CURRENT_TYPE, nullptr); extra->datatype2 = encode_datatype(MPI_CURRENT_TYPE2, nullptr); extra->num_processes = comm_size; TRACE_smpi_collective_in(rank, -1, __FUNCTION__,extra); mpi_coll_allgatherv_fun(sendbuf, sendcount, MPI_CURRENT_TYPE, recvbuf, recvcounts, disps, MPI_CURRENT_TYPE2, MPI_COMM_WORLD); TRACE_smpi_collective_out(rank, -1, __FUNCTION__); log_timed_action (action, clock); xbt_free(recvcounts); xbt_free(disps); } static void action_allToAllv(const char *const *action) { /* The structure of the allToAllV action for the rank 0 (total 4 processes) is the following: 0 allToAllV 100 1 7 10 12 100 1 70 10 5 where: 1) 100 is the size of the send buffer *sizeof(int), 2) 1 7 10 12 is the sendcounts array 3) 100*sizeof(int) is the size of the receiver buffer 4) 1 70 10 5 is the recvcounts array */ double clock = smpi_process_simulated_elapsed(); int comm_size = smpi_comm_size(MPI_COMM_WORLD); CHECK_ACTION_PARAMS(action, 2*comm_size+2, 2) int *sendcounts = xbt_new0(int, comm_size); int *recvcounts = xbt_new0(int, comm_size); int *senddisps = xbt_new0(int, comm_size); int *recvdisps = xbt_new0(int, comm_size); MPI_Datatype MPI_CURRENT_TYPE2; int send_buf_size=parse_double(action[2]); int recv_buf_size=parse_double(action[3+comm_size]); if(action[4+2*comm_size] && action[5+2*comm_size]) { MPI_CURRENT_TYPE=decode_datatype(action[4+2*comm_size]); MPI_CURRENT_TYPE2=decode_datatype(action[5+2*comm_size]); } else{ MPI_CURRENT_TYPE=MPI_DEFAULT_TYPE; MPI_CURRENT_TYPE2=MPI_DEFAULT_TYPE; } void *sendbuf = smpi_get_tmp_sendbuffer(send_buf_size* smpi_datatype_size(MPI_CURRENT_TYPE)); void *recvbuf = smpi_get_tmp_recvbuffer(recv_buf_size* smpi_datatype_size(MPI_CURRENT_TYPE2)); for(int i=0;itype = TRACING_ALLTOALLV; extra->recvcounts= xbt_new(int, comm_size); extra->sendcounts= xbt_new(int, comm_size); extra->num_processes = comm_size; for(int i=0; i< comm_size; i++){//copy data to avoid bad free extra->send_size += sendcounts[i]; extra->sendcounts[i] = sendcounts[i]; extra->recv_size += recvcounts[i]; extra->recvcounts[i] = recvcounts[i]; } extra->datatype1 = encode_datatype(MPI_CURRENT_TYPE, nullptr); extra->datatype2 = encode_datatype(MPI_CURRENT_TYPE2, nullptr); TRACE_smpi_collective_in(rank, -1, __FUNCTION__,extra); mpi_coll_alltoallv_fun(sendbuf, sendcounts, senddisps, MPI_CURRENT_TYPE,recvbuf, recvcounts, recvdisps, MPI_CURRENT_TYPE, MPI_COMM_WORLD); TRACE_smpi_collective_out(rank, -1, __FUNCTION__); log_timed_action (action, clock); xbt_free(sendcounts); xbt_free(recvcounts); xbt_free(senddisps); xbt_free(recvdisps); } void smpi_replay_run(int *argc, char***argv){ /* First initializes everything */ smpi_process_init(argc, argv); smpi_process_mark_as_initialized(); smpi_process_set_replaying(true); int rank = smpi_process_index(); TRACE_smpi_init(rank); TRACE_smpi_computing_init(rank); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_INIT; char *operation =bprintf("%s_init",__FUNCTION__); TRACE_smpi_collective_in(rank, -1, operation, extra); TRACE_smpi_collective_out(rank, -1, operation); xbt_free(operation); if (_xbt_replay_action_init()==0) { xbt_replay_action_register("init", action_init); xbt_replay_action_register("finalize", action_finalize); xbt_replay_action_register("comm_size", action_comm_size); xbt_replay_action_register("comm_split", action_comm_split); xbt_replay_action_register("comm_dup", action_comm_dup); xbt_replay_action_register("send", action_send); xbt_replay_action_register("Isend", action_Isend); xbt_replay_action_register("recv", action_recv); xbt_replay_action_register("Irecv", action_Irecv); xbt_replay_action_register("test", action_test); xbt_replay_action_register("wait", action_wait); xbt_replay_action_register("waitAll", action_waitall); xbt_replay_action_register("barrier", action_barrier); xbt_replay_action_register("bcast", action_bcast); xbt_replay_action_register("reduce", action_reduce); xbt_replay_action_register("allReduce", action_allReduce); xbt_replay_action_register("allToAll", action_allToAll); xbt_replay_action_register("allToAllV", action_allToAllv); xbt_replay_action_register("gather", action_gather); xbt_replay_action_register("gatherV", action_gatherv); xbt_replay_action_register("allGather", action_allgather); xbt_replay_action_register("allGatherV", action_allgatherv); xbt_replay_action_register("reduceScatter", action_reducescatter); xbt_replay_action_register("compute", action_compute); } //if we have a delayed start, sleep here. if(*argc>2){ char *endptr; double value = strtod((*argv)[2], &endptr); if (*endptr != '\0') THROWF(unknown_error, 0, "%s is not a double", (*argv)[2]); XBT_VERB("Delayed start for instance - Sleeping for %f flops ",value ); smpi_execute_flops(value); } else { //UGLY: force a context switch to be sure that all MSG_processes begin initialization XBT_DEBUG("Force context switch by smpi_execute_flops - Sleeping for 0.0 flops "); smpi_execute_flops(0.0); } /* Actually run the replay */ xbt_replay_action_runner(*argc, *argv); /* and now, finalize everything */ double sim_time= 1.; /* One active process will stop. Decrease the counter*/ XBT_DEBUG("There are %zu elements in reqq[*]", get_reqq_self()->size()); if (!get_reqq_self()->empty()){ unsigned int count_requests=get_reqq_self()->size(); MPI_Request requests[count_requests]; MPI_Status status[count_requests]; unsigned int i=0; for (auto req: *get_reqq_self()){ requests[i] = req; i++; } smpi_mpi_waitall(count_requests, requests, status); active_processes--; } else { active_processes--; } if(active_processes==0){ /* Last process alive speaking */ /* end the simulated timer */ sim_time = smpi_process_simulated_elapsed(); XBT_INFO("Simulation time %f", sim_time); _xbt_replay_action_exit(); xbt_free(sendbuffer); xbt_free(recvbuffer); } instr_extra_data extra_fin = xbt_new0(s_instr_extra_data_t,1); extra_fin->type = TRACING_FINALIZE; operation =bprintf("%s_finalize",__FUNCTION__); TRACE_smpi_collective_in(rank, -1, operation, extra_fin); smpi_process_finalize(); TRACE_smpi_collective_out(rank, -1, operation); TRACE_smpi_finalize(smpi_process_index()); smpi_process_destroy(); xbt_free(operation); } SimGrid-3.14.159/src/smpi/smpi_mpi_dt_private.h000644 001750 001750 00000015613 13031031273 023045 0ustar00mquinsonmquinson000000 000000 /* smpi_mpi_dt_private.h -- functions of smpi_mpi_dt.c that are exported to other SMPI modules. */ /* Copyright (c) 2009-2010, 2012-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_DT_PRIVATE_H #define SMPI_DT_PRIVATE_H #include #include "private.h" SG_BEGIN_DECL() #define DT_FLAG_DESTROYED 0x0001 /**< user destroyed but some other layers still have a reference */ #define DT_FLAG_COMMITED 0x0002 /**< ready to be used for a send/recv operation */ #define DT_FLAG_CONTIGUOUS 0x0004 /**< contiguous datatype */ #define DT_FLAG_OVERLAP 0x0008 /**< datatype is unpropper for a recv operation */ #define DT_FLAG_USER_LB 0x0010 /**< has a user defined LB */ #define DT_FLAG_USER_UB 0x0020 /**< has a user defined UB */ #define DT_FLAG_PREDEFINED 0x0040 /**< cannot be removed: initial and predefined datatypes */ #define DT_FLAG_NO_GAPS 0x0080 /**< no gaps around the datatype */ #define DT_FLAG_DATA 0x0100 /**< data or control structure */ #define DT_FLAG_ONE_SIDED 0x0200 /**< datatype can be used for one sided operations */ #define DT_FLAG_UNAVAILABLE 0x0400 /**< datatypes unavailable on the build (OS or compiler dependant) */ #define DT_FLAG_VECTOR 0x0800 /**< valid only for loops. The loop contain only one element **< without extent. It correspond to the vector type. */ /* * We should make the difference here between the predefined contiguous and non contiguous * datatypes. The DT_FLAG_BASIC is held by all predefined contiguous datatypes. */ #define DT_FLAG_BASIC (DT_FLAG_PREDEFINED | DT_FLAG_CONTIGUOUS | DT_FLAG_NO_GAPS | DT_FLAG_DATA | DT_FLAG_COMMITED) extern const MPI_Datatype MPI_PTR; /* Structures that handle complex data type information, used for serialization/unserialization of messages */ typedef struct s_smpi_mpi_contiguous{ s_smpi_subtype_t base; MPI_Datatype old_type; MPI_Aint lb; size_t size_oldtype; int block_count; } s_smpi_mpi_contiguous_t; typedef struct s_smpi_mpi_vector{ s_smpi_subtype_t base; MPI_Datatype old_type; size_t size_oldtype; int block_stride; int block_length; int block_count; } s_smpi_mpi_vector_t; typedef struct s_smpi_mpi_hvector{ s_smpi_subtype_t base; MPI_Datatype old_type; size_t size_oldtype; MPI_Aint block_stride; int block_length; int block_count; } s_smpi_mpi_hvector_t; typedef struct s_smpi_mpi_indexed{ s_smpi_subtype_t base; MPI_Datatype old_type; size_t size_oldtype; int* block_lengths; int* block_indices; int block_count; } s_smpi_mpi_indexed_t; typedef struct s_smpi_mpi_hindexed{ s_smpi_subtype_t base; MPI_Datatype old_type; size_t size_oldtype; int* block_lengths; MPI_Aint* block_indices; int block_count; } s_smpi_mpi_hindexed_t; typedef struct s_smpi_mpi_struct{ s_smpi_subtype_t base; MPI_Datatype old_type; size_t size_oldtype; int* block_lengths; MPI_Aint* block_indices; int block_count; MPI_Datatype* old_types; } s_smpi_mpi_struct_t; /* Functions to handle serialization/unserialization of messages, 3 for each type of MPI_Type One for creating the substructure to handle, one for serialization, one for unserialization */ XBT_PRIVATE void unserialize_contiguous( const void *contiguous_vector, void *noncontiguous_vector, int count, void *type, MPI_Op op); XBT_PRIVATE void serialize_contiguous( const void *noncontiguous_vector, void *contiguous_vector, int count,void *type); XBT_PRIVATE void free_contiguous(MPI_Datatype* type); XBT_PRIVATE void use_contiguous(MPI_Datatype* type); XBT_PRIVATE s_smpi_mpi_contiguous_t* smpi_datatype_contiguous_create( MPI_Aint lb, int block_count, MPI_Datatype old_type, int size_oldtype); XBT_PRIVATE void unserialize_vector( const void *contiguous_vector, void *noncontiguous_vector, int count,void *type, MPI_Op op); XBT_PRIVATE void serialize_vector( const void *noncontiguous_vector, void *contiguous_vector, int count, void *type); XBT_PRIVATE void free_vector(MPI_Datatype* type); XBT_PRIVATE void use_vector(MPI_Datatype* type); XBT_PRIVATE s_smpi_mpi_vector_t* smpi_datatype_vector_create( int block_stride, int block_length, int block_count, MPI_Datatype old_type, int size_oldtype); XBT_PRIVATE void unserialize_hvector( const void *contiguous_vector, void *noncontiguous_vector, int count, void *type, MPI_Op op); XBT_PRIVATE void serialize_hvector( const void *noncontiguous_vector, void *contiguous_vector, int count, void *type); XBT_PRIVATE void free_hvector(MPI_Datatype* type); XBT_PRIVATE void use_hvector(MPI_Datatype* type); XBT_PRIVATE s_smpi_mpi_hvector_t* smpi_datatype_hvector_create( MPI_Aint block_stride, int block_length, int block_count, MPI_Datatype old_type, int size_oldtype); XBT_PRIVATE void unserialize_indexed( const void *contiguous_indexed, void *noncontiguous_indexed, int count, void *type, MPI_Op op); XBT_PRIVATE void serialize_indexed( const void *noncontiguous_vector, void *contiguous_vector, int count, void *type); XBT_PRIVATE void free_indexed(MPI_Datatype* type); XBT_PRIVATE void use_indexed(MPI_Datatype* type); XBT_PRIVATE s_smpi_mpi_indexed_t* smpi_datatype_indexed_create(int* block_lengths, int* block_indices, int block_count, MPI_Datatype old_type, int size_oldtype); XBT_PRIVATE void unserialize_hindexed( const void *contiguous_indexed, void *noncontiguous_indexed, int count, void *type, MPI_Op op); XBT_PRIVATE void serialize_hindexed( const void *noncontiguous_vector, void *contiguous_vector, int count, void *type); XBT_PRIVATE void free_hindexed(MPI_Datatype* type); XBT_PRIVATE void use_hindexed(MPI_Datatype* type); XBT_PRIVATE s_smpi_mpi_hindexed_t* smpi_datatype_hindexed_create(int* block_lengths, MPI_Aint* block_indices, int block_count, MPI_Datatype old_type, int size_oldtype); XBT_PRIVATE void unserialize_struct( const void *contiguous_indexed, void *noncontiguous_indexed, int count, void *type, MPI_Op op); XBT_PRIVATE void serialize_struct( const void *noncontiguous_vector, void *contiguous_vector, int count, void *type); XBT_PRIVATE void free_struct(MPI_Datatype* type); XBT_PRIVATE void use_struct(MPI_Datatype* type); XBT_PRIVATE s_smpi_mpi_struct_t* smpi_datatype_struct_create(int* block_lengths, MPI_Aint* block_indices, int block_count, MPI_Datatype* old_types); SG_END_DECL() #endif SimGrid-3.14.159/src/smpi/smpi_pmpi.cpp000644 001750 001750 00000265740 13031031272 021346 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2016. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include "private.h" #include "smpi_mpi_dt_private.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_pmpi, smpi, "Logging specific to SMPI (pmpi)"); //this function need to be here because of the calls to smpi_bench void TRACE_smpi_set_category(const char *category) { //need to end bench otherwise categories for execution tasks are wrong smpi_bench_end(); TRACE_internal_smpi_set_category (category); //begin bench after changing process's category smpi_bench_begin(); } /* PMPI User level calls */ int PMPI_Init(int *argc, char ***argv) { // PMPI_Init is call only one time by only by SMPI process int already_init; MPI_Initialized(&already_init); if(already_init == 0){ smpi_process_init(argc, argv); smpi_process_mark_as_initialized(); int rank = smpi_process_index(); TRACE_smpi_init(rank); TRACE_smpi_computing_init(rank); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_INIT; TRACE_smpi_collective_in(rank, -1, __FUNCTION__, extra); TRACE_smpi_collective_out(rank, -1, __FUNCTION__); smpi_bench_begin(); } smpi_mpi_init(); return MPI_SUCCESS; } int PMPI_Finalize() { smpi_bench_end(); int rank = smpi_process_index(); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_FINALIZE; TRACE_smpi_collective_in(rank, -1, __FUNCTION__, extra); smpi_process_finalize(); TRACE_smpi_collective_out(rank, -1, __FUNCTION__); TRACE_smpi_finalize(smpi_process_index()); smpi_process_destroy(); return MPI_SUCCESS; } int PMPI_Finalized(int* flag) { *flag=smpi_process_finalized(); return MPI_SUCCESS; } int PMPI_Get_version (int *version,int *subversion){ *version = MPI_VERSION; *subversion= MPI_SUBVERSION; return MPI_SUCCESS; } int PMPI_Get_library_version (char *version,int *len){ int retval = MPI_SUCCESS; smpi_bench_end(); snprintf(version,MPI_MAX_LIBRARY_VERSION_STRING,"SMPI Version %d.%d. Copyright The Simgrid Team 2007-2015", SIMGRID_VERSION_MAJOR, SIMGRID_VERSION_MINOR); *len = strlen(version) > MPI_MAX_LIBRARY_VERSION_STRING ? MPI_MAX_LIBRARY_VERSION_STRING : strlen(version); smpi_bench_begin(); return retval; } int PMPI_Init_thread(int *argc, char ***argv, int required, int *provided) { if (provided != nullptr) { *provided = MPI_THREAD_SINGLE; } return MPI_Init(argc, argv); } int PMPI_Query_thread(int *provided) { int retval = 0; if (provided == nullptr) { retval = MPI_ERR_ARG; } else { *provided = MPI_THREAD_SINGLE; retval = MPI_SUCCESS; } return retval; } int PMPI_Is_thread_main(int *flag) { int retval = 0; if (flag == nullptr) { retval = MPI_ERR_ARG; } else { *flag = smpi_process_index() == 0; retval = MPI_SUCCESS; } return retval; } int PMPI_Abort(MPI_Comm comm, int errorcode) { smpi_bench_end(); smpi_process_destroy(); // FIXME: should kill all processes in comm instead simcall_process_kill(SIMIX_process_self()); return MPI_SUCCESS; } double PMPI_Wtime() { return smpi_mpi_wtime(); } extern double sg_maxmin_precision; double PMPI_Wtick() { return sg_maxmin_precision; } int PMPI_Address(void *location, MPI_Aint * address) { int retval = 0; if (address==nullptr) { retval = MPI_ERR_ARG; } else { *address = reinterpret_cast(location); retval = MPI_SUCCESS; } return retval; } int PMPI_Get_address(void *location, MPI_Aint * address) { return PMPI_Address(location, address); } int PMPI_Type_free(MPI_Datatype * datatype) { int retval = 0; /* Free a predefined datatype is an error according to the standard, and should be checked for */ if (*datatype == MPI_DATATYPE_NULL) { retval = MPI_ERR_ARG; } else { smpi_datatype_unuse(*datatype); retval = MPI_SUCCESS; } return retval; } int PMPI_Type_size(MPI_Datatype datatype, int *size) { int retval = 0; if (datatype == MPI_DATATYPE_NULL) { retval = MPI_ERR_TYPE; } else if (size == nullptr) { retval = MPI_ERR_ARG; } else { *size = static_cast(smpi_datatype_size(datatype)); retval = MPI_SUCCESS; } return retval; } int PMPI_Type_get_extent(MPI_Datatype datatype, MPI_Aint * lb, MPI_Aint * extent) { int retval = 0; if (datatype == MPI_DATATYPE_NULL) { retval = MPI_ERR_TYPE; } else if (lb == nullptr || extent == nullptr) { retval = MPI_ERR_ARG; } else { retval = smpi_datatype_extent(datatype, lb, extent); } return retval; } int PMPI_Type_get_true_extent(MPI_Datatype datatype, MPI_Aint * lb, MPI_Aint * extent) { return PMPI_Type_get_extent(datatype, lb, extent); } int PMPI_Type_extent(MPI_Datatype datatype, MPI_Aint * extent) { int retval = 0; if (datatype == MPI_DATATYPE_NULL) { retval = MPI_ERR_TYPE; } else if (extent == nullptr) { retval = MPI_ERR_ARG; } else { *extent = smpi_datatype_get_extent(datatype); retval = MPI_SUCCESS; } return retval; } int PMPI_Type_lb(MPI_Datatype datatype, MPI_Aint * disp) { int retval = 0; if (datatype == MPI_DATATYPE_NULL) { retval = MPI_ERR_TYPE; } else if (disp == nullptr) { retval = MPI_ERR_ARG; } else { *disp = smpi_datatype_lb(datatype); retval = MPI_SUCCESS; } return retval; } int PMPI_Type_ub(MPI_Datatype datatype, MPI_Aint * disp) { int retval = 0; if (datatype == MPI_DATATYPE_NULL) { retval = MPI_ERR_TYPE; } else if (disp == nullptr) { retval = MPI_ERR_ARG; } else { *disp = smpi_datatype_ub(datatype); retval = MPI_SUCCESS; } return retval; } int PMPI_Type_dup(MPI_Datatype datatype, MPI_Datatype *newtype){ int retval = 0; if (datatype == MPI_DATATYPE_NULL) { retval = MPI_ERR_TYPE; } else { retval = smpi_datatype_dup(datatype, newtype); } return retval; } int PMPI_Op_create(MPI_User_function * function, int commute, MPI_Op * op) { int retval = 0; if (function == nullptr || op == nullptr) { retval = MPI_ERR_ARG; } else { *op = smpi_op_new(function, (commute!=0)); retval = MPI_SUCCESS; } return retval; } int PMPI_Op_free(MPI_Op * op) { int retval = 0; if (op == nullptr) { retval = MPI_ERR_ARG; } else if (*op == MPI_OP_NULL) { retval = MPI_ERR_OP; } else { smpi_op_destroy(*op); *op = MPI_OP_NULL; retval = MPI_SUCCESS; } return retval; } int PMPI_Group_free(MPI_Group * group) { int retval = 0; if (group == nullptr) { retval = MPI_ERR_ARG; } else { smpi_group_destroy(*group); *group = MPI_GROUP_NULL; retval = MPI_SUCCESS; } return retval; } int PMPI_Group_size(MPI_Group group, int *size) { int retval = 0; if (group == MPI_GROUP_NULL) { retval = MPI_ERR_GROUP; } else if (size == nullptr) { retval = MPI_ERR_ARG; } else { *size = smpi_group_size(group); retval = MPI_SUCCESS; } return retval; } int PMPI_Group_rank(MPI_Group group, int *rank) { int retval = 0; if (group == MPI_GROUP_NULL) { retval = MPI_ERR_GROUP; } else if (rank == nullptr) { retval = MPI_ERR_ARG; } else { *rank = smpi_group_rank(group, smpi_process_index()); retval = MPI_SUCCESS; } return retval; } int PMPI_Group_translate_ranks(MPI_Group group1, int n, int *ranks1, MPI_Group group2, int *ranks2) { int retval, i, index; if (group1 == MPI_GROUP_NULL || group2 == MPI_GROUP_NULL) { retval = MPI_ERR_GROUP; } else { for (i = 0; i < n; i++) { if(ranks1[i]==MPI_PROC_NULL){ ranks2[i]=MPI_PROC_NULL; }else{ index = smpi_group_index(group1, ranks1[i]); ranks2[i] = smpi_group_rank(group2, index); } } retval = MPI_SUCCESS; } return retval; } int PMPI_Group_compare(MPI_Group group1, MPI_Group group2, int *result) { int retval = 0; if (group1 == MPI_GROUP_NULL || group2 == MPI_GROUP_NULL) { retval = MPI_ERR_GROUP; } else if (result == nullptr) { retval = MPI_ERR_ARG; } else { *result = smpi_group_compare(group1, group2); retval = MPI_SUCCESS; } return retval; } int PMPI_Group_union(MPI_Group group1, MPI_Group group2, MPI_Group * newgroup) { int retval, i, proc1, proc2, size, size2; if (group1 == MPI_GROUP_NULL || group2 == MPI_GROUP_NULL) { retval = MPI_ERR_GROUP; } else if (newgroup == nullptr) { retval = MPI_ERR_ARG; } else { size = smpi_group_size(group1); size2 = smpi_group_size(group2); for (i = 0; i < size2; i++) { proc2 = smpi_group_index(group2, i); proc1 = smpi_group_rank(group1, proc2); if (proc1 == MPI_UNDEFINED) { size++; } } if (size == 0) { *newgroup = MPI_GROUP_EMPTY; } else { *newgroup = smpi_group_new(size); size2 = smpi_group_size(group1); for (i = 0; i < size2; i++) { proc1 = smpi_group_index(group1, i); smpi_group_set_mapping(*newgroup, proc1, i); } for (i = size2; i < size; i++) { proc2 = smpi_group_index(group2, i - size2); smpi_group_set_mapping(*newgroup, proc2, i); } } retval = MPI_SUCCESS; } return retval; } int PMPI_Group_intersection(MPI_Group group1, MPI_Group group2, MPI_Group * newgroup) { int retval, i, proc1, proc2, size; if (group1 == MPI_GROUP_NULL || group2 == MPI_GROUP_NULL) { retval = MPI_ERR_GROUP; } else if (newgroup == nullptr) { retval = MPI_ERR_ARG; } else { size = smpi_group_size(group2); for (i = 0; i < size; i++) { proc2 = smpi_group_index(group2, i); proc1 = smpi_group_rank(group1, proc2); if (proc1 == MPI_UNDEFINED) { size--; } } if (size == 0) { *newgroup = MPI_GROUP_EMPTY; } else { *newgroup = smpi_group_new(size); int j=0; for (i = 0; i < smpi_group_size(group2); i++) { proc2 = smpi_group_index(group2, i); proc1 = smpi_group_rank(group1, proc2); if (proc1 != MPI_UNDEFINED) { smpi_group_set_mapping(*newgroup, proc2, j); j++; } } } retval = MPI_SUCCESS; } return retval; } int PMPI_Group_difference(MPI_Group group1, MPI_Group group2, MPI_Group * newgroup) { int retval, i, proc1, proc2, size, size2; if (group1 == MPI_GROUP_NULL || group2 == MPI_GROUP_NULL) { retval = MPI_ERR_GROUP; } else if (newgroup == nullptr) { retval = MPI_ERR_ARG; } else { size = smpi_group_size(group1); size2 = size; for (i = 0; i < size2; i++) { proc1 = smpi_group_index(group1, i); proc2 = smpi_group_rank(group2, proc1); if (proc2 != MPI_UNDEFINED) { size--; } } if (size == 0) { *newgroup = MPI_GROUP_EMPTY; } else { *newgroup = smpi_group_new(size); for (i = 0; i < size2; i++) { proc1 = smpi_group_index(group1, i); proc2 = smpi_group_rank(group2, proc1); if (proc2 == MPI_UNDEFINED) { smpi_group_set_mapping(*newgroup, proc1, i); } } } retval = MPI_SUCCESS; } return retval; } int PMPI_Group_incl(MPI_Group group, int n, int *ranks, MPI_Group * newgroup) { int retval; if (group == MPI_GROUP_NULL) { retval = MPI_ERR_GROUP; } else if (newgroup == nullptr) { retval = MPI_ERR_ARG; } else { retval = smpi_group_incl(group, n, ranks, newgroup); } return retval; } int PMPI_Group_excl(MPI_Group group, int n, int *ranks, MPI_Group * newgroup) { int retval, i, j, newsize, oldsize, index; if (group == MPI_GROUP_NULL) { retval = MPI_ERR_GROUP; } else if (newgroup == nullptr) { retval = MPI_ERR_ARG; } else { if (n == 0) { *newgroup = group; if (group != smpi_comm_group(MPI_COMM_WORLD) && group != smpi_comm_group(MPI_COMM_SELF) && group != MPI_GROUP_EMPTY) smpi_group_use(group); } else if (n == smpi_group_size(group)) { *newgroup = MPI_GROUP_EMPTY; } else { oldsize=smpi_group_size(group); newsize = oldsize - n; *newgroup = smpi_group_new(newsize); int* to_exclude=xbt_new0(int, smpi_group_size(group)); for(i=0; i= 0 && rank < smpi_group_size(group); /* Last */ ) { size++; if(rank == ranges[i][1]){/*already last ?*/ break; } rank += ranges[i][2]; /* Stride */ if (ranges[i][0] ranges[i][1]) break; }else{ if(rank < ranges[i][1]) break; } } } *newgroup = smpi_group_new(size); j = 0; for (i = 0; i < n; i++) { for (rank = ranges[i][0]; /* First */ rank >= 0 && rank < smpi_group_size(group); /* Last */ ) { index = smpi_group_index(group, rank); smpi_group_set_mapping(*newgroup, index, j); j++; if(rank == ranges[i][1]){/*already last ?*/ break; } rank += ranges[i][2]; /* Stride */ if (ranges[i][0] ranges[i][1]) break; }else{ if(rank < ranges[i][1]) break; } } } } retval = MPI_SUCCESS; } return retval; } int PMPI_Group_range_excl(MPI_Group group, int n, int ranges[][3], MPI_Group * newgroup) { int retval, i, rank, newrank,oldrank, size, index, add; if (group == MPI_GROUP_NULL) { retval = MPI_ERR_GROUP; } else if (newgroup == nullptr) { retval = MPI_ERR_ARG; } else { if (n == 0) { *newgroup = group; if (group!= smpi_comm_group(MPI_COMM_WORLD) && group != smpi_comm_group(MPI_COMM_SELF) && group != MPI_GROUP_EMPTY) smpi_group_use(group); } else { size = smpi_group_size(group); for (i = 0; i < n; i++) { for (rank = ranges[i][0]; /* First */ rank >= 0 && rank < smpi_group_size(group); /* Last */ ) { size--; if(rank == ranges[i][1]){/*already last ?*/ break; } rank += ranges[i][2]; /* Stride */ if (ranges[i][0] ranges[i][1]) break; }else{ if(rank < ranges[i][1]) break; } } } if (size == 0) { *newgroup = MPI_GROUP_EMPTY; } else { *newgroup = smpi_group_new(size); newrank=0; oldrank=0; while (newrank < size) { add=1; for (i = 0; i < n; i++) { for (rank = ranges[i][0]; rank >= 0 && rank < smpi_group_size(group); ){ if(rank==oldrank){ add=0; break; } if(rank == ranges[i][1]){/*already last ?*/ break; } rank += ranges[i][2]; /* Stride */ if (ranges[i][0] ranges[i][1]) break; }else{ if(rank < ranges[i][1]) break; } } } if(add==1){ index = smpi_group_index(group, oldrank); smpi_group_set_mapping(*newgroup, index, newrank); newrank++; } oldrank++; } } } retval = MPI_SUCCESS; } return retval; } int PMPI_Comm_rank(MPI_Comm comm, int *rank) { int retval = 0; if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (rank == nullptr) { retval = MPI_ERR_ARG; } else { *rank = smpi_comm_rank(comm); retval = MPI_SUCCESS; } return retval; } int PMPI_Comm_size(MPI_Comm comm, int *size) { int retval = 0; if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (size == nullptr) { retval = MPI_ERR_ARG; } else { *size = smpi_comm_size(comm); retval = MPI_SUCCESS; } return retval; } int PMPI_Comm_get_name (MPI_Comm comm, char* name, int* len) { int retval = 0; if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (name == nullptr || len == nullptr) { retval = MPI_ERR_ARG; } else { smpi_comm_get_name(comm, name, len); retval = MPI_SUCCESS; } return retval; } int PMPI_Comm_group(MPI_Comm comm, MPI_Group * group) { int retval = 0; if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (group == nullptr) { retval = MPI_ERR_ARG; } else { *group = smpi_comm_group(comm); if(*group!= smpi_comm_group(MPI_COMM_WORLD) && *group != MPI_GROUP_NULL && *group != MPI_GROUP_EMPTY) smpi_group_use(*group); retval = MPI_SUCCESS; } return retval; } int PMPI_Comm_compare(MPI_Comm comm1, MPI_Comm comm2, int *result) { int retval = 0; if (comm1 == MPI_COMM_NULL || comm2 == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (result == nullptr) { retval = MPI_ERR_ARG; } else { if (comm1 == comm2) { /* Same communicators means same groups */ *result = MPI_IDENT; } else { *result = smpi_group_compare(smpi_comm_group(comm1), smpi_comm_group(comm2)); if (*result == MPI_IDENT) { *result = MPI_CONGRUENT; } } retval = MPI_SUCCESS; } return retval; } int PMPI_Comm_dup(MPI_Comm comm, MPI_Comm * newcomm) { int retval = 0; if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (newcomm == nullptr) { retval = MPI_ERR_ARG; } else { retval = smpi_comm_dup(comm, newcomm); } return retval; } int PMPI_Comm_create(MPI_Comm comm, MPI_Group group, MPI_Comm * newcomm) { int retval = 0; if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (group == MPI_GROUP_NULL) { retval = MPI_ERR_GROUP; } else if (newcomm == nullptr) { retval = MPI_ERR_ARG; } else if(smpi_group_rank(group,smpi_process_index())==MPI_UNDEFINED){ *newcomm= MPI_COMM_NULL; retval = MPI_SUCCESS; }else{ smpi_group_use(group); *newcomm = smpi_comm_new(group, nullptr); retval = MPI_SUCCESS; } return retval; } int PMPI_Comm_free(MPI_Comm * comm) { int retval = 0; if (comm == nullptr) { retval = MPI_ERR_ARG; } else if (*comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else { smpi_comm_destroy(*comm); *comm = MPI_COMM_NULL; retval = MPI_SUCCESS; } return retval; } int PMPI_Comm_disconnect(MPI_Comm * comm) { /* TODO: wait until all communication in comm are done */ int retval = 0; if (comm == nullptr) { retval = MPI_ERR_ARG; } else if (*comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else { smpi_comm_destroy(*comm); *comm = MPI_COMM_NULL; retval = MPI_SUCCESS; } return retval; } int PMPI_Comm_split(MPI_Comm comm, int color, int key, MPI_Comm* comm_out) { int retval = 0; smpi_bench_end(); if (comm_out == nullptr) { retval = MPI_ERR_ARG; } else if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else { *comm_out = smpi_comm_split(comm, color, key); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Send_init(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm, MPI_Request * request) { int retval = 0; smpi_bench_end(); if (request == nullptr) { retval = MPI_ERR_ARG; } else if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (!is_datatype_valid(datatype)) { retval = MPI_ERR_TYPE; } else if (dst == MPI_PROC_NULL) { retval = MPI_SUCCESS; } else { *request = smpi_mpi_send_init(buf, count, datatype, dst, tag, comm); retval = MPI_SUCCESS; } smpi_bench_begin(); if (retval != MPI_SUCCESS && request != nullptr) *request = MPI_REQUEST_NULL; return retval; } int PMPI_Recv_init(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm, MPI_Request * request) { int retval = 0; smpi_bench_end(); if (request == nullptr) { retval = MPI_ERR_ARG; } else if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (!is_datatype_valid(datatype)) { retval = MPI_ERR_TYPE; } else if (src == MPI_PROC_NULL) { retval = MPI_SUCCESS; } else { *request = smpi_mpi_recv_init(buf, count, datatype, src, tag, comm); retval = MPI_SUCCESS; } smpi_bench_begin(); if (retval != MPI_SUCCESS && request != nullptr) *request = MPI_REQUEST_NULL; return retval; } int PMPI_Ssend_init(void* buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm, MPI_Request* request) { int retval = 0; smpi_bench_end(); if (request == nullptr) { retval = MPI_ERR_ARG; } else if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (!is_datatype_valid(datatype)) { retval = MPI_ERR_TYPE; } else if (dst == MPI_PROC_NULL) { retval = MPI_SUCCESS; } else { *request = smpi_mpi_ssend_init(buf, count, datatype, dst, tag, comm); retval = MPI_SUCCESS; } smpi_bench_begin(); if (retval != MPI_SUCCESS && request != nullptr) *request = MPI_REQUEST_NULL; return retval; } int PMPI_Start(MPI_Request * request) { int retval = 0; smpi_bench_end(); if (request == nullptr || *request == MPI_REQUEST_NULL) { retval = MPI_ERR_REQUEST; } else { smpi_mpi_start(*request); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Startall(int count, MPI_Request * requests) { int retval; int i = 0; smpi_bench_end(); if (requests == nullptr) { retval = MPI_ERR_ARG; } else { retval = MPI_SUCCESS; for (i = 0 ; i < count ; i++) { if(requests[i] == MPI_REQUEST_NULL) { retval = MPI_ERR_REQUEST; } } if(retval != MPI_ERR_REQUEST) { smpi_mpi_startall(count, requests); } } smpi_bench_begin(); return retval; } int PMPI_Request_free(MPI_Request * request) { int retval = 0; smpi_bench_end(); if (*request == MPI_REQUEST_NULL) { retval = MPI_ERR_ARG; } else { smpi_mpi_request_free(request); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Irecv(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm, MPI_Request * request) { int retval = 0; smpi_bench_end(); if (request == nullptr) { retval = MPI_ERR_ARG; } else if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (src == MPI_PROC_NULL) { *request = MPI_REQUEST_NULL; retval = MPI_SUCCESS; } else if (src!=MPI_ANY_SOURCE && (src >= smpi_group_size(smpi_comm_group(comm)) || src <0)){ retval = MPI_ERR_RANK; } else if ((count < 0) || (buf==nullptr && count > 0)) { retval = MPI_ERR_COUNT; } else if (!is_datatype_valid(datatype)) { retval = MPI_ERR_TYPE; } else if(tag<0 && tag != MPI_ANY_TAG){ retval = MPI_ERR_TAG; } else { int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; int src_traced = smpi_group_index(smpi_comm_group(comm), src); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_IRECV; extra->src = src_traced; extra->dst = rank; int known=0; extra->datatype1 = encode_datatype(datatype, &known); int dt_size_send = 1; if(known==0) dt_size_send = smpi_datatype_size(datatype); extra->send_size = count*dt_size_send; TRACE_smpi_ptp_in(rank, src_traced, rank, __FUNCTION__, extra); *request = smpi_mpi_irecv(buf, count, datatype, src, tag, comm); retval = MPI_SUCCESS; TRACE_smpi_ptp_out(rank, src_traced, rank, __FUNCTION__); (*request)->recv = 1; } smpi_bench_begin(); if (retval != MPI_SUCCESS && request != nullptr) *request = MPI_REQUEST_NULL; return retval; } int PMPI_Isend(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm, MPI_Request * request) { int retval = 0; smpi_bench_end(); if (request == nullptr) { retval = MPI_ERR_ARG; } else if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (dst == MPI_PROC_NULL) { *request = MPI_REQUEST_NULL; retval = MPI_SUCCESS; } else if (dst >= smpi_group_size(smpi_comm_group(comm)) || dst <0){ retval = MPI_ERR_RANK; } else if ((count < 0) || (buf==nullptr && count > 0)) { retval = MPI_ERR_COUNT; } else if (!is_datatype_valid(datatype)) { retval = MPI_ERR_TYPE; } else if(tag<0 && tag != MPI_ANY_TAG){ retval = MPI_ERR_TAG; } else { int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; int dst_traced = smpi_group_index(smpi_comm_group(comm), dst); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_ISEND; extra->src = rank; extra->dst = dst_traced; int known=0; extra->datatype1 = encode_datatype(datatype, &known); int dt_size_send = 1; if(known==0) dt_size_send = smpi_datatype_size(datatype); extra->send_size = count*dt_size_send; TRACE_smpi_ptp_in(rank, rank, dst_traced, __FUNCTION__, extra); TRACE_smpi_send(rank, rank, dst_traced, tag, count*smpi_datatype_size(datatype)); *request = smpi_mpi_isend(buf, count, datatype, dst, tag, comm); retval = MPI_SUCCESS; TRACE_smpi_ptp_out(rank, rank, dst_traced, __FUNCTION__); (*request)->send = 1; } smpi_bench_begin(); if (retval != MPI_SUCCESS && request!=nullptr) *request = MPI_REQUEST_NULL; return retval; } int PMPI_Issend(void* buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm, MPI_Request* request) { int retval = 0; smpi_bench_end(); if (request == nullptr) { retval = MPI_ERR_ARG; } else if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (dst == MPI_PROC_NULL) { *request = MPI_REQUEST_NULL; retval = MPI_SUCCESS; } else if (dst >= smpi_group_size(smpi_comm_group(comm)) || dst <0){ retval = MPI_ERR_RANK; } else if ((count < 0)|| (buf==nullptr && count > 0)) { retval = MPI_ERR_COUNT; } else if (!is_datatype_valid(datatype)) { retval = MPI_ERR_TYPE; } else if(tag<0 && tag != MPI_ANY_TAG){ retval = MPI_ERR_TAG; } else { int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; int dst_traced = smpi_group_index(smpi_comm_group(comm), dst); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_ISSEND; extra->src = rank; extra->dst = dst_traced; int known=0; extra->datatype1 = encode_datatype(datatype, &known); int dt_size_send = 1; if(known==0) dt_size_send = smpi_datatype_size(datatype); extra->send_size = count*dt_size_send; TRACE_smpi_ptp_in(rank, rank, dst_traced, __FUNCTION__, extra); TRACE_smpi_send(rank, rank, dst_traced, tag, count*smpi_datatype_size(datatype)); *request = smpi_mpi_issend(buf, count, datatype, dst, tag, comm); retval = MPI_SUCCESS; TRACE_smpi_ptp_out(rank, rank, dst_traced, __FUNCTION__); (*request)->send = 1; } smpi_bench_begin(); if (retval != MPI_SUCCESS && request!=nullptr) *request = MPI_REQUEST_NULL; return retval; } int PMPI_Recv(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm, MPI_Status * status) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (src == MPI_PROC_NULL) { smpi_empty_status(status); status->MPI_SOURCE = MPI_PROC_NULL; retval = MPI_SUCCESS; } else if (src!=MPI_ANY_SOURCE && (src >= smpi_group_size(smpi_comm_group(comm)) || src <0)){ retval = MPI_ERR_RANK; } else if ((count < 0) || (buf==nullptr && count > 0)) { retval = MPI_ERR_COUNT; } else if (!is_datatype_valid(datatype)) { retval = MPI_ERR_TYPE; } else if(tag<0 && tag != MPI_ANY_TAG){ retval = MPI_ERR_TAG; } else { int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; int src_traced = smpi_group_index(smpi_comm_group(comm), src); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_RECV; extra->src = src_traced; extra->dst = rank; int known=0; extra->datatype1 = encode_datatype(datatype, &known); int dt_size_send = 1; if(known==0) dt_size_send = smpi_datatype_size(datatype); extra->send_size = count*dt_size_send; TRACE_smpi_ptp_in(rank, src_traced, rank, __FUNCTION__, extra); smpi_mpi_recv(buf, count, datatype, src, tag, comm, status); retval = MPI_SUCCESS; //the src may not have been known at the beginning of the recv (MPI_ANY_SOURCE) if(status!=MPI_STATUS_IGNORE){ src_traced = smpi_group_index(smpi_comm_group(comm), status->MPI_SOURCE); if (!TRACE_smpi_view_internals()) { TRACE_smpi_recv(rank, src_traced, rank, tag); } } TRACE_smpi_ptp_out(rank, src_traced, rank, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Send(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (dst == MPI_PROC_NULL) { retval = MPI_SUCCESS; } else if (dst >= smpi_group_size(smpi_comm_group(comm)) || dst <0){ retval = MPI_ERR_RANK; } else if ((count < 0) || (buf == nullptr && count > 0)) { retval = MPI_ERR_COUNT; } else if (!is_datatype_valid(datatype)) { retval = MPI_ERR_TYPE; } else if(tag < 0 && tag != MPI_ANY_TAG){ retval = MPI_ERR_TAG; } else { int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; int dst_traced = smpi_group_index(smpi_comm_group(comm), dst); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_SEND; extra->src = rank; extra->dst = dst_traced; int known = 0; extra->datatype1 = encode_datatype(datatype, &known); int dt_size_send = 1; if (known == 0) { dt_size_send = smpi_datatype_size(datatype); } extra->send_size = count*dt_size_send; TRACE_smpi_ptp_in(rank, rank, dst_traced, __FUNCTION__, extra); if (!TRACE_smpi_view_internals()) { TRACE_smpi_send(rank, rank, dst_traced, tag,count*smpi_datatype_size(datatype)); } smpi_mpi_send(buf, count, datatype, dst, tag, comm); retval = MPI_SUCCESS; TRACE_smpi_ptp_out(rank, rank, dst_traced, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Ssend(void* buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (dst == MPI_PROC_NULL) { retval = MPI_SUCCESS; } else if (dst >= smpi_group_size(smpi_comm_group(comm)) || dst <0){ retval = MPI_ERR_RANK; } else if ((count < 0) || (buf==nullptr && count > 0)) { retval = MPI_ERR_COUNT; } else if (!is_datatype_valid(datatype)){ retval = MPI_ERR_TYPE; } else if(tag<0 && tag != MPI_ANY_TAG){ retval = MPI_ERR_TAG; } else { int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; int dst_traced = smpi_group_index(smpi_comm_group(comm), dst); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_SSEND; extra->src = rank; extra->dst = dst_traced; int known = 0; extra->datatype1 = encode_datatype(datatype, &known); int dt_size_send = 1; if(known == 0) { dt_size_send = smpi_datatype_size(datatype); } extra->send_size = count*dt_size_send; TRACE_smpi_ptp_in(rank, rank, dst_traced, __FUNCTION__, extra); TRACE_smpi_send(rank, rank, dst_traced, tag,count*smpi_datatype_size(datatype)); smpi_mpi_ssend(buf, count, datatype, dst, tag, comm); retval = MPI_SUCCESS; TRACE_smpi_ptp_out(rank, rank, dst_traced, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Sendrecv(void *sendbuf, int sendcount, MPI_Datatype sendtype, int dst, int sendtag, void *recvbuf, int recvcount, MPI_Datatype recvtype, int src, int recvtag, MPI_Comm comm, MPI_Status * status) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (!is_datatype_valid(sendtype) || !is_datatype_valid(recvtype)) { retval = MPI_ERR_TYPE; } else if (src == MPI_PROC_NULL || dst == MPI_PROC_NULL) { smpi_empty_status(status); status->MPI_SOURCE = MPI_PROC_NULL; retval = MPI_SUCCESS; }else if (dst >= smpi_group_size(smpi_comm_group(comm)) || dst <0 || (src!=MPI_ANY_SOURCE && (src >= smpi_group_size(smpi_comm_group(comm)) || src <0))){ retval = MPI_ERR_RANK; } else if ((sendcount < 0 || recvcount<0) || (sendbuf==nullptr && sendcount > 0) || (recvbuf==nullptr && recvcount>0)) { retval = MPI_ERR_COUNT; } else if((sendtag<0 && sendtag != MPI_ANY_TAG)||(recvtag<0 && recvtag != MPI_ANY_TAG)){ retval = MPI_ERR_TAG; } else { int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; int dst_traced = smpi_group_index(smpi_comm_group(comm), dst); int src_traced = smpi_group_index(smpi_comm_group(comm), src); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_SENDRECV; extra->src = src_traced; extra->dst = dst_traced; int known=0; extra->datatype1 = encode_datatype(sendtype, &known); int dt_size_send = 1; if(known==0) dt_size_send = smpi_datatype_size(sendtype); extra->send_size = sendcount*dt_size_send; extra->datatype2 = encode_datatype(recvtype, &known); int dt_size_recv = 1; if(known==0) dt_size_recv = smpi_datatype_size(recvtype); extra->recv_size = recvcount*dt_size_recv; TRACE_smpi_ptp_in(rank, src_traced, dst_traced, __FUNCTION__, extra); TRACE_smpi_send(rank, rank, dst_traced, sendtag,sendcount*smpi_datatype_size(sendtype)); smpi_mpi_sendrecv(sendbuf, sendcount, sendtype, dst, sendtag, recvbuf, recvcount, recvtype, src, recvtag, comm, status); retval = MPI_SUCCESS; TRACE_smpi_ptp_out(rank, src_traced, dst_traced, __FUNCTION__); TRACE_smpi_recv(rank, src_traced, rank, recvtag); } smpi_bench_begin(); return retval; } int PMPI_Sendrecv_replace(void *buf, int count, MPI_Datatype datatype, int dst, int sendtag, int src, int recvtag, MPI_Comm comm, MPI_Status * status) { void *recvbuf; int retval = 0; if (!is_datatype_valid(datatype)) { retval = MPI_ERR_TYPE; } else if (count < 0) { retval = MPI_ERR_COUNT; } else { int size = smpi_datatype_get_extent(datatype) * count; recvbuf = xbt_new0(char, size); retval = MPI_Sendrecv(buf, count, datatype, dst, sendtag, recvbuf, count, datatype, src, recvtag, comm, status); if(retval==MPI_SUCCESS){ smpi_datatype_copy(recvbuf, count, datatype, buf, count, datatype); } xbt_free(recvbuf); } return retval; } int PMPI_Test(MPI_Request * request, int *flag, MPI_Status * status) { int retval = 0; smpi_bench_end(); if (request == nullptr || flag == nullptr) { retval = MPI_ERR_ARG; } else if (*request == MPI_REQUEST_NULL) { *flag= true; smpi_empty_status(status); retval = MPI_SUCCESS; } else { int rank = ((*request)->comm != MPI_COMM_NULL) ? smpi_process_index() : -1; instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_TEST; TRACE_smpi_testing_in(rank, extra); *flag = smpi_mpi_test(request, status); TRACE_smpi_testing_out(rank); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Testany(int count, MPI_Request requests[], int *index, int *flag, MPI_Status * status) { int retval = 0; smpi_bench_end(); if (index == nullptr || flag == nullptr) { retval = MPI_ERR_ARG; } else { *flag = smpi_mpi_testany(count, requests, index, status); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Testall(int count, MPI_Request* requests, int* flag, MPI_Status* statuses) { int retval = 0; smpi_bench_end(); if (flag == nullptr) { retval = MPI_ERR_ARG; } else { *flag = smpi_mpi_testall(count, requests, statuses); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Probe(int source, int tag, MPI_Comm comm, MPI_Status* status) { int retval = 0; smpi_bench_end(); if (status == nullptr) { retval = MPI_ERR_ARG; } else if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (source == MPI_PROC_NULL) { smpi_empty_status(status); status->MPI_SOURCE = MPI_PROC_NULL; retval = MPI_SUCCESS; } else { smpi_mpi_probe(source, tag, comm, status); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Iprobe(int source, int tag, MPI_Comm comm, int* flag, MPI_Status* status) { int retval = 0; smpi_bench_end(); if ((flag == nullptr) || (status == nullptr)) { retval = MPI_ERR_ARG; } else if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (source == MPI_PROC_NULL) { *flag=true; smpi_empty_status(status); status->MPI_SOURCE = MPI_PROC_NULL; retval = MPI_SUCCESS; } else { smpi_mpi_iprobe(source, tag, comm, flag, status); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Wait(MPI_Request * request, MPI_Status * status) { int retval = 0; smpi_bench_end(); smpi_empty_status(status); if (request == nullptr) { retval = MPI_ERR_ARG; } else if (*request == MPI_REQUEST_NULL) { retval = MPI_SUCCESS; } else { int rank = (request!=nullptr && (*request)->comm != MPI_COMM_NULL) ? smpi_process_index() : -1; int src_traced = (*request)->src; int dst_traced = (*request)->dst; int tag_traced= (*request)->tag; MPI_Comm comm = (*request)->comm; int is_wait_for_receive = (*request)->recv; instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_WAIT; TRACE_smpi_ptp_in(rank, src_traced, dst_traced, __FUNCTION__, extra); smpi_mpi_wait(request, status); retval = MPI_SUCCESS; //the src may not have been known at the beginning of the recv (MPI_ANY_SOURCE) TRACE_smpi_ptp_out(rank, src_traced, dst_traced, __FUNCTION__); if (is_wait_for_receive) { if(src_traced==MPI_ANY_SOURCE) src_traced = (status!=MPI_STATUS_IGNORE) ? smpi_group_rank(smpi_comm_group(comm), status->MPI_SOURCE) : src_traced; TRACE_smpi_recv(rank, src_traced, dst_traced, tag_traced); } } smpi_bench_begin(); return retval; } int PMPI_Waitany(int count, MPI_Request requests[], int *index, MPI_Status * status) { if (index == nullptr) return MPI_ERR_ARG; smpi_bench_end(); //save requests information for tracing int i; typedef struct { int src; int dst; int recv; int tag; MPI_Comm comm; } savedvalstype; savedvalstype* savedvals=nullptr; if(count>0){ savedvals = xbt_new0(savedvalstype, count); } for (i = 0; i < count; i++) { MPI_Request req = requests[i]; //already received requests are no longer valid if (req) { savedvals[i]=(savedvalstype){req->src, req->dst, req->recv, req->tag, req->comm}; } } int rank_traced = smpi_process_index(); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_WAITANY; extra->send_size=count; TRACE_smpi_ptp_in(rank_traced, -1, -1, __FUNCTION__,extra); *index = smpi_mpi_waitany(count, requests, status); if(*index!=MPI_UNDEFINED){ int src_traced = savedvals[*index].src; //the src may not have been known at the beginning of the recv (MPI_ANY_SOURCE) int dst_traced = savedvals[*index].dst; int is_wait_for_receive = savedvals[*index].recv; if (is_wait_for_receive) { if(savedvals[*index].src==MPI_ANY_SOURCE) src_traced = (status!=MPI_STATUSES_IGNORE) ? smpi_group_rank(smpi_comm_group(savedvals[*index].comm), status->MPI_SOURCE) : savedvals[*index].src; TRACE_smpi_recv(rank_traced, src_traced, dst_traced, savedvals[*index].tag); } TRACE_smpi_ptp_out(rank_traced, src_traced, dst_traced, __FUNCTION__); } xbt_free(savedvals); smpi_bench_begin(); return MPI_SUCCESS; } int PMPI_Waitall(int count, MPI_Request requests[], MPI_Status status[]) { smpi_bench_end(); //save information from requests int i; typedef struct { int src; int dst; int recv; int tag; int valid; MPI_Comm comm; } savedvalstype; savedvalstype* savedvals=xbt_new0(savedvalstype, count); for (i = 0; i < count; i++) { MPI_Request req = requests[i]; if(req!=MPI_REQUEST_NULL){ savedvals[i]=(savedvalstype){req->src, req->dst, req->recv, req->tag, 1, req->comm}; }else{ savedvals[i].valid=0; } } int rank_traced = smpi_process_index(); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_WAITALL; extra->send_size=count; TRACE_smpi_ptp_in(rank_traced, -1, -1, __FUNCTION__,extra); int retval = smpi_mpi_waitall(count, requests, status); for (i = 0; i < count; i++) { if(savedvals[i].valid){ //the src may not have been known at the beginning of the recv (MPI_ANY_SOURCE) int src_traced = savedvals[i].src; int dst_traced = savedvals[i].dst; int is_wait_for_receive = savedvals[i].recv; if (is_wait_for_receive) { if(src_traced==MPI_ANY_SOURCE) src_traced = (status!=MPI_STATUSES_IGNORE) ? smpi_group_rank(smpi_comm_group(savedvals[i].comm), status[i].MPI_SOURCE) : savedvals[i].src; TRACE_smpi_recv(rank_traced, src_traced, dst_traced,savedvals[i].tag); } } } TRACE_smpi_ptp_out(rank_traced, -1, -1, __FUNCTION__); xbt_free(savedvals); smpi_bench_begin(); return retval; } int PMPI_Waitsome(int incount, MPI_Request requests[], int *outcount, int *indices, MPI_Status status[]) { int retval = 0; smpi_bench_end(); if (outcount == nullptr) { retval = MPI_ERR_ARG; } else { *outcount = smpi_mpi_waitsome(incount, requests, indices, status); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Testsome(int incount, MPI_Request requests[], int* outcount, int* indices, MPI_Status status[]) { int retval = 0; smpi_bench_end(); if (outcount == nullptr) { retval = MPI_ERR_ARG; } else { *outcount = smpi_mpi_testsome(incount, requests, indices, status); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Bcast(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (!is_datatype_valid(datatype)) { retval = MPI_ERR_ARG; } else { int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; int root_traced = smpi_group_index(smpi_comm_group(comm), root); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_BCAST; extra->root = root_traced; int known=0; extra->datatype1 = encode_datatype(datatype, &known); int dt_size_send = 1; if(known==0) dt_size_send = smpi_datatype_size(datatype); extra->send_size = count*dt_size_send; TRACE_smpi_collective_in(rank, root_traced, __FUNCTION__, extra); if(smpi_comm_size(comm)>1) mpi_coll_bcast_fun(buf, count, datatype, root, comm); retval = MPI_SUCCESS; TRACE_smpi_collective_out(rank, root_traced, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Barrier(MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else { int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_BARRIER; TRACE_smpi_collective_in(rank, -1, __FUNCTION__, extra); mpi_coll_barrier_fun(comm); retval = MPI_SUCCESS; TRACE_smpi_collective_out(rank, -1, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Gather(void *sendbuf, int sendcount, MPI_Datatype sendtype,void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if ((( sendbuf != MPI_IN_PLACE) && (sendtype == MPI_DATATYPE_NULL)) || ((smpi_comm_rank(comm) == root) && (recvtype == MPI_DATATYPE_NULL))){ retval = MPI_ERR_TYPE; } else if ((( sendbuf != MPI_IN_PLACE) && (sendcount <0)) || ((smpi_comm_rank(comm) == root) && (recvcount <0))){ retval = MPI_ERR_COUNT; } else { char* sendtmpbuf = static_cast(sendbuf); int sendtmpcount = sendcount; MPI_Datatype sendtmptype = sendtype; if( (smpi_comm_rank(comm) == root) && (sendbuf == MPI_IN_PLACE )) { sendtmpcount=0; sendtmptype=recvtype; } int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; int root_traced = smpi_group_index(smpi_comm_group(comm), root); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_GATHER; extra->root = root_traced; int known=0; extra->datatype1 = encode_datatype(sendtmptype, &known); int dt_size_send = 1; if(known==0) dt_size_send = smpi_datatype_size(sendtmptype); extra->send_size = sendtmpcount*dt_size_send; extra->datatype2 = encode_datatype(recvtype, &known); int dt_size_recv = 1; if((smpi_comm_rank(comm)==root) && known==0) dt_size_recv = smpi_datatype_size(recvtype); extra->recv_size = recvcount*dt_size_recv; TRACE_smpi_collective_in(rank, root_traced, __FUNCTION__, extra); mpi_coll_gather_fun(sendtmpbuf, sendtmpcount, sendtmptype, recvbuf, recvcount, recvtype, root, comm); retval = MPI_SUCCESS; TRACE_smpi_collective_out(rank, root_traced, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Gatherv(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvtype, int root, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if ((( sendbuf != MPI_IN_PLACE) && (sendtype == MPI_DATATYPE_NULL)) || ((smpi_comm_rank(comm) == root) && (recvtype == MPI_DATATYPE_NULL))){ retval = MPI_ERR_TYPE; } else if (( sendbuf != MPI_IN_PLACE) && (sendcount <0)){ retval = MPI_ERR_COUNT; } else if (recvcounts == nullptr || displs == nullptr) { retval = MPI_ERR_ARG; } else { char* sendtmpbuf = static_cast(sendbuf); int sendtmpcount = sendcount; MPI_Datatype sendtmptype = sendtype; if( (smpi_comm_rank(comm) == root) && (sendbuf == MPI_IN_PLACE )) { sendtmpcount=0; sendtmptype=recvtype; } int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; int root_traced = smpi_group_index(smpi_comm_group(comm), root); int i=0; int size = smpi_comm_size(comm); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_GATHERV; extra->num_processes = size; extra->root = root_traced; int known=0; extra->datatype1 = encode_datatype(sendtmptype, &known); int dt_size_send = 1; if(known==0) dt_size_send = smpi_datatype_size(sendtype); extra->send_size = sendtmpcount*dt_size_send; extra->datatype2 = encode_datatype(recvtype, &known); int dt_size_recv = 1; if(known==0) dt_size_recv = smpi_datatype_size(recvtype); if((smpi_comm_rank(comm)==root)){ extra->recvcounts= xbt_new(int,size); for(i=0; i< size; i++)//copy data to avoid bad free extra->recvcounts[i] = recvcounts[i]*dt_size_recv; } TRACE_smpi_collective_in(rank, root_traced, __FUNCTION__,extra); smpi_mpi_gatherv(sendtmpbuf, sendtmpcount, sendtmptype, recvbuf, recvcounts, displs, recvtype, root, comm); retval = MPI_SUCCESS; TRACE_smpi_collective_out(rank, root_traced, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Allgather(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if ((( sendbuf != MPI_IN_PLACE) && (sendtype == MPI_DATATYPE_NULL)) || (recvtype == MPI_DATATYPE_NULL)){ retval = MPI_ERR_TYPE; } else if ((( sendbuf != MPI_IN_PLACE) && (sendcount <0)) || (recvcount <0)){ retval = MPI_ERR_COUNT; } else { if(sendbuf == MPI_IN_PLACE) { sendbuf=static_cast(recvbuf)+smpi_datatype_get_extent(recvtype)*recvcount*smpi_comm_rank(comm); sendcount=recvcount; sendtype=recvtype; } int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_ALLGATHER; int known=0; extra->datatype1 = encode_datatype(sendtype, &known); int dt_size_send = 1; if(known==0) dt_size_send = smpi_datatype_size(sendtype); extra->send_size = sendcount*dt_size_send; extra->datatype2 = encode_datatype(recvtype, &known); int dt_size_recv = 1; if(known==0) dt_size_recv = smpi_datatype_size(recvtype); extra->recv_size = recvcount*dt_size_recv; TRACE_smpi_collective_in(rank, -1, __FUNCTION__, extra); mpi_coll_allgather_fun(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm); retval = MPI_SUCCESS; TRACE_smpi_collective_out(rank, -1, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Allgatherv(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvtype, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if ((( sendbuf != MPI_IN_PLACE) && (sendtype == MPI_DATATYPE_NULL)) || (recvtype == MPI_DATATYPE_NULL)){ retval = MPI_ERR_TYPE; } else if (( sendbuf != MPI_IN_PLACE) && (sendcount <0)){ retval = MPI_ERR_COUNT; } else if (recvcounts == nullptr || displs == nullptr) { retval = MPI_ERR_ARG; } else { if(sendbuf == MPI_IN_PLACE) { sendbuf=static_cast(recvbuf)+smpi_datatype_get_extent(recvtype)*displs[smpi_comm_rank(comm)]; sendcount=recvcounts[smpi_comm_rank(comm)]; sendtype=recvtype; } int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; int i=0; int size = smpi_comm_size(comm); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_ALLGATHERV; extra->num_processes = size; int known=0; extra->datatype1 = encode_datatype(sendtype, &known); int dt_size_send = 1; if(known==0) dt_size_send = smpi_datatype_size(sendtype); extra->send_size = sendcount*dt_size_send; extra->datatype2 = encode_datatype(recvtype, &known); int dt_size_recv = 1; if(known==0) dt_size_recv = smpi_datatype_size(recvtype); extra->recvcounts= xbt_new(int, size); for(i=0; i< size; i++)//copy data to avoid bad free extra->recvcounts[i] = recvcounts[i]*dt_size_recv; TRACE_smpi_collective_in(rank, -1, __FUNCTION__,extra); mpi_coll_allgatherv_fun(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm); retval = MPI_SUCCESS; TRACE_smpi_collective_out(rank, -1, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Scatter(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (((smpi_comm_rank(comm)==root) && (!is_datatype_valid(sendtype))) || ((recvbuf !=MPI_IN_PLACE) && (!is_datatype_valid(recvtype)))){ retval = MPI_ERR_TYPE; } else if ((sendbuf == recvbuf) || ((smpi_comm_rank(comm)==root) && sendcount>0 && (sendbuf == nullptr))){ retval = MPI_ERR_BUFFER; }else { if (recvbuf == MPI_IN_PLACE) { recvtype=sendtype; recvcount=sendcount; } int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; int root_traced = smpi_group_index(smpi_comm_group(comm), root); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_SCATTER; extra->root = root_traced; int known=0; extra->datatype1 = encode_datatype(sendtype, &known); int dt_size_send = 1; if((smpi_comm_rank(comm)==root) && known==0) dt_size_send = smpi_datatype_size(sendtype); extra->send_size = sendcount*dt_size_send; extra->datatype2 = encode_datatype(recvtype, &known); int dt_size_recv = 1; if(known==0) dt_size_recv = smpi_datatype_size(recvtype); extra->recv_size = recvcount*dt_size_recv; TRACE_smpi_collective_in(rank, root_traced, __FUNCTION__,extra); mpi_coll_scatter_fun(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm); retval = MPI_SUCCESS; TRACE_smpi_collective_out(rank, root_traced, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Scatterv(void *sendbuf, int *sendcounts, int *displs, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (sendcounts == nullptr || displs == nullptr) { retval = MPI_ERR_ARG; } else if (((smpi_comm_rank(comm)==root) && (sendtype == MPI_DATATYPE_NULL)) || ((recvbuf !=MPI_IN_PLACE) && (recvtype == MPI_DATATYPE_NULL))) { retval = MPI_ERR_TYPE; } else { if (recvbuf == MPI_IN_PLACE) { recvtype=sendtype; recvcount=sendcounts[smpi_comm_rank(comm)]; } int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; int root_traced = smpi_group_index(smpi_comm_group(comm), root); int i=0; int size = smpi_comm_size(comm); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_SCATTERV; extra->num_processes = size; extra->root = root_traced; int known=0; extra->datatype1 = encode_datatype(sendtype, &known); int dt_size_send = 1; if(known==0) dt_size_send = smpi_datatype_size(sendtype); if((smpi_comm_rank(comm)==root)){ extra->sendcounts= xbt_new(int, size); for(i=0; i< size; i++)//copy data to avoid bad free extra->sendcounts[i] = sendcounts[i]*dt_size_send; } extra->datatype2 = encode_datatype(recvtype, &known); int dt_size_recv = 1; if(known==0) dt_size_recv = smpi_datatype_size(recvtype); extra->recv_size = recvcount*dt_size_recv; TRACE_smpi_collective_in(rank, root_traced, __FUNCTION__,extra); smpi_mpi_scatterv(sendbuf, sendcounts, displs, sendtype, recvbuf, recvcount, recvtype, root, comm); retval = MPI_SUCCESS; TRACE_smpi_collective_out(rank, root_traced, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Reduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (!is_datatype_valid(datatype) || op == MPI_OP_NULL) { retval = MPI_ERR_ARG; } else { int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; int root_traced = smpi_group_index(smpi_comm_group(comm), root); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_REDUCE; int known=0; extra->datatype1 = encode_datatype(datatype, &known); int dt_size_send = 1; if(known==0) dt_size_send = smpi_datatype_size(datatype); extra->send_size = count*dt_size_send; extra->root = root_traced; TRACE_smpi_collective_in(rank, root_traced, __FUNCTION__,extra); mpi_coll_reduce_fun(sendbuf, recvbuf, count, datatype, op, root, comm); retval = MPI_SUCCESS; TRACE_smpi_collective_out(rank, root_traced, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Reduce_local(void *inbuf, void *inoutbuf, int count, MPI_Datatype datatype, MPI_Op op){ int retval = 0; smpi_bench_end(); if (!is_datatype_valid(datatype) || op == MPI_OP_NULL) { retval = MPI_ERR_ARG; } else { smpi_op_apply(op, inbuf, inoutbuf, &count, &datatype); retval=MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Allreduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (!is_datatype_valid(datatype)) { retval = MPI_ERR_TYPE; } else if (op == MPI_OP_NULL) { retval = MPI_ERR_OP; } else { char* sendtmpbuf = static_cast(sendbuf); if( sendbuf == MPI_IN_PLACE ) { sendtmpbuf = static_cast(xbt_malloc(count*smpi_datatype_get_extent(datatype))); smpi_datatype_copy(recvbuf, count, datatype,sendtmpbuf, count, datatype); } int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_ALLREDUCE; int known=0; extra->datatype1 = encode_datatype(datatype, &known); int dt_size_send = 1; if(known==0) dt_size_send = smpi_datatype_size(datatype); extra->send_size = count*dt_size_send; TRACE_smpi_collective_in(rank, -1, __FUNCTION__,extra); mpi_coll_allreduce_fun(sendtmpbuf, recvbuf, count, datatype, op, comm); if( sendbuf == MPI_IN_PLACE ) xbt_free(sendtmpbuf); retval = MPI_SUCCESS; TRACE_smpi_collective_out(rank, -1, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Scan(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (!is_datatype_valid(datatype)) { retval = MPI_ERR_TYPE; } else if (op == MPI_OP_NULL) { retval = MPI_ERR_OP; } else { int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_SCAN; int known=0; extra->datatype1 = encode_datatype(datatype, &known); int dt_size_send = 1; if(known==0) dt_size_send = smpi_datatype_size(datatype); extra->send_size = count*dt_size_send; TRACE_smpi_collective_in(rank, -1, __FUNCTION__,extra); smpi_mpi_scan(sendbuf, recvbuf, count, datatype, op, comm); retval = MPI_SUCCESS; TRACE_smpi_collective_out(rank, -1, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Exscan(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm){ int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (!is_datatype_valid(datatype)) { retval = MPI_ERR_TYPE; } else if (op == MPI_OP_NULL) { retval = MPI_ERR_OP; } else { int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_EXSCAN; int known=0; extra->datatype1 = encode_datatype(datatype, &known); int dt_size_send = 1; if(known==0) dt_size_send = smpi_datatype_size(datatype); extra->send_size = count*dt_size_send; TRACE_smpi_collective_in(rank, -1, __FUNCTION__,extra); smpi_mpi_exscan(sendbuf, recvbuf, count, datatype, op, comm); retval = MPI_SUCCESS; TRACE_smpi_collective_out(rank, -1, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Reduce_scatter(void *sendbuf, void *recvbuf, int *recvcounts, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (!is_datatype_valid(datatype)) { retval = MPI_ERR_TYPE; } else if (op == MPI_OP_NULL) { retval = MPI_ERR_OP; } else if (recvcounts == nullptr) { retval = MPI_ERR_ARG; } else { int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; int i=0; int size = smpi_comm_size(comm); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_REDUCE_SCATTER; extra->num_processes = size; int known=0; extra->datatype1 = encode_datatype(datatype, &known); int dt_size_send = 1; if(known==0) dt_size_send = smpi_datatype_size(datatype); extra->send_size = 0; extra->recvcounts= xbt_new(int, size); for(i=0; i< size; i++)//copy data to avoid bad free extra->recvcounts[i] = recvcounts[i]*dt_size_send; TRACE_smpi_collective_in(rank, -1, __FUNCTION__,extra); void* sendtmpbuf=sendbuf; if(sendbuf==MPI_IN_PLACE) sendtmpbuf=recvbuf; mpi_coll_reduce_scatter_fun(sendtmpbuf, recvbuf, recvcounts, datatype, op, comm); retval = MPI_SUCCESS; TRACE_smpi_collective_out(rank, -1, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Reduce_scatter_block(void *sendbuf, void *recvbuf, int recvcount, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int retval,i; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (!is_datatype_valid(datatype)) { retval = MPI_ERR_TYPE; } else if (op == MPI_OP_NULL) { retval = MPI_ERR_OP; } else if (recvcount < 0) { retval = MPI_ERR_ARG; } else { int count=smpi_comm_size(comm); int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_REDUCE_SCATTER; extra->num_processes = count; int known=0; extra->datatype1 = encode_datatype(datatype, &known); int dt_size_send = 1; if(known==0) dt_size_send = smpi_datatype_size(datatype); extra->send_size = 0; extra->recvcounts= xbt_new(int, count); for(i=0; i< count; i++)//copy data to avoid bad free extra->recvcounts[i] = recvcount*dt_size_send; TRACE_smpi_collective_in(rank, -1, __FUNCTION__,extra); int* recvcounts=static_cast(xbt_malloc(count)); for (i=0; itype = TRACING_ALLTOALL; int known=0; extra->datatype1 = encode_datatype(sendtype, &known); if(known==0) extra->send_size = sendcount*smpi_datatype_size(sendtype); else extra->send_size = sendcount; extra->datatype2 = encode_datatype(recvtype, &known); if(known==0) extra->recv_size = recvcount*smpi_datatype_size(recvtype); else extra->recv_size = recvcount; TRACE_smpi_collective_in(rank, -1, __FUNCTION__,extra); retval = mpi_coll_alltoall_fun(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm); TRACE_smpi_collective_out(rank, -1, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Alltoallv(void *sendbuf, int *sendcounts, int *senddisps,MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *recvdisps, MPI_Datatype recvtype, MPI_Comm comm) { int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval = MPI_ERR_COMM; } else if (sendtype == MPI_DATATYPE_NULL || recvtype == MPI_DATATYPE_NULL) { retval = MPI_ERR_TYPE; } else if (sendcounts == nullptr || senddisps == nullptr || recvcounts == nullptr || recvdisps == nullptr) { retval = MPI_ERR_ARG; } else { int rank = comm != MPI_COMM_NULL ? smpi_process_index() : -1; int i=0; int size = smpi_comm_size(comm); instr_extra_data extra = xbt_new0(s_instr_extra_data_t,1); extra->type = TRACING_ALLTOALLV; extra->send_size = 0; extra->recv_size = 0; extra->recvcounts= xbt_new(int, size); extra->sendcounts= xbt_new(int, size); int known=0; extra->datatype1 = encode_datatype(sendtype, &known); int dt_size_send = 1; if(known==0) dt_size_send = smpi_datatype_size(sendtype); int dt_size_recv = 1; extra->datatype2 = encode_datatype(recvtype, &known); if(known==0) dt_size_recv = smpi_datatype_size(recvtype); for(i=0; i< size; i++){//copy data to avoid bad free extra->send_size += sendcounts[i]*dt_size_send; extra->recv_size += recvcounts[i]*dt_size_recv; extra->sendcounts[i] = sendcounts[i]*dt_size_send; extra->recvcounts[i] = recvcounts[i]*dt_size_recv; } extra->num_processes = size; TRACE_smpi_collective_in(rank, -1, __FUNCTION__,extra); retval = mpi_coll_alltoallv_fun(sendbuf, sendcounts, senddisps, sendtype, recvbuf, recvcounts, recvdisps, recvtype, comm); TRACE_smpi_collective_out(rank, -1, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Get_processor_name(char *name, int *resultlen) { int retval = MPI_SUCCESS; strncpy(name, SIMIX_host_self()->cname(), strlen(SIMIX_host_self()->cname()) < MPI_MAX_PROCESSOR_NAME - 1 ? strlen(SIMIX_host_self()->cname()) + 1 : MPI_MAX_PROCESSOR_NAME - 1); *resultlen = strlen(name) > MPI_MAX_PROCESSOR_NAME ? MPI_MAX_PROCESSOR_NAME : strlen(name); return retval; } int PMPI_Get_count(MPI_Status * status, MPI_Datatype datatype, int *count) { int retval = MPI_SUCCESS; size_t size; if (status == nullptr || count == nullptr) { retval = MPI_ERR_ARG; } else if (!is_datatype_valid(datatype)) { retval = MPI_ERR_TYPE; } else { size = smpi_datatype_size(datatype); if (size == 0) { *count = 0; } else if (status->count % size != 0) { retval = MPI_UNDEFINED; } else { *count = smpi_mpi_get_count(status, datatype); } } return retval; } int PMPI_Type_contiguous(int count, MPI_Datatype old_type, MPI_Datatype* new_type) { int retval = 0; if (old_type == MPI_DATATYPE_NULL) { retval = MPI_ERR_TYPE; } else if (count<0){ retval = MPI_ERR_COUNT; } else { retval = smpi_datatype_contiguous(count, old_type, new_type, 0); } return retval; } int PMPI_Type_commit(MPI_Datatype* datatype) { int retval = 0; if (datatype == nullptr || *datatype == MPI_DATATYPE_NULL) { retval = MPI_ERR_TYPE; } else { smpi_datatype_commit(datatype); retval = MPI_SUCCESS; } return retval; } int PMPI_Type_vector(int count, int blocklen, int stride, MPI_Datatype old_type, MPI_Datatype* new_type) { int retval = 0; if (old_type == MPI_DATATYPE_NULL) { retval = MPI_ERR_TYPE; } else if (count<0 || blocklen<0){ retval = MPI_ERR_COUNT; } else { retval = smpi_datatype_vector(count, blocklen, stride, old_type, new_type); } return retval; } int PMPI_Type_hvector(int count, int blocklen, MPI_Aint stride, MPI_Datatype old_type, MPI_Datatype* new_type) { int retval = 0; if (old_type == MPI_DATATYPE_NULL) { retval = MPI_ERR_TYPE; } else if (count<0 || blocklen<0){ retval = MPI_ERR_COUNT; } else { retval = smpi_datatype_hvector(count, blocklen, stride, old_type, new_type); } return retval; } int PMPI_Type_create_hvector(int count, int blocklen, MPI_Aint stride, MPI_Datatype old_type, MPI_Datatype* new_type) { return MPI_Type_hvector(count, blocklen, stride, old_type, new_type); } int PMPI_Type_indexed(int count, int* blocklens, int* indices, MPI_Datatype old_type, MPI_Datatype* new_type) { int retval = 0; if (old_type == MPI_DATATYPE_NULL) { retval = MPI_ERR_TYPE; } else if (count<0){ retval = MPI_ERR_COUNT; } else { retval = smpi_datatype_indexed(count, blocklens, indices, old_type, new_type); } return retval; } int PMPI_Type_create_indexed(int count, int* blocklens, int* indices, MPI_Datatype old_type, MPI_Datatype* new_type) { int retval = 0; if (old_type == MPI_DATATYPE_NULL) { retval = MPI_ERR_TYPE; } else if (count<0){ retval = MPI_ERR_COUNT; } else { retval = smpi_datatype_indexed(count, blocklens, indices, old_type, new_type); } return retval; } int PMPI_Type_create_indexed_block(int count, int blocklength, int* indices, MPI_Datatype old_type, MPI_Datatype* new_type) { int retval,i; if (old_type == MPI_DATATYPE_NULL) { retval = MPI_ERR_TYPE; } else if (count<0){ retval = MPI_ERR_COUNT; } else { int* blocklens=static_cast(xbt_malloc(blocklength*count)); for (i=0; i 0 && (dims == nullptr || periodic == nullptr)) || comm_cart == nullptr) { retval = MPI_ERR_ARG; } else{ retval = smpi_mpi_cart_create(comm_old, ndims, dims, periodic, reorder, comm_cart); } return retval; } int PMPI_Cart_rank(MPI_Comm comm, int* coords, int* rank) { if(comm == MPI_COMM_NULL || smpi_comm_topo(comm) == nullptr) { return MPI_ERR_TOPOLOGY; } if (coords == nullptr) { return MPI_ERR_ARG; } return smpi_mpi_cart_rank(comm, coords, rank); } int PMPI_Cart_shift(MPI_Comm comm, int direction, int displ, int* source, int* dest) { if(comm == MPI_COMM_NULL || smpi_comm_topo(comm) == nullptr) { return MPI_ERR_TOPOLOGY; } if (source == nullptr || dest == nullptr || direction < 0 ) { return MPI_ERR_ARG; } return smpi_mpi_cart_shift(comm, direction, displ, source, dest); } int PMPI_Cart_coords(MPI_Comm comm, int rank, int maxdims, int* coords) { if(comm == MPI_COMM_NULL || smpi_comm_topo(comm) == nullptr) { return MPI_ERR_TOPOLOGY; } if (rank < 0 || rank >= smpi_comm_size(comm)) { return MPI_ERR_RANK; } if (maxdims <= 0) { return MPI_ERR_ARG; } if(coords == nullptr) { return MPI_ERR_ARG; } return smpi_mpi_cart_coords(comm, rank, maxdims, coords); } int PMPI_Cart_get(MPI_Comm comm, int maxdims, int* dims, int* periods, int* coords) { if(comm == nullptr || smpi_comm_topo(comm) == nullptr) { return MPI_ERR_TOPOLOGY; } if(maxdims <= 0 || dims == nullptr || periods == nullptr || coords == nullptr) { return MPI_ERR_ARG; } return smpi_mpi_cart_get(comm, maxdims, dims, periods, coords); } int PMPI_Cartdim_get(MPI_Comm comm, int* ndims) { if (comm == MPI_COMM_NULL || smpi_comm_topo(comm) == nullptr) { return MPI_ERR_TOPOLOGY; } if (ndims == nullptr) { return MPI_ERR_ARG; } return smpi_mpi_cartdim_get(comm, ndims); } int PMPI_Dims_create(int nnodes, int ndims, int* dims) { if(dims == nullptr) { return MPI_ERR_ARG; } if (ndims < 1 || nnodes < 1) { return MPI_ERR_DIMS; } return smpi_mpi_dims_create(nnodes, ndims, dims); } int PMPI_Cart_sub(MPI_Comm comm, int* remain_dims, MPI_Comm* comm_new) { if(comm == MPI_COMM_NULL || smpi_comm_topo(comm) == nullptr) { return MPI_ERR_TOPOLOGY; } if (comm_new == nullptr) { return MPI_ERR_ARG; } return smpi_mpi_cart_sub(comm, remain_dims, comm_new); } int PMPI_Type_create_resized(MPI_Datatype oldtype,MPI_Aint lb, MPI_Aint extent, MPI_Datatype *newtype){ if(oldtype == MPI_DATATYPE_NULL) { return MPI_ERR_TYPE; } int blocks[3] = { 1, 1, 1 }; MPI_Aint disps[3] = { lb, 0, lb+extent }; MPI_Datatype types[3] = { MPI_LB, oldtype, MPI_UB }; s_smpi_mpi_struct_t* subtype = smpi_datatype_struct_create( blocks, disps, 3, types); smpi_datatype_create(newtype,oldtype->size, lb, lb + extent, sizeof(s_smpi_mpi_struct_t) , subtype, DT_FLAG_VECTOR); (*newtype)->flags &= ~DT_FLAG_COMMITED; return MPI_SUCCESS; } int PMPI_Win_create( void *base, MPI_Aint size, int disp_unit, MPI_Info info, MPI_Comm comm, MPI_Win *win){ int retval = 0; smpi_bench_end(); if (comm == MPI_COMM_NULL) { retval= MPI_ERR_COMM; }else if ((base == nullptr && size != 0) || disp_unit <= 0 || size < 0 ){ retval= MPI_ERR_OTHER; }else{ *win = smpi_mpi_win_create( base, size, disp_unit, info, comm); retval = MPI_SUCCESS; } smpi_bench_begin(); return retval; } int PMPI_Win_free( MPI_Win* win){ int retval = 0; smpi_bench_end(); if (win == nullptr || *win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; }else{ retval=smpi_mpi_win_free(win); } smpi_bench_begin(); return retval; } int PMPI_Win_set_name(MPI_Win win, char * name) { int retval = 0; if (win == MPI_WIN_NULL) { retval = MPI_ERR_TYPE; } else if (name == nullptr) { retval = MPI_ERR_ARG; } else { smpi_mpi_win_set_name(win, name); retval = MPI_SUCCESS; } return retval; } int PMPI_Win_get_name(MPI_Win win, char * name, int* len) { int retval = MPI_SUCCESS; if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (name == nullptr) { retval = MPI_ERR_ARG; } else { smpi_mpi_win_get_name(win, name, len); } return retval; } int PMPI_Win_get_group(MPI_Win win, MPI_Group * group){ int retval = MPI_SUCCESS; if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; }else { smpi_mpi_win_get_group(win, group); smpi_group_use(*group); } return retval; } int PMPI_Win_fence( int assert, MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else { int rank = smpi_process_index(); TRACE_smpi_collective_in(rank, -1, __FUNCTION__, nullptr); retval = smpi_mpi_win_fence(assert, win); TRACE_smpi_collective_out(rank, -1, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Get( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (target_rank == MPI_PROC_NULL) { retval = MPI_SUCCESS; } else if (target_rank <0){ retval = MPI_ERR_RANK; } else if (target_disp <0){ retval = MPI_ERR_ARG; } else if ((origin_count < 0 || target_count < 0) || (origin_addr==nullptr && origin_count > 0)){ retval = MPI_ERR_COUNT; } else if ((!is_datatype_valid(origin_datatype)) || (!is_datatype_valid(target_datatype))) { retval = MPI_ERR_TYPE; } else { int rank = smpi_process_index(); MPI_Group group; smpi_mpi_win_get_group(win, &group); int src_traced = smpi_group_index(group, target_rank); TRACE_smpi_ptp_in(rank, src_traced, rank, __FUNCTION__, nullptr); retval = smpi_mpi_get( origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, win); TRACE_smpi_ptp_out(rank, src_traced, rank, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Put( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (target_rank == MPI_PROC_NULL) { retval = MPI_SUCCESS; } else if (target_rank <0){ retval = MPI_ERR_RANK; } else if (target_disp <0){ retval = MPI_ERR_ARG; } else if ((origin_count < 0 || target_count < 0) || (origin_addr==nullptr && origin_count > 0)){ retval = MPI_ERR_COUNT; } else if ((!is_datatype_valid(origin_datatype)) || (!is_datatype_valid(target_datatype))) { retval = MPI_ERR_TYPE; } else { int rank = smpi_process_index(); MPI_Group group; smpi_mpi_win_get_group(win, &group); int dst_traced = smpi_group_index(group, target_rank); TRACE_smpi_ptp_in(rank, rank, dst_traced, __FUNCTION__, nullptr); TRACE_smpi_send(rank, rank, dst_traced, SMPI_RMA_TAG, origin_count*smpi_datatype_size(origin_datatype)); retval = smpi_mpi_put( origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, win); TRACE_smpi_ptp_out(rank, rank, dst_traced, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Accumulate( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Op op, MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (target_rank == MPI_PROC_NULL) { retval = MPI_SUCCESS; } else if (target_rank <0){ retval = MPI_ERR_RANK; } else if (target_disp <0){ retval = MPI_ERR_ARG; } else if ((origin_count < 0 || target_count < 0) || (origin_addr==nullptr && origin_count > 0)){ retval = MPI_ERR_COUNT; } else if ((!is_datatype_valid(origin_datatype)) || (!is_datatype_valid(target_datatype))) { retval = MPI_ERR_TYPE; } else if (op == MPI_OP_NULL) { retval = MPI_ERR_OP; } else { int rank = smpi_process_index(); MPI_Group group; smpi_mpi_win_get_group(win, &group); int src_traced = smpi_group_index(group, target_rank); TRACE_smpi_ptp_in(rank, src_traced, rank, __FUNCTION__, nullptr); retval = smpi_mpi_accumulate( origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, op, win); TRACE_smpi_ptp_out(rank, src_traced, rank, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Win_post(MPI_Group group, int assert, MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (group==MPI_GROUP_NULL){ retval = MPI_ERR_GROUP; } else { int rank = smpi_process_index(); TRACE_smpi_collective_in(rank, -1, __FUNCTION__, nullptr); retval = smpi_mpi_win_post(group,assert,win); TRACE_smpi_collective_out(rank, -1, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Win_start(MPI_Group group, int assert, MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else if (group==MPI_GROUP_NULL){ retval = MPI_ERR_GROUP; } else { int rank = smpi_process_index(); TRACE_smpi_collective_in(rank, -1, __FUNCTION__, nullptr); retval = smpi_mpi_win_start(group,assert,win); TRACE_smpi_collective_out(rank, -1, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Win_complete(MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else { int rank = smpi_process_index(); TRACE_smpi_collective_in(rank, -1, __FUNCTION__, nullptr); retval = smpi_mpi_win_complete(win); TRACE_smpi_collective_out(rank, -1, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Win_wait(MPI_Win win){ int retval = 0; smpi_bench_end(); if (win == MPI_WIN_NULL) { retval = MPI_ERR_WIN; } else { int rank = smpi_process_index(); TRACE_smpi_collective_in(rank, -1, __FUNCTION__, nullptr); retval = smpi_mpi_win_wait(win); TRACE_smpi_collective_out(rank, -1, __FUNCTION__); } smpi_bench_begin(); return retval; } int PMPI_Alloc_mem(MPI_Aint size, MPI_Info info, void *baseptr){ void *ptr = xbt_malloc(size); if(ptr==nullptr) return MPI_ERR_NO_MEM; else { *static_cast(baseptr) = ptr; return MPI_SUCCESS; } } int PMPI_Free_mem(void *baseptr){ xbt_free(baseptr); return MPI_SUCCESS; } int PMPI_Type_set_name(MPI_Datatype datatype, char * name) { int retval = 0; if (datatype == MPI_DATATYPE_NULL) { retval = MPI_ERR_TYPE; } else if (name == nullptr) { retval = MPI_ERR_ARG; } else { smpi_datatype_set_name(datatype, name); retval = MPI_SUCCESS; } return retval; } int PMPI_Type_get_name(MPI_Datatype datatype, char * name, int* len) { int retval = 0; if (datatype == MPI_DATATYPE_NULL) { retval = MPI_ERR_TYPE; } else if (name == nullptr) { retval = MPI_ERR_ARG; } else { smpi_datatype_get_name(datatype, name, len); retval = MPI_SUCCESS; } return retval; } MPI_Datatype PMPI_Type_f2c(MPI_Fint datatype){ return smpi_type_f2c(datatype); } MPI_Fint PMPI_Type_c2f(MPI_Datatype datatype){ return smpi_type_c2f( datatype); } MPI_Group PMPI_Group_f2c(MPI_Fint group){ return smpi_group_f2c( group); } MPI_Fint PMPI_Group_c2f(MPI_Group group){ return smpi_group_c2f(group); } MPI_Request PMPI_Request_f2c(MPI_Fint request){ return smpi_request_f2c(request); } MPI_Fint PMPI_Request_c2f(MPI_Request request) { return smpi_request_c2f(request); } MPI_Win PMPI_Win_f2c(MPI_Fint win){ return smpi_win_f2c(win); } MPI_Fint PMPI_Win_c2f(MPI_Win win){ return smpi_win_c2f(win); } MPI_Op PMPI_Op_f2c(MPI_Fint op){ return smpi_op_f2c(op); } MPI_Fint PMPI_Op_c2f(MPI_Op op){ return smpi_op_c2f(op); } MPI_Comm PMPI_Comm_f2c(MPI_Fint comm){ return smpi_comm_f2c(comm); } MPI_Fint PMPI_Comm_c2f(MPI_Comm comm){ return smpi_comm_c2f(comm); } MPI_Info PMPI_Info_f2c(MPI_Fint info){ return smpi_info_f2c(info); } MPI_Fint PMPI_Info_c2f(MPI_Info info){ return smpi_info_c2f(info); } int PMPI_Keyval_create(MPI_Copy_function* copy_fn, MPI_Delete_function* delete_fn, int* keyval, void* extra_state) { return smpi_comm_keyval_create(copy_fn, delete_fn, keyval, extra_state); } int PMPI_Keyval_free(int* keyval) { return smpi_comm_keyval_free(keyval); } int PMPI_Attr_delete(MPI_Comm comm, int keyval) { if(keyval == MPI_TAG_UB||keyval == MPI_HOST||keyval == MPI_IO ||keyval == MPI_WTIME_IS_GLOBAL||keyval == MPI_APPNUM ||keyval == MPI_UNIVERSE_SIZE||keyval == MPI_LASTUSEDCODE) return MPI_ERR_ARG; else if (comm==MPI_COMM_NULL) return MPI_ERR_COMM; else return smpi_comm_attr_delete(comm, keyval); } int PMPI_Attr_get(MPI_Comm comm, int keyval, void* attr_value, int* flag) { static int one = 1; static int zero = 0; static int tag_ub = 1000000; static int last_used_code = MPI_ERR_LASTCODE; if (comm==MPI_COMM_NULL){ *flag = 0; return MPI_ERR_COMM; } switch (keyval) { case MPI_HOST: case MPI_IO: case MPI_APPNUM: *flag = 1; *static_cast(attr_value) = &zero; return MPI_SUCCESS; case MPI_UNIVERSE_SIZE: *flag = 1; *static_cast(attr_value) = &smpi_universe_size; return MPI_SUCCESS; case MPI_LASTUSEDCODE: *flag = 1; *static_cast(attr_value) = &last_used_code; return MPI_SUCCESS; case MPI_TAG_UB: *flag=1; *static_cast(attr_value) = &tag_ub; return MPI_SUCCESS; case MPI_WTIME_IS_GLOBAL: *flag = 1; *static_cast(attr_value) = &one; return MPI_SUCCESS; default: return smpi_comm_attr_get(comm, keyval, attr_value, flag); } } int PMPI_Attr_put(MPI_Comm comm, int keyval, void* attr_value) { if(keyval == MPI_TAG_UB||keyval == MPI_HOST||keyval == MPI_IO ||keyval == MPI_WTIME_IS_GLOBAL||keyval == MPI_APPNUM ||keyval == MPI_UNIVERSE_SIZE||keyval == MPI_LASTUSEDCODE) return MPI_ERR_ARG; else if (comm==MPI_COMM_NULL) return MPI_ERR_COMM; else return smpi_comm_attr_put(comm, keyval, attr_value); } int PMPI_Comm_get_attr (MPI_Comm comm, int comm_keyval, void *attribute_val, int *flag) { return PMPI_Attr_get(comm, comm_keyval, attribute_val,flag); } int PMPI_Comm_set_attr (MPI_Comm comm, int comm_keyval, void *attribute_val) { return PMPI_Attr_put(comm, comm_keyval, attribute_val); } int PMPI_Comm_delete_attr (MPI_Comm comm, int comm_keyval) { return PMPI_Attr_delete(comm, comm_keyval); } int PMPI_Comm_create_keyval(MPI_Comm_copy_attr_function* copy_fn, MPI_Comm_delete_attr_function* delete_fn, int* keyval, void* extra_state) { return PMPI_Keyval_create(copy_fn, delete_fn, keyval, extra_state); } int PMPI_Comm_free_keyval(int* keyval) { return PMPI_Keyval_free(keyval); } int PMPI_Type_get_attr (MPI_Datatype type, int type_keyval, void *attribute_val, int* flag) { if (type==MPI_DATATYPE_NULL) return MPI_ERR_TYPE; else return smpi_type_attr_get(type, type_keyval, attribute_val, flag); } int PMPI_Type_set_attr (MPI_Datatype type, int type_keyval, void *attribute_val) { if (type==MPI_DATATYPE_NULL) return MPI_ERR_TYPE; else return smpi_type_attr_put(type, type_keyval, attribute_val); } int PMPI_Type_delete_attr (MPI_Datatype type, int type_keyval) { if (type==MPI_DATATYPE_NULL) return MPI_ERR_TYPE; else return smpi_type_attr_delete(type, type_keyval); } int PMPI_Type_create_keyval(MPI_Type_copy_attr_function* copy_fn, MPI_Type_delete_attr_function* delete_fn, int* keyval, void* extra_state) { return smpi_type_keyval_create(copy_fn, delete_fn, keyval, extra_state); } int PMPI_Type_free_keyval(int* keyval) { return smpi_type_keyval_free(keyval); } int PMPI_Info_create( MPI_Info *info){ if (info == nullptr) return MPI_ERR_ARG; *info = xbt_new(s_smpi_mpi_info_t, 1); (*info)->info_dict= xbt_dict_new_homogeneous(xbt_free_f); (*info)->refcount=1; return MPI_SUCCESS; } int PMPI_Info_set( MPI_Info info, char *key, char *value){ if (info == nullptr || key == nullptr || value == nullptr) return MPI_ERR_ARG; xbt_dict_set(info->info_dict, key, xbt_strdup(value), nullptr); return MPI_SUCCESS; } int PMPI_Info_free( MPI_Info *info){ if (info == nullptr || *info==nullptr) return MPI_ERR_ARG; (*info)->refcount--; if((*info)->refcount==0){ xbt_dict_free(&((*info)->info_dict)); xbt_free(*info); } *info=MPI_INFO_NULL; return MPI_SUCCESS; } int PMPI_Info_get(MPI_Info info,char *key,int valuelen, char *value, int *flag){ *flag=false; if (info == nullptr || key == nullptr || valuelen <0) return MPI_ERR_ARG; if (value == nullptr) return MPI_ERR_INFO_VALUE; char* tmpvalue=static_cast(xbt_dict_get_or_null(info->info_dict, key)); if(tmpvalue){ memset(value, 0, valuelen); memcpy(value,tmpvalue, (strlen(tmpvalue) + 1 < static_cast(valuelen)) ? strlen(tmpvalue) + 1 : valuelen); *flag=true; } return MPI_SUCCESS; } int PMPI_Info_dup(MPI_Info info, MPI_Info *newinfo){ if (info == nullptr || newinfo==nullptr) return MPI_ERR_ARG; *newinfo = xbt_new(s_smpi_mpi_info_t, 1); (*newinfo)->info_dict= xbt_dict_new_homogeneous(xbt_free_f); (*newinfo)->refcount=1; xbt_dict_cursor_t cursor = nullptr; int *key; void* data; xbt_dict_foreach(info->info_dict,cursor,key,data){ xbt_dict_set((*newinfo)->info_dict, reinterpret_cast(key), xbt_strdup(reinterpret_cast(data)), nullptr); } return MPI_SUCCESS; } int PMPI_Info_delete(MPI_Info info, char *key){ if (info == nullptr || key==nullptr) return MPI_ERR_ARG; try { xbt_dict_remove(info->info_dict, key); } catch(xbt_ex& e){ return MPI_ERR_INFO_NOKEY; } return MPI_SUCCESS; } int PMPI_Info_get_nkeys( MPI_Info info, int *nkeys){ if (info == nullptr || nkeys==nullptr) return MPI_ERR_ARG; *nkeys=xbt_dict_size(info->info_dict); return MPI_SUCCESS; } int PMPI_Info_get_nthkey( MPI_Info info, int n, char *key){ if (info == nullptr || key==nullptr || n<0 || n> MPI_MAX_INFO_KEY) return MPI_ERR_ARG; xbt_dict_cursor_t cursor = nullptr; char *keyn; void* data; int num=0; xbt_dict_foreach(info->info_dict,cursor,keyn,data){ if(num==n){ strncpy(key,keyn,strlen(keyn)+1); xbt_dict_cursor_free(&cursor); return MPI_SUCCESS; } num++; } return MPI_ERR_ARG; } int PMPI_Info_get_valuelen( MPI_Info info, char *key, int *valuelen, int *flag){ *flag=false; if (info == nullptr || key == nullptr || valuelen==nullptr) return MPI_ERR_ARG; char* tmpvalue=(char*)xbt_dict_get_or_null(info->info_dict, key); if(tmpvalue){ *valuelen=strlen(tmpvalue); *flag=true; } return MPI_SUCCESS; } int PMPI_Unpack(void* inbuf, int incount, int* position, void* outbuf, int outcount, MPI_Datatype type, MPI_Comm comm) { if(incount<0 || outcount < 0 || inbuf==nullptr || outbuf==nullptr) return MPI_ERR_ARG; if(!is_datatype_valid(type)) return MPI_ERR_TYPE; if(comm==MPI_COMM_NULL) return MPI_ERR_COMM; return smpi_mpi_unpack(inbuf, incount, position, outbuf,outcount,type, comm); } int PMPI_Pack(void* inbuf, int incount, MPI_Datatype type, void* outbuf, int outcount, int* position, MPI_Comm comm) { if(incount<0 || outcount < 0|| inbuf==nullptr || outbuf==nullptr) return MPI_ERR_ARG; if(!is_datatype_valid(type)) return MPI_ERR_TYPE; if(comm==MPI_COMM_NULL) return MPI_ERR_COMM; return smpi_mpi_pack(inbuf, incount, type, outbuf,outcount,position, comm); } int PMPI_Pack_size(int incount, MPI_Datatype datatype, MPI_Comm comm, int* size) { if(incount<0) return MPI_ERR_ARG; if(!is_datatype_valid(datatype)) return MPI_ERR_TYPE; if(comm==MPI_COMM_NULL) return MPI_ERR_COMM; *size=incount*smpi_datatype_size(datatype); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/smpif90.in000644 001750 001750 00000004031 13031031276 020450 0ustar00mquinsonmquinson000000 000000 #! /bin/sh # Copyright (c) 2012-2015. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. SIMGRID_VERSION="@SIMGRID_VERSION_STRING@" SIMGRID_GITHASH="@SIMGRID_GITHASH@" F90=@SAVED_Fortran_COMPILER@ INCLUDEARGS="@includeflag@" CMAKE_LINKARGS="-L@libdir@" @SMPITOOLS_SH@ list_set FFLAGS "-ff2c" "-fno-second-underscore" list_set LINKARGS "-lsimgrid" "-lm" "-lgfortran" list_set TMPFILES main_name=main cleanup () { eval $(list_get TMPFILES) rm -f "$@" } trap 'cleanup' EXIT filter_and_compile() { list_add TMPFILES "${TMPFILE}" #replace "program main_name by subroutine user\_main (and the end clause as well)" sed 's/[[:space:]]*[pP][rR][oO][gG][rR][aA][mM][[:space:]]*\([a-zA-Z0-9\-\_]*\)/ subroutine user\_main /g;s/[[:space:]]*[uU][sS][eE][[:space:]]*[mM][pP][iI]/\include \"mpif\.h\" /g' "${ARG}" > "${TMPFILE}" SRCFILE="${TMPFILE}" list_add CMDLINE "${SRCFILE}" } list_set CMDLINE "${F90}" list_add_not_empty CMDLINE "${FFLAGS}" while [ $# -gt 0 ]; do ARG="$1" shift case "${ARG}" in -c) CMAKE_LINKARGS="" LINKARGS="" list_add CMDLINE "-c" ;; *.f90) TMPFILE=$(mymktemp "${ARG}" ".f90") filter_and_compile ;; *.F90) TMPFILE=$(mymktemp "${ARG}" ".F90") filter_and_compile ;; '-version' | '--version') printf '%b\n' "$SIMGRID_VERSION" exit 0 ;; "-git-version" | "--git-version") printf '%b\n' "$SIMGRID_GITHASH" exit 0 ;; '-compiler-version' | '--compiler-version') ${F90} --version ;; *) list_add CMDLINE "${ARG}" ;; esac done list_add_not_empty CMDLINE ${INCLUDEARGS} list_add_not_empty CMDLINE ${CMAKE_LINKARGS} list_add_not_empty CMDLINE "${LINKARGS}" eval $(list_get CMDLINE) "$@" SimGrid-3.14.159/src/smpi/smpicxx.in000755 001750 001750 00000003342 13031031276 020663 0ustar00mquinsonmquinson000000 000000 #! /bin/sh # Copyright (c) 2014-2015. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. SIMGRID_VERSION="@SIMGRID_VERSION_STRING@" SIMGRID_GITHASH="@SIMGRID_GITHASH@" CXX=@CMAKE_CXX_COMPILER@ INCLUDEARGS="@includeflag@" CMAKE_LINKARGS="-L@libdir@" @SMPITOOLS_SH@ list_set CXXFLAGS list_set LINKARGS if [ "@WIN32@" != "1" ]; then list_add CXXFLAGS "-Dmain=smpi_simulated_main_" list_add LINKARGS "-lsimgrid" else list_add CXXFLAGS "-include" "@includedir@/smpi/smpi_main.h" list_add LINKARGS "@libdir@\libsimgrid.dll" fi list_set CMDARGS while [ $# -gt 0 ]; do ARG="$1" shift case "${ARG}" in -c) CMAKE_LINKARGS="" LINKARGS="" list_add CMDARGS "-c" ;; *.c) SRCFILE="$(readlink -f ${ARG} 2>/dev/null)" if [ -z "$SRCFILE" ] ; then SRCFILE="$ARG" fi list_add CMDARGS "${SRCFILE}" ;; '-version' | '--version') printf '%b\n' "$SIMGRID_VERSION" exit 0 ;; "-git-version" | "--git-version") printf '%b\n' "$SIMGRID_GITHASH" exit 0 ;; '-compiler-version' | '--compiler-version') ${CXX} --version ;; *) list_add CMDARGS "${ARG}" ;; esac done list_set CMDLINE "${CXX}" list_add_not_empty CMDLINE "${CXXFLAGS}" list_add_not_empty CMDLINE ${INCLUDEARGS} list_add_not_empty CMDLINE ${CMAKE_LINKARGS} list_add_not_empty CMDLINE "${CMDARGS}" list_add_not_empty CMDLINE "${LINKARGS}" eval $(list_get CMDLINE) "$@" SimGrid-3.14.159/src/smpi/smpi_base.cpp000644 001750 001750 00000153265 13031031272 021311 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include "private.h" #include "xbt/virtu.h" #include "mc/mc.h" #include "src/mc/mc_replay.h" #include "xbt/replay.h" #include #include "src/simix/smx_private.h" #include "surf/surf.h" #include "simgrid/sg_config.h" #include "smpi/smpi_utils.hpp" #include "colls/colls.h" #include "src/kernel/activity/SynchroComm.hpp" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_base, smpi, "Logging specific to SMPI (base)"); static int match_recv(void* a, void* b, smx_activity_t ignored) { MPI_Request ref = static_cast(a); MPI_Request req = static_cast(b); XBT_DEBUG("Trying to match a recv of src %d against %d, tag %d against %d",ref->src,req->src, ref->tag, req->tag); xbt_assert(ref, "Cannot match recv against null reference"); xbt_assert(req, "Cannot match recv against null request"); if((ref->src == MPI_ANY_SOURCE || req->src == ref->src) && ((ref->tag == MPI_ANY_TAG && req->tag >=0) || req->tag == ref->tag)){ //we match, we can transfer some values if(ref->src == MPI_ANY_SOURCE) ref->real_src = req->src; if(ref->tag == MPI_ANY_TAG) ref->real_tag = req->tag; if(ref->real_size < req->real_size) ref->truncated = 1; if(req->detached==1) ref->detached_sender=req; //tie the sender to the receiver, as it is detached and has to be freed in the receiver XBT_DEBUG("match succeeded"); return 1; }else return 0; } static int match_send(void* a, void* b,smx_activity_t ignored) { MPI_Request ref = static_cast(a); MPI_Request req = static_cast(b); XBT_DEBUG("Trying to match a send of src %d against %d, tag %d against %d",ref->src,req->src, ref->tag, req->tag); xbt_assert(ref, "Cannot match send against null reference"); xbt_assert(req, "Cannot match send against null request"); if((req->src == MPI_ANY_SOURCE || req->src == ref->src) && ((req->tag == MPI_ANY_TAG && ref->tag >=0)|| req->tag == ref->tag)){ if(req->src == MPI_ANY_SOURCE) req->real_src = ref->src; if(req->tag == MPI_ANY_TAG) req->real_tag = ref->tag; if(req->real_size < ref->real_size) req->truncated = 1; if(ref->detached==1) req->detached_sender=ref; //tie the sender to the receiver, as it is detached and has to be freed in the receiver XBT_DEBUG("match succeeded"); return 1; } else return 0; } std::vector smpi_os_values; std::vector smpi_or_values; std::vector smpi_ois_values; static simgrid::config::Flag smpi_wtime_sleep( "smpi/wtime", "Minimum time to inject inside a call to MPI_Wtime", 0.0); static simgrid::config::Flag smpi_init_sleep( "smpi/init", "Time to inject inside a call to MPI_Init", 0.0); static simgrid::config::Flag smpi_iprobe_sleep( "smpi/iprobe", "Minimum time to inject inside a call to MPI_Iprobe", 1e-4); static simgrid::config::Flag smpi_test_sleep( "smpi/test", "Minimum time to inject inside a call to MPI_Test", 1e-4); static double smpi_os(size_t size) { if (smpi_os_values.empty()) { smpi_os_values = parse_factor(xbt_cfg_get_string("smpi/os")); } double current=smpi_os_values.empty()?0.0:smpi_os_values[0].values[0]+smpi_os_values[0].values[1]*size; // Iterate over all the sections that were specified and find the right // value. (fact.factor represents the interval sizes; we want to find the // section that has fact.factor <= size and no other such fact.factor <= size) // Note: parse_factor() (used before) already sorts the vector we iterate over! for (auto& fact : smpi_os_values) { if (size <= fact.factor) { // Values already too large, use the previously computed value of current! XBT_DEBUG("os : %zu <= %zu return %.10f", size, fact.factor, current); return current; }else{ // If the next section is too large, the current section must be used. // Hence, save the cost, as we might have to use it. current = fact.values[0]+fact.values[1]*size; } } XBT_DEBUG("Searching for smpi/os: %zu is larger than the largest boundary, return %.10f", size, current); return current; } static double smpi_ois(size_t size) { if (smpi_ois_values.empty()) { smpi_ois_values = parse_factor(xbt_cfg_get_string("smpi/ois")); } double current=smpi_ois_values.empty()?0.0:smpi_ois_values[0].values[0]+smpi_ois_values[0].values[1]*size; // Iterate over all the sections that were specified and find the right value. (fact.factor represents the interval // sizes; we want to find the section that has fact.factor <= size and no other such fact.factor <= size) // Note: parse_factor() (used before) already sorts the vector we iterate over! for (auto& fact : smpi_ois_values) { if (size <= fact.factor) { // Values already too large, use the previously computed value of current! XBT_DEBUG("ois : %zu <= %zu return %.10f", size, fact.factor, current); return current; }else{ // If the next section is too large, the current section must be used. // Hence, save the cost, as we might have to use it. current = fact.values[0]+fact.values[1]*size; } } XBT_DEBUG("Searching for smpi/ois: %zu is larger than the largest boundary, return %.10f", size, current); return current; } static double smpi_or(size_t size) { if (smpi_or_values.empty()) { smpi_or_values = parse_factor(xbt_cfg_get_string("smpi/or")); } double current=smpi_or_values.empty()?0.0:smpi_or_values.front().values[0]+smpi_or_values.front().values[1]*size; // Iterate over all the sections that were specified and find the right value. (fact.factor represents the interval // sizes; we want to find the section that has fact.factor <= size and no other such fact.factor <= size) // Note: parse_factor() (used before) already sorts the vector we iterate over! for (auto fact : smpi_or_values) { if (size <= fact.factor) { // Values already too large, use the previously computed value of current! XBT_DEBUG("or : %zu <= %zu return %.10f", size, fact.factor, current); return current; } else { // If the next section is too large, the current section must be used. // Hence, save the cost, as we might have to use it. current=fact.values[0]+fact.values[1]*size; } } XBT_DEBUG("smpi_or: %zu is larger than largest boundary, return %.10f", size, current); return current; } void smpi_mpi_init() { if(smpi_init_sleep > 0) simcall_process_sleep(smpi_init_sleep); } double smpi_mpi_wtime(){ double time; if (smpi_process_initialized() != 0 && smpi_process_finalized() == 0 && smpi_process_get_sampling() == 0) { smpi_bench_end(); time = SIMIX_get_clock(); // to avoid deadlocks if used as a break condition, such as // while (MPI_Wtime(...) < time_limit) { // .... // } // because the time will not normally advance when only calls to MPI_Wtime // are made -> deadlock (MPI_Wtime never reaches the time limit) if(smpi_wtime_sleep > 0) simcall_process_sleep(smpi_wtime_sleep); smpi_bench_begin(); } else { time = SIMIX_get_clock(); } return time; } static MPI_Request build_request(void *buf, int count, MPI_Datatype datatype, int src, int dst, int tag, MPI_Comm comm, unsigned flags) { MPI_Request request = nullptr; void *old_buf = nullptr; request = xbt_new(s_smpi_mpi_request_t, 1); s_smpi_subtype_t *subtype = static_cast(datatype->substruct); if((((flags & RECV) != 0) && ((flags & ACCUMULATE) !=0)) || (datatype->sizeof_substruct != 0)){ // This part handles the problem of non-contiguous memory old_buf = buf; buf = count==0 ? nullptr : xbt_malloc(count*smpi_datatype_size(datatype)); if ((datatype->sizeof_substruct != 0) && ((flags & SEND) != 0)) { subtype->serialize(old_buf, buf, count, datatype->substruct); } } request->buf = buf; // This part handles the problem of non-contiguous memory (for the unserialisation at the reception) request->old_buf = old_buf; request->old_type = datatype; request->size = smpi_datatype_size(datatype) * count; smpi_datatype_use(datatype); request->src = src; request->dst = dst; request->tag = tag; request->comm = comm; smpi_comm_use(request->comm); request->action = nullptr; request->flags = flags; request->detached = 0; request->detached_sender = nullptr; request->real_src = 0; request->truncated = 0; request->real_size = 0; request->real_tag = 0; if (flags & PERSISTENT) request->refcount = 1; else request->refcount = 0; request->op = MPI_REPLACE; request->send = 0; request->recv = 0; return request; } void smpi_empty_status(MPI_Status * status) { if(status != MPI_STATUS_IGNORE) { status->MPI_SOURCE = MPI_ANY_SOURCE; status->MPI_TAG = MPI_ANY_TAG; status->MPI_ERROR = MPI_SUCCESS; status->count=0; } } static void smpi_mpi_request_free_voidp(void* request) { MPI_Request req = static_cast(request); smpi_mpi_request_free(&req); } /* MPI Low level calls */ MPI_Request smpi_mpi_send_init(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ request = build_request(buf==MPI_BOTTOM ? nullptr : buf, count, datatype, smpi_process_index(), smpi_group_index(smpi_comm_group(comm), dst), tag, comm, PERSISTENT | SEND | PREPARED); return request; } MPI_Request smpi_mpi_ssend_init(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ request = build_request(buf==MPI_BOTTOM ? nullptr : buf, count, datatype, smpi_process_index(), smpi_group_index(smpi_comm_group(comm), dst), tag, comm, PERSISTENT | SSEND | SEND | PREPARED); return request; } MPI_Request smpi_mpi_recv_init(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ request = build_request(buf==MPI_BOTTOM ? nullptr : buf, count, datatype, src == MPI_ANY_SOURCE ? MPI_ANY_SOURCE : smpi_group_index(smpi_comm_group(comm), src), smpi_process_index(), tag, comm, PERSISTENT | RECV | PREPARED); return request; } void smpi_mpi_start(MPI_Request request) { smx_mailbox_t mailbox; xbt_assert(request->action == nullptr, "Cannot (re-)start unfinished communication"); request->flags &= ~PREPARED; request->flags &= ~FINISHED; request->refcount++; if ((request->flags & RECV) != 0) { print_request("New recv", request); int async_small_thresh = xbt_cfg_get_int("smpi/async-small-thresh"); xbt_mutex_t mut = smpi_process_mailboxes_mutex(); if (async_small_thresh != 0 || (request->flags & RMA) != 0) xbt_mutex_acquire(mut); if (async_small_thresh == 0 && (request->flags & RMA) == 0 ) { mailbox = smpi_process_mailbox(); } else if (((request->flags & RMA) != 0) || static_cast(request->size) < async_small_thresh) { //We have to check both mailboxes (because SSEND messages are sent to the large mbox). //begin with the more appropriate one : the small one. mailbox = smpi_process_mailbox_small(); XBT_DEBUG("Is there a corresponding send already posted in the small mailbox %p (in case of SSEND)?", mailbox); smx_activity_t action = simcall_comm_iprobe(mailbox, 0, request->src,request->tag, &match_recv, static_cast(request)); if (action == nullptr) { mailbox = smpi_process_mailbox(); XBT_DEBUG("No, nothing in the small mailbox test the other one : %p", mailbox); action = simcall_comm_iprobe(mailbox, 0, request->src,request->tag, &match_recv, static_cast(request)); if (action == nullptr) { XBT_DEBUG("Still nothing, switch back to the small mailbox : %p", mailbox); mailbox = smpi_process_mailbox_small(); } } else { XBT_DEBUG("yes there was something for us in the large mailbox"); } } else { mailbox = smpi_process_mailbox_small(); XBT_DEBUG("Is there a corresponding send already posted the small mailbox?"); smx_activity_t action = simcall_comm_iprobe(mailbox, 0, request->src,request->tag, &match_recv, (void*)request); if (action == nullptr) { XBT_DEBUG("No, nothing in the permanent receive mailbox"); mailbox = smpi_process_mailbox(); } else { XBT_DEBUG("yes there was something for us in the small mailbox"); } } // we make a copy here, as the size is modified by simix, and we may reuse the request in another receive later request->real_size=request->size; request->action = simcall_comm_irecv(SIMIX_process_self(), mailbox, request->buf, &request->real_size, &match_recv, ! smpi_process_get_replaying()? &smpi_comm_copy_buffer_callback : &smpi_comm_null_copy_buffer_callback, request, -1.0); XBT_DEBUG("recv simcall posted"); if (async_small_thresh != 0 || (request->flags & RMA) != 0 ) xbt_mutex_release(mut); } else { /* the RECV flag was not set, so this is a send */ int receiver = request->dst; int rank = request->src; if (TRACE_smpi_view_internals()) { TRACE_smpi_send(rank, rank, receiver, request->tag, request->size); } print_request("New send", request); void* buf = request->buf; if ((request->flags & SSEND) == 0 && ( (request->flags & RMA) != 0 || static_cast(request->size) < xbt_cfg_get_int("smpi/send-is-detached-thresh") ) ) { void *oldbuf = nullptr; request->detached = 1; XBT_DEBUG("Send request %p is detached", request); request->refcount++; if(request->old_type->sizeof_substruct == 0){ oldbuf = request->buf; if (!smpi_process_get_replaying() && oldbuf != nullptr && request->size!=0){ if((smpi_privatize_global_variables != 0) && (static_cast(request->buf) >= smpi_start_data_exe) && (static_cast(request->buf) < smpi_start_data_exe + smpi_size_data_exe )){ XBT_DEBUG("Privatization : We are sending from a zone inside global memory. Switch data segment "); smpi_switch_data_segment(request->src); } buf = xbt_malloc(request->size); memcpy(buf,oldbuf,request->size); XBT_DEBUG("buf %p copied into %p",oldbuf,buf); } } } //if we are giving back the control to the user without waiting for completion, we have to inject timings double sleeptime = 0.0; if(request->detached != 0 || ((request->flags & (ISEND|SSEND)) != 0)){// issend should be treated as isend //isend and send timings may be different sleeptime = ((request->flags & ISEND) != 0) ? smpi_ois(request->size) : smpi_os(request->size); } if(sleeptime > 0.0){ simcall_process_sleep(sleeptime); XBT_DEBUG("sending size of %zu : sleep %f ", request->size, sleeptime); } int async_small_thresh = xbt_cfg_get_int("smpi/async-small-thresh"); xbt_mutex_t mut=smpi_process_remote_mailboxes_mutex(receiver); if (async_small_thresh != 0 || (request->flags & RMA) != 0) xbt_mutex_acquire(mut); if (!(async_small_thresh != 0 || (request->flags & RMA) !=0)) { mailbox = smpi_process_remote_mailbox(receiver); } else if (((request->flags & RMA) != 0) || static_cast(request->size) < async_small_thresh) { // eager mode mailbox = smpi_process_remote_mailbox(receiver); XBT_DEBUG("Is there a corresponding recv already posted in the large mailbox %p?", mailbox); smx_activity_t action = simcall_comm_iprobe(mailbox, 1,request->dst, request->tag, &match_send, static_cast(request)); if (action == nullptr) { if ((request->flags & SSEND) == 0){ mailbox = smpi_process_remote_mailbox_small(receiver); XBT_DEBUG("No, nothing in the large mailbox, message is to be sent on the small one %p", mailbox); } else { mailbox = smpi_process_remote_mailbox_small(receiver); XBT_DEBUG("SSEND : Is there a corresponding recv already posted in the small mailbox %p?", mailbox); action = simcall_comm_iprobe(mailbox, 1,request->dst, request->tag, &match_send, static_cast(request)); if (action == nullptr) { XBT_DEBUG("No, we are first, send to large mailbox"); mailbox = smpi_process_remote_mailbox(receiver); } } } else { XBT_DEBUG("Yes there was something for us in the large mailbox"); } } else { mailbox = smpi_process_remote_mailbox(receiver); XBT_DEBUG("Send request %p is in the large mailbox %p (buf: %p)",mailbox, request,request->buf); } // we make a copy here, as the size is modified by simix, and we may reuse the request in another receive later request->real_size=request->size; request->action = simcall_comm_isend(SIMIX_process_from_PID(request->src+1), mailbox, request->size, -1.0, buf, request->real_size, &match_send, &xbt_free_f, // how to free the userdata if a detached send fails !smpi_process_get_replaying() ? &smpi_comm_copy_buffer_callback : &smpi_comm_null_copy_buffer_callback, request, // detach if msg size < eager/rdv switch limit request->detached); XBT_DEBUG("send simcall posted"); /* FIXME: detached sends are not traceable (request->action == nullptr) */ if (request->action != nullptr) simcall_set_category(request->action, TRACE_internal_smpi_get_category()); if (async_small_thresh != 0 || ((request->flags & RMA)!=0)) xbt_mutex_release(mut); } } void smpi_mpi_startall(int count, MPI_Request * requests) { if(requests== nullptr) return; for(int i = 0; i < count; i++) { smpi_mpi_start(requests[i]); } } void smpi_mpi_request_free(MPI_Request * request) { if((*request) != MPI_REQUEST_NULL){ (*request)->refcount--; if((*request)->refcount<0) xbt_die("wrong refcount"); if((*request)->refcount==0){ smpi_datatype_unuse((*request)->old_type); smpi_comm_unuse((*request)->comm); print_request("Destroying", (*request)); xbt_free(*request); *request = MPI_REQUEST_NULL; }else{ print_request("Decrementing", (*request)); } }else{ xbt_die("freeing an already free request"); } } MPI_Request smpi_rma_send_init(void *buf, int count, MPI_Datatype datatype, int src, int dst, int tag, MPI_Comm comm, MPI_Op op) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ if(op==MPI_OP_NULL){ request = build_request(buf==MPI_BOTTOM ? nullptr : buf , count, datatype, src, dst, tag, comm, RMA | NON_PERSISTENT | ISEND | SEND | PREPARED); }else{ request = build_request(buf==MPI_BOTTOM ? nullptr : buf, count, datatype, src, dst, tag, comm, RMA | NON_PERSISTENT | ISEND | SEND | PREPARED | ACCUMULATE); request->op = op; } return request; } MPI_Request smpi_rma_recv_init(void *buf, int count, MPI_Datatype datatype, int src, int dst, int tag, MPI_Comm comm, MPI_Op op) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ if(op==MPI_OP_NULL){ request = build_request(buf==MPI_BOTTOM ? nullptr : buf, count, datatype, src, dst, tag, comm, RMA | NON_PERSISTENT | RECV | PREPARED); }else{ request = build_request(buf==MPI_BOTTOM ? nullptr : buf, count, datatype, src, dst, tag, comm, RMA | NON_PERSISTENT | RECV | PREPARED | ACCUMULATE); request->op = op; } return request; } MPI_Request smpi_isend_init(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ request = build_request(buf==MPI_BOTTOM ? nullptr : buf , count, datatype, smpi_process_index(), smpi_group_index(smpi_comm_group(comm), dst), tag,comm, PERSISTENT | ISEND | SEND | PREPARED); return request; } MPI_Request smpi_mpi_isend(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ request = build_request(buf==MPI_BOTTOM ? nullptr : buf, count, datatype, smpi_process_index(), smpi_group_index(smpi_comm_group(comm), dst), tag, comm, NON_PERSISTENT | ISEND | SEND); smpi_mpi_start(request); return request; } MPI_Request smpi_mpi_issend(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ request = build_request(buf==MPI_BOTTOM ? nullptr : buf, count, datatype, smpi_process_index(), smpi_group_index(smpi_comm_group(comm), dst), tag,comm, NON_PERSISTENT | ISEND | SSEND | SEND); smpi_mpi_start(request); return request; } MPI_Request smpi_irecv_init(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ request = build_request(buf==MPI_BOTTOM ? nullptr : buf, count, datatype, src == MPI_ANY_SOURCE ? MPI_ANY_SOURCE : smpi_group_index(smpi_comm_group(comm), src), smpi_process_index(), tag, comm, PERSISTENT | RECV | PREPARED); return request; } MPI_Request smpi_mpi_irecv(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ request = build_request(buf==MPI_BOTTOM ? nullptr : buf, count, datatype, src == MPI_ANY_SOURCE ? MPI_ANY_SOURCE : smpi_group_index(smpi_comm_group(comm), src), smpi_process_index(), tag, comm, NON_PERSISTENT | RECV); smpi_mpi_start(request); return request; } void smpi_mpi_recv(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm, MPI_Status * status) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ request = smpi_mpi_irecv(buf, count, datatype, src, tag, comm); smpi_mpi_wait(&request, status); request = nullptr; } void smpi_mpi_send(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ request = build_request(buf==MPI_BOTTOM ? nullptr : buf, count, datatype, smpi_process_index(), smpi_group_index(smpi_comm_group(comm), dst), tag, comm, NON_PERSISTENT | SEND); smpi_mpi_start(request); smpi_mpi_wait(&request, MPI_STATUS_IGNORE); request = nullptr; } void smpi_mpi_ssend(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm) { MPI_Request request = nullptr; /* MC needs the comm to be set to nullptr during the call */ request = build_request(buf==MPI_BOTTOM ? nullptr : buf, count, datatype, smpi_process_index(), smpi_group_index(smpi_comm_group(comm), dst), tag, comm, NON_PERSISTENT | SSEND | SEND); smpi_mpi_start(request); smpi_mpi_wait(&request, MPI_STATUS_IGNORE); request = nullptr; } void smpi_mpi_sendrecv(void *sendbuf, int sendcount, MPI_Datatype sendtype,int dst, int sendtag, void *recvbuf, int recvcount, MPI_Datatype recvtype, int src, int recvtag, MPI_Comm comm, MPI_Status * status) { MPI_Request requests[2]; MPI_Status stats[2]; int myid=smpi_process_index(); if ((smpi_group_index(smpi_comm_group(comm), dst) == myid) && (smpi_group_index(smpi_comm_group(comm), src) == myid)){ smpi_datatype_copy(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype); return; } requests[0] = smpi_isend_init(sendbuf, sendcount, sendtype, dst, sendtag, comm); requests[1] = smpi_irecv_init(recvbuf, recvcount, recvtype, src, recvtag, comm); smpi_mpi_startall(2, requests); smpi_mpi_waitall(2, requests, stats); smpi_mpi_request_free(&requests[0]); smpi_mpi_request_free(&requests[1]); if(status != MPI_STATUS_IGNORE) { // Copy receive status *status = stats[1]; } } int smpi_mpi_get_count(MPI_Status * status, MPI_Datatype datatype) { return status->count / smpi_datatype_size(datatype); } static void finish_wait(MPI_Request * request, MPI_Status * status) { MPI_Request req = *request; smpi_empty_status(status); if(!((req->detached != 0) && ((req->flags & SEND) != 0)) && ((req->flags & PREPARED) == 0)){ if(status != MPI_STATUS_IGNORE) { int src = req->src == MPI_ANY_SOURCE ? req->real_src : req->src; status->MPI_SOURCE = smpi_group_rank(smpi_comm_group(req->comm), src); status->MPI_TAG = req->tag == MPI_ANY_TAG ? req->real_tag : req->tag; status->MPI_ERROR = req->truncated != 0 ? MPI_ERR_TRUNCATE : MPI_SUCCESS; // this handles the case were size in receive differs from size in send status->count = req->real_size; } print_request("Finishing", req); MPI_Datatype datatype = req->old_type; if(((req->flags & ACCUMULATE) != 0) || (datatype->sizeof_substruct != 0)){ if (!smpi_process_get_replaying()){ if( smpi_privatize_global_variables != 0 && (static_cast(req->old_buf) >= smpi_start_data_exe) && ((char*)req->old_buf < smpi_start_data_exe + smpi_size_data_exe )){ XBT_VERB("Privatization : We are unserializing to a zone in global memory - Switch data segment "); smpi_switch_data_segment(smpi_process_index()); } } if(datatype->sizeof_substruct != 0){ // This part handles the problem of non-contignous memory the unserialization at the reception s_smpi_subtype_t *subtype = static_cast(datatype->substruct); if(req->flags & RECV) subtype->unserialize(req->buf, req->old_buf, req->real_size/smpi_datatype_size(datatype) , datatype->substruct, req->op); xbt_free(req->buf); }else if(req->flags & RECV){//apply op on contiguous buffer for accumulate int n =req->real_size/smpi_datatype_size(datatype); smpi_op_apply(req->op, req->buf, req->old_buf, &n, &datatype); xbt_free(req->buf); } } } if (TRACE_smpi_view_internals() && ((req->flags & RECV) != 0)){ int rank = smpi_process_index(); int src_traced = (req->src == MPI_ANY_SOURCE ? req->real_src : req->src); TRACE_smpi_recv(rank, src_traced, rank,req->tag); } if(req->detached_sender != nullptr){ //integrate pseudo-timing for buffering of small messages, do not bother to execute the simcall if 0 double sleeptime = smpi_or(req->real_size); if(sleeptime > 0.0){ simcall_process_sleep(sleeptime); XBT_DEBUG("receiving size of %zu : sleep %f ", req->real_size, sleeptime); } smpi_mpi_request_free(&(req->detached_sender)); } if(req->flags & PERSISTENT) req->action = nullptr; req->flags |= FINISHED; smpi_mpi_request_free(request); } int smpi_mpi_test(MPI_Request * request, MPI_Status * status) { //assume that request is not MPI_REQUEST_NULL (filtered in PMPI_Test or smpi_mpi_testall before) // to avoid deadlocks if used as a break condition, such as // while (MPI_Test(request, flag, status) && flag) { // } // because the time will not normally advance when only calls to MPI_Test are made -> deadlock // multiplier to the sleeptime, to increase speed of execution, each failed test will increase it static int nsleeps = 1; if(smpi_test_sleep > 0) simcall_process_sleep(nsleeps*smpi_test_sleep); smpi_empty_status(status); int flag = 1; if (((*request)->flags & PREPARED) == 0) { if ((*request)->action != nullptr) flag = simcall_comm_test((*request)->action); if (flag) { finish_wait(request, status); nsleeps=1;//reset the number of sleeps we will do next time if (*request != MPI_REQUEST_NULL && ((*request)->flags & PERSISTENT)==0) *request = MPI_REQUEST_NULL; } else if (xbt_cfg_get_boolean("smpi/grow-injected-times")){ nsleeps++; } } return flag; } int smpi_mpi_testany(int count, MPI_Request requests[], int *index, MPI_Status * status) { std::vector comms; comms.reserve(count); int i; int flag = 0; *index = MPI_UNDEFINED; std::vector map; /** Maps all matching comms back to their location in requests **/ for(i = 0; i < count; i++) { if ((requests[i] != MPI_REQUEST_NULL) && requests[i]->action && !(requests[i]->flags & PREPARED)) { comms.push_back(requests[i]->action); map.push_back(i); } } if(!map.empty()) { //multiplier to the sleeptime, to increase speed of execution, each failed testany will increase it static int nsleeps = 1; if(smpi_test_sleep > 0) simcall_process_sleep(nsleeps*smpi_test_sleep); i = simcall_comm_testany(comms.data(), comms.size()); // The i-th element in comms matches! if (i != -1) { // -1 is not MPI_UNDEFINED but a SIMIX return code. (nothing matches) *index = map[i]; finish_wait(&requests[*index], status); flag = 1; nsleeps = 1; if (requests[*index] != MPI_REQUEST_NULL && (requests[*index]->flags & NON_PERSISTENT)) { requests[*index] = MPI_REQUEST_NULL; } } else { nsleeps++; } } else { //all requests are null or inactive, return true flag = 1; smpi_empty_status(status); } return flag; } int smpi_mpi_testall(int count, MPI_Request requests[], MPI_Status status[]) { MPI_Status stat; MPI_Status *pstat = status == MPI_STATUSES_IGNORE ? MPI_STATUS_IGNORE : &stat; int flag=1; for(int i=0; iflags & PREPARED)) { if (smpi_mpi_test(&requests[i], pstat)!=1){ flag=0; }else{ requests[i]=MPI_REQUEST_NULL; } }else{ smpi_empty_status(pstat); } if(status != MPI_STATUSES_IGNORE) { status[i] = *pstat; } } return flag; } void smpi_mpi_probe(int source, int tag, MPI_Comm comm, MPI_Status* status){ int flag=0; //FIXME find another way to avoid busy waiting ? // the issue here is that we have to wait on a nonexistent comm while(flag==0){ smpi_mpi_iprobe(source, tag, comm, &flag, status); XBT_DEBUG("Busy Waiting on probing : %d", flag); } } void smpi_mpi_iprobe(int source, int tag, MPI_Comm comm, int* flag, MPI_Status* status){ MPI_Request request = build_request(nullptr, 0, MPI_CHAR, source == MPI_ANY_SOURCE ? MPI_ANY_SOURCE : smpi_group_index(smpi_comm_group(comm), source), smpi_comm_rank(comm), tag, comm, PERSISTENT | RECV); // to avoid deadlock, we have to sleep some time here, or the timer won't advance and we will only do iprobe simcalls // (especially when used as a break condition, such as while(MPI_Iprobe(...)) ... ) // multiplier to the sleeptime, to increase speed of execution, each failed iprobe will increase it static int nsleeps = 1; if(smpi_iprobe_sleep > 0) simcall_process_sleep(nsleeps*smpi_iprobe_sleep); // behave like a receive, but don't do it smx_mailbox_t mailbox; print_request("New iprobe", request); // We have to test both mailboxes as we don't know if we will receive one one or another if (xbt_cfg_get_int("smpi/async-small-thresh") > 0){ mailbox = smpi_process_mailbox_small(); XBT_DEBUG("Trying to probe the perm recv mailbox"); request->action = simcall_comm_iprobe(mailbox, 0, request->src, request->tag, &match_recv, static_cast(request)); } if (request->action == nullptr){ mailbox = smpi_process_mailbox(); XBT_DEBUG("trying to probe the other mailbox"); request->action = simcall_comm_iprobe(mailbox, 0, request->src,request->tag, &match_recv, static_cast(request)); } if (request->action != nullptr){ simgrid::kernel::activity::Comm *sync_comm = static_cast(request->action); MPI_Request req = static_cast(sync_comm->src_data); *flag = 1; if(status != MPI_STATUS_IGNORE && (req->flags & PREPARED) == 0) { status->MPI_SOURCE = smpi_group_rank(smpi_comm_group(comm), req->src); status->MPI_TAG = req->tag; status->MPI_ERROR = MPI_SUCCESS; status->count = req->real_size; } nsleeps = 1;//reset the number of sleeps we will do next time } else { *flag = 0; if (xbt_cfg_get_boolean("smpi/grow-injected-times")) nsleeps++; } smpi_mpi_request_free(&request); return; } void smpi_mpi_wait(MPI_Request * request, MPI_Status * status) { print_request("Waiting", *request); if ((*request)->flags & PREPARED) { smpi_empty_status(status); return; } if ((*request)->action != nullptr) // this is not a detached send simcall_comm_wait((*request)->action, -1.0); finish_wait(request, status); if (*request != MPI_REQUEST_NULL && (((*request)->flags & NON_PERSISTENT)!=0)) *request = MPI_REQUEST_NULL; } int smpi_mpi_waitany(int count, MPI_Request requests[], MPI_Status * status) { xbt_dynar_t comms; int i; int size = 0; int index = MPI_UNDEFINED; int *map; if(count > 0) { // Wait for a request to complete comms = xbt_dynar_new(sizeof(smx_activity_t), nullptr); map = xbt_new(int, count); XBT_DEBUG("Wait for one of %d", count); for(i = 0; i < count; i++) { if (requests[i] != MPI_REQUEST_NULL && !(requests[i]->flags & PREPARED) && !(requests[i]->flags & FINISHED)) { if (requests[i]->action != nullptr) { XBT_DEBUG("Waiting any %p ", requests[i]); xbt_dynar_push(comms, &requests[i]->action); map[size] = i; size++; }else{ //This is a finished detached request, let's return this one size=0;//so we free the dynar but don't do the waitany call index=i; finish_wait(&requests[i], status);//cleanup if refcount = 0 if (requests[i] != MPI_REQUEST_NULL && (requests[i]->flags & NON_PERSISTENT)) requests[i]=MPI_REQUEST_NULL;//set to null break; } } } if(size > 0) { i = simcall_comm_waitany(comms, -1); // not MPI_UNDEFINED, as this is a simix return code if (i != -1) { index = map[i]; finish_wait(&requests[index], status); if (requests[i] != MPI_REQUEST_NULL && (requests[i]->flags & NON_PERSISTENT)) requests[index] = MPI_REQUEST_NULL; } } xbt_free(map); xbt_dynar_free(&comms); } if (index==MPI_UNDEFINED) smpi_empty_status(status); return index; } int smpi_mpi_waitall(int count, MPI_Request requests[], MPI_Status status[]) { int index, c; MPI_Status stat; MPI_Status *pstat = status == MPI_STATUSES_IGNORE ? MPI_STATUS_IGNORE : &stat; int retvalue = MPI_SUCCESS; //tag invalid requests in the set if (status != MPI_STATUSES_IGNORE) { for (c = 0; c < count; c++) { if (requests[c] == MPI_REQUEST_NULL || requests[c]->dst == MPI_PROC_NULL || (requests[c]->flags & PREPARED)) { smpi_empty_status(&status[c]); } else if (requests[c]->src == MPI_PROC_NULL) { smpi_empty_status(&status[c]); status[c].MPI_SOURCE = MPI_PROC_NULL; } } } for(c = 0; c < count; c++) { if (MC_is_active() || MC_record_replay_is_active()) { smpi_mpi_wait(&requests[c], pstat); index = c; } else { index = smpi_mpi_waitany(count, requests, pstat); if (index == MPI_UNDEFINED) break; if (requests[index] != MPI_REQUEST_NULL && (requests[index]->flags & NON_PERSISTENT)) requests[index]=MPI_REQUEST_NULL; } if (status != MPI_STATUSES_IGNORE) { status[index] = *pstat; if (status[index].MPI_ERROR == MPI_ERR_TRUNCATE) retvalue = MPI_ERR_IN_STATUS; } } return retvalue; } int smpi_mpi_waitsome(int incount, MPI_Request requests[], int *indices, MPI_Status status[]) { int i; int count = 0; int index; MPI_Status stat; MPI_Status *pstat = status == MPI_STATUSES_IGNORE ? MPI_STATUS_IGNORE : &stat; for(i = 0; i < incount; i++) { index=smpi_mpi_waitany(incount, requests, pstat); if(index!=MPI_UNDEFINED){ indices[count] = index; count++; if(status != MPI_STATUSES_IGNORE) { status[index] = *pstat; } if (requests[index] != MPI_REQUEST_NULL && (requests[index]->flags & NON_PERSISTENT)) requests[index]=MPI_REQUEST_NULL; }else{ return MPI_UNDEFINED; } } return count; } int smpi_mpi_testsome(int incount, MPI_Request requests[], int *indices, MPI_Status status[]) { int i; int count = 0; int count_dead = 0; MPI_Status stat; MPI_Status *pstat = status == MPI_STATUSES_IGNORE ? MPI_STATUS_IGNORE : &stat; for(i = 0; i < incount; i++) { if((requests[i] != MPI_REQUEST_NULL)) { if(smpi_mpi_test(&requests[i], pstat)) { indices[i] = 1; count++; if(status != MPI_STATUSES_IGNORE) { status[i] = *pstat; } if ((requests[i] != MPI_REQUEST_NULL) && requests[i]->flags & NON_PERSISTENT) requests[i]=MPI_REQUEST_NULL; } }else{ count_dead++; } } if(count_dead==incount) return MPI_UNDEFINED; else return count; } void smpi_mpi_bcast(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { smpi_coll_tuned_bcast_binomial_tree(buf, count, datatype, root, comm); } void smpi_mpi_barrier(MPI_Comm comm) { smpi_coll_tuned_barrier_ompi_basic_linear(comm); } void smpi_mpi_gather(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm) { int system_tag = COLL_TAG_GATHER; MPI_Aint lb = 0; MPI_Aint recvext = 0; int rank = smpi_comm_rank(comm); int size = smpi_comm_size(comm); if(rank != root) { // Send buffer to root smpi_mpi_send(sendbuf, sendcount, sendtype, root, system_tag, comm); } else { smpi_datatype_extent(recvtype, &lb, &recvext); // Local copy from root smpi_datatype_copy(sendbuf, sendcount, sendtype, static_cast(recvbuf) + root * recvcount * recvext, recvcount, recvtype); // Receive buffers from senders MPI_Request *requests = xbt_new(MPI_Request, size - 1); int index = 0; for (int src = 0; src < size; src++) { if(src != root) { requests[index] = smpi_irecv_init(static_cast(recvbuf) + src * recvcount * recvext, recvcount, recvtype, src, system_tag, comm); index++; } } // Wait for completion of irecv's. smpi_mpi_startall(size - 1, requests); smpi_mpi_waitall(size - 1, requests, MPI_STATUS_IGNORE); for (int src = 0; src < size-1; src++) { smpi_mpi_request_free(&requests[src]); } xbt_free(requests); } } void smpi_mpi_reduce_scatter(void *sendbuf, void *recvbuf, int *recvcounts, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int rank = smpi_comm_rank(comm); /* arbitrarily choose root as rank 0 */ int size = smpi_comm_size(comm); int count = 0; int *displs = xbt_new(int, size); for (int i = 0; i < size; i++) { displs[i] = count; count += recvcounts[i]; } void *tmpbuf = static_cast(smpi_get_tmp_sendbuffer(count*smpi_datatype_get_extent(datatype))); mpi_coll_reduce_fun(sendbuf, tmpbuf, count, datatype, op, 0, comm); smpi_mpi_scatterv(tmpbuf, recvcounts, displs, datatype, recvbuf, recvcounts[rank], datatype, 0, comm); xbt_free(displs); smpi_free_tmp_buffer(tmpbuf); } void smpi_mpi_gatherv(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvtype, int root, MPI_Comm comm) { int system_tag = COLL_TAG_GATHERV; MPI_Aint lb = 0; MPI_Aint recvext = 0; int rank = smpi_comm_rank(comm); int size = smpi_comm_size(comm); if (rank != root) { // Send buffer to root smpi_mpi_send(sendbuf, sendcount, sendtype, root, system_tag, comm); } else { smpi_datatype_extent(recvtype, &lb, &recvext); // Local copy from root smpi_datatype_copy(sendbuf, sendcount, sendtype, static_cast(recvbuf) + displs[root] * recvext, recvcounts[root], recvtype); // Receive buffers from senders MPI_Request *requests = xbt_new(MPI_Request, size - 1); int index = 0; for (int src = 0; src < size; src++) { if(src != root) { requests[index] = smpi_irecv_init(static_cast(recvbuf) + displs[src] * recvext, recvcounts[src], recvtype, src, system_tag, comm); index++; } } // Wait for completion of irecv's. smpi_mpi_startall(size - 1, requests); smpi_mpi_waitall(size - 1, requests, MPI_STATUS_IGNORE); for (int src = 0; src < size-1; src++) { smpi_mpi_request_free(&requests[src]); } xbt_free(requests); } } void smpi_mpi_allgather(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf,int recvcount, MPI_Datatype recvtype, MPI_Comm comm) { int system_tag = COLL_TAG_ALLGATHER; MPI_Aint lb = 0; MPI_Aint recvext = 0; MPI_Request *requests; int rank = smpi_comm_rank(comm); int size = smpi_comm_size(comm); // FIXME: check for errors smpi_datatype_extent(recvtype, &lb, &recvext); // Local copy from self smpi_datatype_copy(sendbuf, sendcount, sendtype, static_cast(recvbuf) + rank * recvcount * recvext, recvcount, recvtype); // Send/Recv buffers to/from others; requests = xbt_new(MPI_Request, 2 * (size - 1)); int index = 0; for (int other = 0; other < size; other++) { if(other != rank) { requests[index] = smpi_isend_init(sendbuf, sendcount, sendtype, other, system_tag,comm); index++; requests[index] = smpi_irecv_init(static_cast(recvbuf) + other * recvcount * recvext, recvcount, recvtype, other, system_tag, comm); index++; } } // Wait for completion of all comms. smpi_mpi_startall(2 * (size - 1), requests); smpi_mpi_waitall(2 * (size - 1), requests, MPI_STATUS_IGNORE); for (int other = 0; other < 2*(size-1); other++) { smpi_mpi_request_free(&requests[other]); } xbt_free(requests); } void smpi_mpi_allgatherv(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvtype, MPI_Comm comm) { int system_tag = COLL_TAG_ALLGATHERV; MPI_Aint lb = 0; MPI_Aint recvext = 0; int rank = smpi_comm_rank(comm); int size = smpi_comm_size(comm); smpi_datatype_extent(recvtype, &lb, &recvext); // Local copy from self smpi_datatype_copy(sendbuf, sendcount, sendtype, static_cast(recvbuf) + displs[rank] * recvext,recvcounts[rank], recvtype); // Send buffers to others; MPI_Request *requests = xbt_new(MPI_Request, 2 * (size - 1)); int index = 0; for (int other = 0; other < size; other++) { if(other != rank) { requests[index] = smpi_isend_init(sendbuf, sendcount, sendtype, other, system_tag, comm); index++; requests[index] = smpi_irecv_init(static_cast(recvbuf) + displs[other] * recvext, recvcounts[other], recvtype, other, system_tag, comm); index++; } } // Wait for completion of all comms. smpi_mpi_startall(2 * (size - 1), requests); smpi_mpi_waitall(2 * (size - 1), requests, MPI_STATUS_IGNORE); for (int other = 0; other < 2*(size-1); other++) { smpi_mpi_request_free(&requests[other]); } xbt_free(requests); } void smpi_mpi_scatter(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm) { int system_tag = COLL_TAG_SCATTER; MPI_Aint lb = 0; MPI_Aint sendext = 0; MPI_Request *requests; int rank = smpi_comm_rank(comm); int size = smpi_comm_size(comm); if(rank != root) { // Recv buffer from root smpi_mpi_recv(recvbuf, recvcount, recvtype, root, system_tag, comm, MPI_STATUS_IGNORE); } else { smpi_datatype_extent(sendtype, &lb, &sendext); // Local copy from root if(recvbuf!=MPI_IN_PLACE){ smpi_datatype_copy(static_cast(sendbuf) + root * sendcount * sendext, sendcount, sendtype, recvbuf, recvcount, recvtype); } // Send buffers to receivers requests = xbt_new(MPI_Request, size - 1); int index = 0; for(int dst = 0; dst < size; dst++) { if(dst != root) { requests[index] = smpi_isend_init(static_cast(sendbuf) + dst * sendcount * sendext, sendcount, sendtype, dst, system_tag, comm); index++; } } // Wait for completion of isend's. smpi_mpi_startall(size - 1, requests); smpi_mpi_waitall(size - 1, requests, MPI_STATUS_IGNORE); for (int dst = 0; dst < size-1; dst++) { smpi_mpi_request_free(&requests[dst]); } xbt_free(requests); } } void smpi_mpi_scatterv(void *sendbuf, int *sendcounts, int *displs, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm) { int system_tag = COLL_TAG_SCATTERV; MPI_Aint lb = 0; MPI_Aint sendext = 0; int rank = smpi_comm_rank(comm); int size = smpi_comm_size(comm); if(rank != root) { // Recv buffer from root smpi_mpi_recv(recvbuf, recvcount, recvtype, root, system_tag, comm, MPI_STATUS_IGNORE); } else { smpi_datatype_extent(sendtype, &lb, &sendext); // Local copy from root if(recvbuf!=MPI_IN_PLACE){ smpi_datatype_copy(static_cast(sendbuf) + displs[root] * sendext, sendcounts[root], sendtype, recvbuf, recvcount, recvtype); } // Send buffers to receivers MPI_Request *requests = xbt_new(MPI_Request, size - 1); int index = 0; for (int dst = 0; dst < size; dst++) { if (dst != root) { requests[index] = smpi_isend_init(static_cast(sendbuf) + displs[dst] * sendext, sendcounts[dst], sendtype, dst, system_tag, comm); index++; } } // Wait for completion of isend's. smpi_mpi_startall(size - 1, requests); smpi_mpi_waitall(size - 1, requests, MPI_STATUS_IGNORE); for (int dst = 0; dst < size-1; dst++) { smpi_mpi_request_free(&requests[dst]); } xbt_free(requests); } } void smpi_mpi_reduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { int system_tag = COLL_TAG_REDUCE; MPI_Aint lb = 0; MPI_Aint dataext = 0; char* sendtmpbuf = static_cast(sendbuf); int rank = smpi_comm_rank(comm); int size = smpi_comm_size(comm); //non commutative case, use a working algo from openmpi if(!smpi_op_is_commute(op)){ smpi_coll_tuned_reduce_ompi_basic_linear(sendtmpbuf, recvbuf, count, datatype, op, root, comm); return; } if( sendbuf == MPI_IN_PLACE ) { sendtmpbuf = static_cast(smpi_get_tmp_sendbuffer(count*smpi_datatype_get_extent(datatype))); smpi_datatype_copy(recvbuf, count, datatype,sendtmpbuf, count, datatype); } if(rank != root) { // Send buffer to root smpi_mpi_send(sendtmpbuf, count, datatype, root, system_tag, comm); } else { smpi_datatype_extent(datatype, &lb, &dataext); // Local copy from root if (sendtmpbuf != nullptr && recvbuf != nullptr) smpi_datatype_copy(sendtmpbuf, count, datatype, recvbuf, count, datatype); // Receive buffers from senders MPI_Request *requests = xbt_new(MPI_Request, size - 1); void **tmpbufs = xbt_new(void *, size - 1); int index = 0; for (int src = 0; src < size; src++) { if (src != root) { if (!smpi_process_get_replaying()) tmpbufs[index] = xbt_malloc(count * dataext); else tmpbufs[index] = smpi_get_tmp_sendbuffer(count * dataext); requests[index] = smpi_irecv_init(tmpbufs[index], count, datatype, src, system_tag, comm); index++; } } // Wait for completion of irecv's. smpi_mpi_startall(size - 1, requests); for (int src = 0; src < size - 1; src++) { index = smpi_mpi_waitany(size - 1, requests, MPI_STATUS_IGNORE); XBT_DEBUG("finished waiting any request with index %d", index); if(index == MPI_UNDEFINED) { break; }else{ smpi_mpi_request_free(&requests[index]); } if(op) /* op can be MPI_OP_NULL that does nothing */ smpi_op_apply(op, tmpbufs[index], recvbuf, &count, &datatype); } for(index = 0; index < size - 1; index++) { smpi_free_tmp_buffer(tmpbufs[index]); } xbt_free(tmpbufs); xbt_free(requests); } if( sendbuf == MPI_IN_PLACE ) { smpi_free_tmp_buffer(sendtmpbuf); } } void smpi_mpi_allreduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { smpi_mpi_reduce(sendbuf, recvbuf, count, datatype, op, 0, comm); smpi_mpi_bcast(recvbuf, count, datatype, 0, comm); } void smpi_mpi_scan(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int system_tag = -888; MPI_Aint lb = 0, dataext = 0; int rank = smpi_comm_rank(comm); int size = smpi_comm_size(comm); smpi_datatype_extent(datatype, &lb, &dataext); // Local copy from self smpi_datatype_copy(sendbuf, count, datatype, recvbuf, count, datatype); // Send/Recv buffers to/from others; MPI_Request *requests = xbt_new(MPI_Request, size - 1); void **tmpbufs = xbt_new(void *, rank); int index = 0; for (int other = 0; other < rank; other++) { tmpbufs[index] = smpi_get_tmp_sendbuffer(count * dataext); requests[index] = smpi_irecv_init(tmpbufs[index], count, datatype, other, system_tag, comm); index++; } for (int other = rank + 1; other < size; other++) { requests[index] = smpi_isend_init(sendbuf, count, datatype, other, system_tag, comm); index++; } // Wait for completion of all comms. smpi_mpi_startall(size - 1, requests); if(smpi_op_is_commute(op)){ for (int other = 0; other < size - 1; other++) { index = smpi_mpi_waitany(size - 1, requests, MPI_STATUS_IGNORE); if(index == MPI_UNDEFINED) { break; } if(index < rank) { // #Request is below rank: it's a irecv smpi_op_apply(op, tmpbufs[index], recvbuf, &count, &datatype); } } }else{ //non commutative case, wait in order for (int other = 0; other < size - 1; other++) { smpi_mpi_wait(&(requests[other]), MPI_STATUS_IGNORE); if(index < rank) { smpi_op_apply(op, tmpbufs[other], recvbuf, &count, &datatype); } } } for(index = 0; index < rank; index++) { smpi_free_tmp_buffer(tmpbufs[index]); } for(index = 0; index < size-1; index++) { smpi_mpi_request_free(&requests[index]); } xbt_free(tmpbufs); xbt_free(requests); } void smpi_mpi_exscan(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int system_tag = -888; MPI_Aint lb = 0, dataext = 0; int recvbuf_is_empty=1; int rank = smpi_comm_rank(comm); int size = smpi_comm_size(comm); smpi_datatype_extent(datatype, &lb, &dataext); // Send/Recv buffers to/from others; MPI_Request *requests = xbt_new(MPI_Request, size - 1); void **tmpbufs = xbt_new(void *, rank); int index = 0; for (int other = 0; other < rank; other++) { tmpbufs[index] = smpi_get_tmp_sendbuffer(count * dataext); requests[index] = smpi_irecv_init(tmpbufs[index], count, datatype, other, system_tag, comm); index++; } for (int other = rank + 1; other < size; other++) { requests[index] = smpi_isend_init(sendbuf, count, datatype, other, system_tag, comm); index++; } // Wait for completion of all comms. smpi_mpi_startall(size - 1, requests); if(smpi_op_is_commute(op)){ for (int other = 0; other < size - 1; other++) { index = smpi_mpi_waitany(size - 1, requests, MPI_STATUS_IGNORE); if(index == MPI_UNDEFINED) { break; } if(index < rank) { if(recvbuf_is_empty){ smpi_datatype_copy(tmpbufs[index], count, datatype, recvbuf, count, datatype); recvbuf_is_empty=0; } else // #Request is below rank: it's a irecv smpi_op_apply(op, tmpbufs[index], recvbuf, &count, &datatype); } } }else{ //non commutative case, wait in order for (int other = 0; other < size - 1; other++) { smpi_mpi_wait(&(requests[other]), MPI_STATUS_IGNORE); if(index < rank) { if(recvbuf_is_empty){ smpi_datatype_copy(tmpbufs[other], count, datatype, recvbuf, count, datatype); recvbuf_is_empty=0; } else smpi_op_apply(op, tmpbufs[other], recvbuf, &count, &datatype); } } } for(index = 0; index < rank; index++) { smpi_free_tmp_buffer(tmpbufs[index]); } for(index = 0; index < size-1; index++) { smpi_mpi_request_free(&requests[index]); } xbt_free(tmpbufs); xbt_free(requests); } SimGrid-3.14.159/src/smpi/smpicc.in000755 001750 001750 00000003513 13031031276 020446 0ustar00mquinsonmquinson000000 000000 #! /bin/sh # Copyright (c) 2007-2015. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. SIMGRID_VERSION="@SIMGRID_VERSION_STRING@" SIMGRID_GITHASH="@SIMGRID_GITHASH@" CC=@CMAKE_C_COMPILER@ INCLUDEARGS="@includeflag@" CMAKE_LINKARGS="-L@libdir@" @SMPITOOLS_SH@ list_set CFLAGS list_set LINKARGS if [ "@WIN32@" != "1" ]; then list_add CFLAGS "-Dmain=smpi_simulated_main_" list_add LINKARGS "-lsimgrid" else list_add CFLAGS "-include" "@includedir@/smpi/smpi_main.h" list_add LINKARGS "@libdir@\libsimgrid.dll" fi list_set CMDARGS while [ $# -gt 0 ]; do ARG="$1" shift case "${ARG}" in -c) CMAKE_LINKARGS="" LINKARGS="" list_add CMDARGS "-c" ;; *.c) SRCFILE="$(readlink -f ${ARG} 2>/dev/null)" if [ -z "$SRCFILE" ] ; then SRCFILE="$ARG" fi list_add CMDARGS "${SRCFILE}" ;; '-version' | '--version') printf '%b\n' "$SIMGRID_VERSION" exit 0 ;; "-git-version" | "--git-version") printf '%b\n' "$SIMGRID_GITHASH" exit 0 ;; '-trace-call-location') list_add_not_empty CMDARGS "-DTRACE_CALL_LOCATION" ;; '-compiler-version' | '--compiler-version') ${CC} --version ;; *) list_add CMDARGS "${ARG}" ;; esac done list_set CMDLINE "${CC}" list_add_not_empty CMDLINE "${CFLAGS}" list_add_not_empty CMDLINE ${INCLUDEARGS} list_add_not_empty CMDLINE ${CMAKE_LINKARGS} list_add_not_empty CMDLINE "${CMDARGS}" list_add_not_empty CMDLINE "${LINKARGS}" eval $(list_get CMDLINE) "$@" SimGrid-3.14.159/src/smpi/smpi_coll.cpp000644 001750 001750 00000033437 13031031272 021326 0ustar00mquinsonmquinson000000 000000 /* smpi_coll.c -- various optimized routing for collectives */ /* Copyright (c) 2009-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include "private.h" #include "colls/colls.h" #include "simgrid/sg_config.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_coll, smpi, "Logging specific to SMPI (coll)"); s_mpi_coll_description_t mpi_coll_gather_description[] = { {"default", "gather default collective", reinterpret_cast(&smpi_mpi_gather)}, COLL_GATHERS(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr} /* this array must be nullptr terminated */ }; s_mpi_coll_description_t mpi_coll_allgather_description[] = { {"default", "allgather default collective", reinterpret_cast(&smpi_mpi_allgather)}, COLL_ALLGATHERS(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr}}; s_mpi_coll_description_t mpi_coll_allgatherv_description[] = {{"default", "allgatherv default collective", reinterpret_cast(&smpi_mpi_allgatherv)}, COLL_ALLGATHERVS(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr} /* this array must be nullptr terminated */ }; s_mpi_coll_description_t mpi_coll_allreduce_description[] = { {"default", "allreduce default collective", reinterpret_cast(&smpi_mpi_allreduce)}, COLL_ALLREDUCES(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr} /* this array must be nullptr terminated */ }; s_mpi_coll_description_t mpi_coll_reduce_scatter_description[] = {{"default", "reduce_scatter default collective", reinterpret_cast(&smpi_mpi_reduce_scatter)}, COLL_REDUCE_SCATTERS(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr} /* this array must be nullptr terminated */ }; s_mpi_coll_description_t mpi_coll_scatter_description[] = { {"default", "scatter default collective", reinterpret_cast(&smpi_mpi_scatter)}, COLL_SCATTERS(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr}}; s_mpi_coll_description_t mpi_coll_barrier_description[] = { {"default", "barrier default collective", reinterpret_cast(&smpi_mpi_barrier)}, COLL_BARRIERS(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr}}; s_mpi_coll_description_t mpi_coll_alltoall_description[] = { {"default", "Ompi alltoall default collective", reinterpret_cast(&smpi_coll_tuned_alltoall_ompi2)}, COLL_ALLTOALLS(COLL_DESCRIPTION, COLL_COMMA), {"bruck", "Alltoall Bruck (SG) collective", reinterpret_cast(&smpi_coll_tuned_alltoall_bruck)}, {"basic_linear", "Alltoall basic linear (SG) collective", reinterpret_cast(&smpi_coll_tuned_alltoall_basic_linear)}, {nullptr, nullptr, nullptr}}; s_mpi_coll_description_t mpi_coll_alltoallv_description[] = { {"default", "Ompi alltoallv default collective", reinterpret_cast(&smpi_coll_basic_alltoallv)}, COLL_ALLTOALLVS(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr} /* this array must be nullptr terminated */ }; s_mpi_coll_description_t mpi_coll_bcast_description[] = { {"default", "bcast default collective ", reinterpret_cast(&smpi_mpi_bcast)}, COLL_BCASTS(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr}}; s_mpi_coll_description_t mpi_coll_reduce_description[] = { {"default", "reduce default collective", reinterpret_cast(&smpi_mpi_reduce)}, COLL_REDUCES(COLL_DESCRIPTION, COLL_COMMA), {nullptr, nullptr, nullptr} }; /** Displays the long description of all registered models, and quit */ void coll_help(const char *category, s_mpi_coll_description_t * table) { printf("Long description of the %s models accepted by this simulator:\n", category); for (int i = 0; table[i].name; i++) printf(" %s: %s\n", table[i].name, table[i].description); } int find_coll_description(s_mpi_coll_description_t * table, char *name, const char *desc) { char *name_list = nullptr; int selector_on=0; if (name==nullptr || name[0] == '\0') { //no argument provided, use active selector's algorithm name=static_cast(xbt_cfg_get_string("smpi/coll-selector")); selector_on=1; } for (int i = 0; table[i].name; i++) if (!strcmp(name, table[i].name)) { if (strcmp(table[i].name,"default")) XBT_INFO("Switch to algorithm %s for collective %s",table[i].name,desc); return i; } if(selector_on){ // collective seems not handled by the active selector, try with default one for (int i = 0; table[i].name; i++) if (!strcmp("default", table[i].name)) { return i; } } if (!table[0].name) xbt_die("No collective is valid for '%s'! This is a bug.",name); name_list = xbt_strdup(table[0].name); for (int i = 1; table[i].name; i++) { name_list = static_cast(xbt_realloc(name_list, strlen(name_list) + strlen(table[i].name) + 3)); strncat(name_list, ", ",2); strncat(name_list, table[i].name, strlen(table[i].name)); } xbt_die("Collective '%s' is invalid! Valid collectives are: %s.", name, name_list); return -1; } int (*mpi_coll_gather_fun)(void *, int, MPI_Datatype, void*, int, MPI_Datatype, int root, MPI_Comm); int (*mpi_coll_allgather_fun)(void *, int, MPI_Datatype, void*, int, MPI_Datatype, MPI_Comm); int (*mpi_coll_allgatherv_fun)(void *, int, MPI_Datatype, void*, int*, int*, MPI_Datatype, MPI_Comm); int (*mpi_coll_allreduce_fun)(void *sbuf, void *rbuf, int rcount, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm); int (*mpi_coll_alltoall_fun)(void *, int, MPI_Datatype, void*, int, MPI_Datatype, MPI_Comm); int (*mpi_coll_alltoallv_fun)(void *, int*, int*, MPI_Datatype, void*, int*, int*, MPI_Datatype, MPI_Comm); int (*mpi_coll_bcast_fun)(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm com); int (*mpi_coll_reduce_fun)(void *buf, void *rbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm); int (*mpi_coll_reduce_scatter_fun)(void *sbuf, void *rbuf, int *rcounts,MPI_Datatype dtype,MPI_Op op,MPI_Comm comm); int (*mpi_coll_scatter_fun)(void *sendbuf, int sendcount, MPI_Datatype sendtype,void *recvbuf, int recvcount, MPI_Datatype recvtype,int root, MPI_Comm comm); int (*mpi_coll_barrier_fun)(MPI_Comm comm); void (*smpi_coll_cleanup_callback)(); int smpi_coll_tuned_alltoall_ompi2(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm) { int size = smpi_comm_size(comm); int sendsize = smpi_datatype_size(sendtype) * sendcount; if (sendsize < 200 && size > 12) { return smpi_coll_tuned_alltoall_bruck(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm); } else if (sendsize < 3000) { return smpi_coll_tuned_alltoall_basic_linear(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm); } else { return smpi_coll_tuned_alltoall_ring(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm); } } /** * Alltoall Bruck * * Openmpi calls this routine when the message size sent to each rank < 2000 bytes and size < 12 * FIXME: uh, check smpi_pmpi again, but this routine is called for > 12, not less... **/ int smpi_coll_tuned_alltoall_bruck(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm) { int system_tag = 777; int i; int count; MPI_Aint lb; MPI_Aint sendext = 0; MPI_Aint recvext = 0; MPI_Request *requests; // FIXME: check implementation int rank = smpi_comm_rank(comm); int size = smpi_comm_size(comm); XBT_DEBUG("<%d> algorithm alltoall_bruck() called.", rank); smpi_datatype_extent(sendtype, &lb, &sendext); smpi_datatype_extent(recvtype, &lb, &recvext); /* Local copy from self */ int err = smpi_datatype_copy(static_cast(sendbuf) + rank * sendcount * sendext, sendcount, sendtype, static_cast(recvbuf) + rank * recvcount * recvext, recvcount, recvtype); if (err == MPI_SUCCESS && size > 1) { /* Initiate all send/recv to/from others. */ requests = xbt_new(MPI_Request, 2 * (size - 1)); count = 0; /* Create all receives that will be posted first */ for (i = 0; i < size; ++i) { if (i != rank) { requests[count] = smpi_irecv_init(static_cast(recvbuf) + i * recvcount * recvext, recvcount, recvtype, i, system_tag, comm); count++; }else{ XBT_DEBUG("<%d> skip request creation [src = %d, recvcount = %d]", rank, i, recvcount); } } /* Now create all sends */ for (i = 0; i < size; ++i) { if (i != rank) { requests[count] = smpi_isend_init(static_cast(sendbuf) + i * sendcount * sendext, sendcount, sendtype, i, system_tag, comm); count++; }else{ XBT_DEBUG("<%d> skip request creation [dst = %d, sendcount = %d]", rank, i, sendcount); } } /* Wait for them all. */ smpi_mpi_startall(count, requests); XBT_DEBUG("<%d> wait for %d requests", rank, count); smpi_mpi_waitall(count, requests, MPI_STATUS_IGNORE); for(i = 0; i < count; i++) { if(requests[i]!=MPI_REQUEST_NULL) smpi_mpi_request_free(&requests[i]); } xbt_free(requests); } return MPI_SUCCESS; } /** * Alltoall basic_linear (STARMPI:alltoall-simple) **/ int smpi_coll_tuned_alltoall_basic_linear(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm) { int system_tag = 888; int i; int count; MPI_Aint lb = 0, sendext = 0, recvext = 0; MPI_Request *requests; /* Initialize. */ int rank = smpi_comm_rank(comm); int size = smpi_comm_size(comm); XBT_DEBUG("<%d> algorithm alltoall_basic_linear() called.", rank); smpi_datatype_extent(sendtype, &lb, &sendext); smpi_datatype_extent(recvtype, &lb, &recvext); /* simple optimization */ int err = smpi_datatype_copy(static_cast(sendbuf) + rank * sendcount * sendext, sendcount, sendtype, static_cast(recvbuf) + rank * recvcount * recvext, recvcount, recvtype); if (err == MPI_SUCCESS && size > 1) { /* Initiate all send/recv to/from others. */ requests = xbt_new(MPI_Request, 2 * (size - 1)); /* Post all receives first -- a simple optimization */ count = 0; for (i = (rank + 1) % size; i != rank; i = (i + 1) % size) { requests[count] = smpi_irecv_init(static_cast(recvbuf) + i * recvcount * recvext, recvcount, recvtype, i, system_tag, comm); count++; } /* Now post all sends in reverse order * - We would like to minimize the search time through message queue * when messages actually arrive in the order in which they were posted. * TODO: check the previous assertion */ for (i = (rank + size - 1) % size; i != rank; i = (i + size - 1) % size) { requests[count] = smpi_isend_init(static_cast(sendbuf) + i * sendcount * sendext, sendcount, sendtype, i, system_tag, comm); count++; } /* Wait for them all. */ smpi_mpi_startall(count, requests); XBT_DEBUG("<%d> wait for %d requests", rank, count); smpi_mpi_waitall(count, requests, MPI_STATUS_IGNORE); for(i = 0; i < count; i++) { if(requests[i]!=MPI_REQUEST_NULL) smpi_mpi_request_free(&requests[i]); } xbt_free(requests); } return err; } int smpi_coll_basic_alltoallv(void *sendbuf, int *sendcounts, int *senddisps, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *recvdisps, MPI_Datatype recvtype, MPI_Comm comm) { int system_tag = 889; int i; int count; MPI_Aint lb = 0; MPI_Aint sendext = 0; MPI_Aint recvext = 0; MPI_Request *requests; /* Initialize. */ int rank = smpi_comm_rank(comm); int size = smpi_comm_size(comm); XBT_DEBUG("<%d> algorithm basic_alltoallv() called.", rank); smpi_datatype_extent(sendtype, &lb, &sendext); smpi_datatype_extent(recvtype, &lb, &recvext); /* Local copy from self */ int err = smpi_datatype_copy(static_cast(sendbuf) + senddisps[rank] * sendext, sendcounts[rank], sendtype, static_cast(recvbuf) + recvdisps[rank] * recvext, recvcounts[rank], recvtype); if (err == MPI_SUCCESS && size > 1) { /* Initiate all send/recv to/from others. */ requests = xbt_new(MPI_Request, 2 * (size - 1)); count = 0; /* Create all receives that will be posted first */ for (i = 0; i < size; ++i) { if (i != rank && recvcounts[i] != 0) { requests[count] = smpi_irecv_init(static_cast(recvbuf) + recvdisps[i] * recvext, recvcounts[i], recvtype, i, system_tag, comm); count++; }else{ XBT_DEBUG("<%d> skip request creation [src = %d, recvcounts[src] = %d]", rank, i, recvcounts[i]); } } /* Now create all sends */ for (i = 0; i < size; ++i) { if (i != rank && sendcounts[i] != 0) { requests[count] = smpi_isend_init(static_cast(sendbuf) + senddisps[i] * sendext, sendcounts[i], sendtype, i, system_tag, comm); count++; }else{ XBT_DEBUG("<%d> skip request creation [dst = %d, sendcounts[dst] = %d]", rank, i, sendcounts[i]); } } /* Wait for them all. */ smpi_mpi_startall(count, requests); XBT_DEBUG("<%d> wait for %d requests", rank, count); smpi_mpi_waitall(count, requests, MPI_STATUS_IGNORE); for(i = 0; i < count; i++) { if(requests[i]!=MPI_REQUEST_NULL) smpi_mpi_request_free(&requests[i]); } xbt_free(requests); } return err; } SimGrid-3.14.159/src/smpi/smpi_utils.cpp000644 001750 001750 00000005001 13031031272 021517 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "smpi/smpi_utils.hpp" #include "xbt/sysdep.h" #include "xbt/log.h" #include "xbt/str.h" #include XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_utils, smpi, "Logging specific to SMPI (utils)"); std::vector parse_factor(const char *smpi_coef_string) { std::vector smpi_factor; /** Setup the tokenizer that parses the string **/ typedef boost::tokenizer> Tokenizer; boost::char_separator sep(";"); boost::char_separator factor_separator(":"); std::string tmp_string(smpi_coef_string); Tokenizer tokens(tmp_string, sep); /** * Iterate over patterns like A:B:C:D;E:F;G:H * These will be broken down into: * A --> B, C, D * E --> F * G --> H */ for (Tokenizer::iterator token_iter = tokens.begin(); token_iter != tokens.end(); token_iter++) { XBT_DEBUG("token : %s", token_iter->c_str()); Tokenizer factor_values(*token_iter, factor_separator); s_smpi_factor_t fact; if (factor_values.begin() == factor_values.end()) { xbt_die("Malformed radical for smpi factor: '%s'", smpi_coef_string); } unsigned int iteration = 0; for (Tokenizer::iterator factor_iter = factor_values.begin(); factor_iter != factor_values.end(); factor_iter++, iteration++) { char *errmsg; if (factor_iter == factor_values.begin()) { /* first element */ errmsg = bprintf("Invalid factor in chunk #%zu: %%s", smpi_factor.size()+1); fact.factor = xbt_str_parse_int(factor_iter->c_str(), errmsg); } else { errmsg = bprintf("Invalid factor value %d in chunk #%zu: %%s", iteration, smpi_factor.size()+1); fact.values.push_back(xbt_str_parse_double(factor_iter->c_str(), errmsg)); } xbt_free(errmsg); } smpi_factor.push_back(fact); XBT_DEBUG("smpi_factor:\t%zu : %zu values, first: %f", fact.factor, smpi_factor.size(), fact.values[0]); } std::sort(smpi_factor.begin(), smpi_factor.end(), [](const s_smpi_factor_t &pa, const s_smpi_factor_t &pb) {return (pa.factor < pb.factor);}); for (auto& fact : smpi_factor) { XBT_DEBUG("smpi_factor:\t%zu : %zu values, first: %f", fact.factor, smpi_factor.size() ,fact.values[0]); } return smpi_factor; } SimGrid-3.14.159/src/smpi/smpi_f77.cpp000644 001750 001750 00000170353 13031031272 020777 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include "smpi_mpi_dt_private.h" #include "private.h" #include "xbt.h" static xbt_dict_t comm_lookup = nullptr; static xbt_dict_t group_lookup = nullptr; static xbt_dict_t request_lookup = nullptr; static xbt_dict_t datatype_lookup = nullptr; static xbt_dict_t op_lookup = nullptr; static xbt_dict_t win_lookup = nullptr; static xbt_dict_t info_lookup = nullptr; static int running_processes = 0; #if defined(__alpha__) || defined(__sparc64__) || defined(__x86_64__) || defined(__ia64__) typedef int integer; typedef unsigned int uinteger; #else typedef long int integer; typedef unsigned long int uinteger; #endif /* Bindings for MPI special values */ struct s_smpi_common { integer _MPI_IN_PLACE; integer _MPI_BOTTOM; integer _MPI_STATUS_IGNORE; integer _MPI_STATUSES_IGNORE; } smpi_; /* Convert between Fortran and C */ #define FORT_ADDR(addr, val) \ (((void *)(addr) == (void*) &(smpi_._ ## val)) \ ? (val) : (void *)(addr)) #define FORT_BOTTOM(addr) FORT_ADDR(addr, MPI_BOTTOM) #define FORT_IN_PLACE(addr) FORT_ADDR(addr, MPI_IN_PLACE) #define FORT_STATUS_IGNORE(addr) static_cast(FORT_ADDR(addr, MPI_STATUS_IGNORE)) #define FORT_STATUSES_IGNORE(addr) static_cast(FORT_ADDR(addr, MPI_STATUSES_IGNORE)) #define KEY_SIZE (sizeof(int) * 2 + 1) static char* get_key(char* key, int id) { snprintf(key, KEY_SIZE, "%x",id); return key; } static char* get_key_id(char* key, int id) { snprintf(key, KEY_SIZE, "%x_%d",id, smpi_process_index()); return key; } static void smpi_init_fortran_types(){ if(comm_lookup == nullptr){ comm_lookup = xbt_dict_new_homogeneous(nullptr); smpi_comm_c2f(MPI_COMM_WORLD); group_lookup = xbt_dict_new_homogeneous(nullptr); request_lookup = xbt_dict_new_homogeneous(nullptr); datatype_lookup = xbt_dict_new_homogeneous(nullptr); win_lookup = xbt_dict_new_homogeneous(nullptr); info_lookup = xbt_dict_new_homogeneous(nullptr); smpi_type_c2f(MPI_BYTE);//MPI_BYTE smpi_type_c2f(MPI_CHAR);//MPI_CHARACTER #if defined(__alpha__) || defined(__sparc64__) || defined(__x86_64__) || defined(__ia64__) smpi_type_c2f(MPI_INT);//MPI_LOGICAL smpi_type_c2f(MPI_INT);//MPI_INTEGER #else smpi_type_c2f(MPI_LONG);//MPI_LOGICAL smpi_type_c2f(MPI_LONG);//MPI_INTEGER #endif smpi_type_c2f(MPI_INT8_T);//MPI_INTEGER1 smpi_type_c2f(MPI_INT16_T);//MPI_INTEGER2 smpi_type_c2f(MPI_INT32_T);//MPI_INTEGER4 smpi_type_c2f(MPI_INT64_T);//MPI_INTEGER8 smpi_type_c2f(MPI_FLOAT);//MPI_REAL smpi_type_c2f(MPI_FLOAT);//MPI_REAL4 smpi_type_c2f(MPI_DOUBLE);//MPI_REAL8 smpi_type_c2f(MPI_DOUBLE);//MPI_DOUBLE_PRECISION smpi_type_c2f(MPI_C_FLOAT_COMPLEX);//MPI_COMPLEX smpi_type_c2f(MPI_C_DOUBLE_COMPLEX);//MPI_DOUBLE_COMPLEX #if defined(__alpha__) || defined(__sparc64__) || defined(__x86_64__) || defined(__ia64__) smpi_type_c2f(MPI_2INT);//MPI_2INTEGER #else smpi_type_c2f(MPI_2LONG);//MPI_2INTEGER #endif smpi_type_c2f(MPI_UINT8_T);//MPI_LOGICAL1 smpi_type_c2f(MPI_UINT16_T);//MPI_LOGICAL2 smpi_type_c2f(MPI_UINT32_T);//MPI_LOGICAL4 smpi_type_c2f(MPI_UINT64_T);//MPI_LOGICAL8 smpi_type_c2f(MPI_2FLOAT);//MPI_2REAL smpi_type_c2f(MPI_2DOUBLE);//MPI_2DOUBLE_PRECISION smpi_type_c2f(MPI_PTR);//MPI_AINT smpi_type_c2f(MPI_UINT64_T);//MPI_OFFSET smpi_type_c2f(MPI_UINT64_T);//MPI_COUNT smpi_type_c2f(MPI_DATATYPE_NULL);//MPI_REAL16 smpi_type_c2f(MPI_DATATYPE_NULL);//MPI_COMPLEX8 smpi_type_c2f(MPI_DATATYPE_NULL);//MPI_COMPLEX16 smpi_type_c2f(MPI_DATATYPE_NULL);//MPI_COMPLEX32 smpi_type_c2f(MPI_DATATYPE_NULL);//MPI_PACKED op_lookup = xbt_dict_new_homogeneous(nullptr); smpi_op_c2f(MPI_MAX); smpi_op_c2f(MPI_MIN); smpi_op_c2f(MPI_MAXLOC); smpi_op_c2f(MPI_MINLOC); smpi_op_c2f(MPI_SUM); smpi_op_c2f(MPI_PROD); smpi_op_c2f(MPI_LAND); smpi_op_c2f(MPI_LOR); smpi_op_c2f(MPI_LXOR); smpi_op_c2f(MPI_BAND); smpi_op_c2f(MPI_BOR); smpi_op_c2f(MPI_BXOR); } } int smpi_comm_c2f(MPI_Comm comm) { static int comm_id = 0; char key[KEY_SIZE]; xbt_dict_set(comm_lookup, comm==MPI_COMM_WORLD? get_key(key, comm_id) : get_key_id(key, comm_id), comm, nullptr); comm_id++; return comm_id-1; } static void free_comm(int comm) { char key[KEY_SIZE]; xbt_dict_remove(comm_lookup, comm==0? get_key(key, comm) : get_key_id(key, comm)); } MPI_Comm smpi_comm_f2c(int comm) { smpi_init_fortran_types(); if(comm == -2) { return MPI_COMM_SELF; } else if(comm==0){ return MPI_COMM_WORLD; } else if(comm_lookup != nullptr && comm >= 0) { char key[KEY_SIZE]; MPI_Comm tmp = static_cast(xbt_dict_get_or_null(comm_lookup,get_key_id(key, comm))); return tmp != nullptr ? tmp : MPI_COMM_NULL ; } else { return MPI_COMM_NULL; } } int smpi_group_c2f(MPI_Group group) { static int group_id = 0; char key[KEY_SIZE]; xbt_dict_set(group_lookup, get_key(key, group_id), group, nullptr); group_id++; return group_id-1; } MPI_Group smpi_group_f2c(int group) { smpi_init_fortran_types(); if(group == -2) { return MPI_GROUP_EMPTY; } else if(group_lookup != nullptr && group >= 0) { char key[KEY_SIZE]; return static_cast(xbt_dict_get_or_null(group_lookup, get_key(key, group))); } else { return MPI_GROUP_NULL; } } static void free_group(int group) { char key[KEY_SIZE]; xbt_dict_remove(group_lookup, get_key(key, group)); } int smpi_request_c2f(MPI_Request req) { static int request_id = 0; char key[KEY_SIZE]; xbt_dict_set(request_lookup, get_key_id(key, request_id), req, nullptr); request_id++; return request_id-1; } MPI_Request smpi_request_f2c(int req) { smpi_init_fortran_types(); char key[KEY_SIZE]; if(req==MPI_FORTRAN_REQUEST_NULL) return MPI_REQUEST_NULL; return static_cast(xbt_dict_get(request_lookup, get_key_id(key, req))); } static void free_request(int request) { char key[KEY_SIZE]; if(request!=MPI_FORTRAN_REQUEST_NULL) xbt_dict_remove(request_lookup, get_key_id(key, request)); } int smpi_type_c2f(MPI_Datatype datatype) { static int datatype_id = 0; char key[KEY_SIZE]; xbt_dict_set(datatype_lookup, get_key(key, datatype_id), datatype, nullptr); datatype_id++; return datatype_id-1; } MPI_Datatype smpi_type_f2c(int datatype) { smpi_init_fortran_types(); char key[KEY_SIZE]; return datatype >= 0 ? static_cast(xbt_dict_get_or_null(datatype_lookup, get_key(key, datatype))): MPI_DATATYPE_NULL; } static void free_datatype(int datatype) { char key[KEY_SIZE]; xbt_dict_remove(datatype_lookup, get_key(key, datatype)); } int smpi_op_c2f(MPI_Op op) { static int op_id = 0; char key[KEY_SIZE]; xbt_dict_set(op_lookup, get_key(key, op_id), op, nullptr); op_id++; return op_id-1; } MPI_Op smpi_op_f2c(int op) { smpi_init_fortran_types(); char key[KEY_SIZE]; return op >= 0 ? static_cast(xbt_dict_get_or_null(op_lookup, get_key(key, op))): MPI_OP_NULL; } static void free_op(int op) { char key[KEY_SIZE]; xbt_dict_remove(op_lookup, get_key(key, op)); } int smpi_win_c2f(MPI_Win win) { static int win_id = 0; char key[KEY_SIZE]; xbt_dict_set(win_lookup, get_key(key, win_id), win, nullptr); win_id++; return win_id-1; } MPI_Win smpi_win_f2c(int win) { smpi_init_fortran_types(); char key[KEY_SIZE]; return win >= 0 ? static_cast(xbt_dict_get_or_null(win_lookup, get_key(key, win))) : MPI_WIN_NULL; } static void free_win(int win) { char key[KEY_SIZE]; xbt_dict_remove(win_lookup, get_key(key, win)); } int smpi_info_c2f(MPI_Info info) { static int info_id = 0; char key[KEY_SIZE]; xbt_dict_set(info_lookup, get_key(key, info_id), info, nullptr); info_id++; return info_id-1; } MPI_Info smpi_info_f2c(int info) { smpi_init_fortran_types(); char key[KEY_SIZE]; return info >= 0 ? static_cast(xbt_dict_get_or_null(info_lookup, get_key(key, info))) : MPI_INFO_NULL; } static void free_info(int info) { char key[KEY_SIZE]; xbt_dict_remove(info_lookup, get_key(key, info)); } void mpi_init_(int* ierr) { smpi_init_fortran_types(); *ierr = MPI_Init(nullptr, nullptr); running_processes++; } void mpi_finalize_(int* ierr) { *ierr = MPI_Finalize(); running_processes--; if(running_processes==0){ xbt_dict_free(&op_lookup); xbt_dict_free(&datatype_lookup); xbt_dict_free(&request_lookup); xbt_dict_free(&group_lookup); xbt_dict_free(&comm_lookup); xbt_dict_free(&win_lookup); xbt_dict_free(&info_lookup); } } void mpi_abort_(int* comm, int* errorcode, int* ierr) { *ierr = MPI_Abort(smpi_comm_f2c(*comm), *errorcode); } void mpi_comm_rank_(int* comm, int* rank, int* ierr) { *ierr = MPI_Comm_rank(smpi_comm_f2c(*comm), rank); } void mpi_comm_size_(int* comm, int* size, int* ierr) { *ierr = MPI_Comm_size(smpi_comm_f2c(*comm), size); } double mpi_wtime_() { return MPI_Wtime(); } double mpi_wtick_() { return MPI_Wtick(); } void mpi_comm_dup_(int* comm, int* newcomm, int* ierr) { MPI_Comm tmp; *ierr = MPI_Comm_dup(smpi_comm_f2c(*comm), &tmp); if(*ierr == MPI_SUCCESS) { *newcomm = smpi_comm_c2f(tmp); } } void mpi_comm_create_(int* comm, int* group, int* newcomm, int* ierr) { MPI_Comm tmp; *ierr = MPI_Comm_create(smpi_comm_f2c(*comm),smpi_group_f2c(*group), &tmp); if(*ierr == MPI_SUCCESS) { *newcomm = smpi_comm_c2f(tmp); } } void mpi_comm_free_(int* comm, int* ierr) { MPI_Comm tmp = smpi_comm_f2c(*comm); *ierr = MPI_Comm_free(&tmp); if(*ierr == MPI_SUCCESS) { free_comm(*comm); } } void mpi_comm_split_(int* comm, int* color, int* key, int* comm_out, int* ierr) { MPI_Comm tmp; *ierr = MPI_Comm_split(smpi_comm_f2c(*comm), *color, *key, &tmp); if(*ierr == MPI_SUCCESS) { *comm_out = smpi_comm_c2f(tmp); } } void mpi_group_incl_(int* group, int* n, int* ranks, int* group_out, int* ierr) { MPI_Group tmp; *ierr = MPI_Group_incl(smpi_group_f2c(*group), *n, ranks, &tmp); if(*ierr == MPI_SUCCESS) { *group_out = smpi_group_c2f(tmp); } } void mpi_comm_group_(int* comm, int* group_out, int* ierr) { MPI_Group tmp; *ierr = MPI_Comm_group(smpi_comm_f2c(*comm), &tmp); if(*ierr == MPI_SUCCESS) { *group_out = smpi_group_c2f(tmp); } } void mpi_initialized_(int* flag, int* ierr){ *ierr = MPI_Initialized(flag); } void mpi_send_init_(void *buf, int* count, int* datatype, int* dst, int* tag, int* comm, int* request, int* ierr) { MPI_Request req; buf = static_cast(FORT_BOTTOM(buf)); *ierr = MPI_Send_init(buf, *count, smpi_type_f2c(*datatype), *dst, *tag, smpi_comm_f2c(*comm), &req); if(*ierr == MPI_SUCCESS) { *request = smpi_request_c2f(req); } } void mpi_isend_(void *buf, int* count, int* datatype, int* dst, int* tag, int* comm, int* request, int* ierr) { MPI_Request req; buf = static_cast(FORT_BOTTOM(buf)); *ierr = MPI_Isend(buf, *count, smpi_type_f2c(*datatype), *dst, *tag, smpi_comm_f2c(*comm), &req); if(*ierr == MPI_SUCCESS) { *request = smpi_request_c2f(req); } } void mpi_irsend_(void *buf, int* count, int* datatype, int* dst, int* tag, int* comm, int* request, int* ierr) { MPI_Request req; buf = static_cast(FORT_BOTTOM(buf)); *ierr = MPI_Irsend(buf, *count, smpi_type_f2c(*datatype), *dst, *tag, smpi_comm_f2c(*comm), &req); if(*ierr == MPI_SUCCESS) { *request = smpi_request_c2f(req); } } void mpi_send_(void* buf, int* count, int* datatype, int* dst, int* tag, int* comm, int* ierr) { buf = static_cast(FORT_BOTTOM(buf)); *ierr = MPI_Send(buf, *count, smpi_type_f2c(*datatype), *dst, *tag, smpi_comm_f2c(*comm)); } void mpi_rsend_(void* buf, int* count, int* datatype, int* dst, int* tag, int* comm, int* ierr) { buf = static_cast(FORT_BOTTOM(buf)); *ierr = MPI_Rsend(buf, *count, smpi_type_f2c(*datatype), *dst, *tag, smpi_comm_f2c(*comm)); } void mpi_sendrecv_(void* sendbuf, int* sendcount, int* sendtype, int* dst, int* sendtag, void *recvbuf, int* recvcount, int* recvtype, int* src, int* recvtag, int* comm, MPI_Status* status, int* ierr) { sendbuf = static_cast( FORT_BOTTOM(sendbuf)); recvbuf = static_cast( FORT_BOTTOM(recvbuf)); *ierr = MPI_Sendrecv(sendbuf, *sendcount, smpi_type_f2c(*sendtype), *dst, *sendtag, recvbuf, *recvcount, smpi_type_f2c(*recvtype), *src, *recvtag, smpi_comm_f2c(*comm), FORT_STATUS_IGNORE(status)); } void mpi_recv_init_(void *buf, int* count, int* datatype, int* src, int* tag, int* comm, int* request, int* ierr) { MPI_Request req; buf = static_cast( FORT_BOTTOM(buf)); *ierr = MPI_Recv_init(buf, *count, smpi_type_f2c(*datatype), *src, *tag, smpi_comm_f2c(*comm), &req); if(*ierr == MPI_SUCCESS) { *request = smpi_request_c2f(req); } } void mpi_irecv_(void *buf, int* count, int* datatype, int* src, int* tag, int* comm, int* request, int* ierr) { MPI_Request req; buf = static_cast( FORT_BOTTOM(buf)); *ierr = MPI_Irecv(buf, *count, smpi_type_f2c(*datatype), *src, *tag, smpi_comm_f2c(*comm), &req); if(*ierr == MPI_SUCCESS) { *request = smpi_request_c2f(req); } } void mpi_recv_(void* buf, int* count, int* datatype, int* src, int* tag, int* comm, MPI_Status* status, int* ierr) { buf = static_cast( FORT_BOTTOM(buf)); *ierr = MPI_Recv(buf, *count, smpi_type_f2c(*datatype), *src, *tag, smpi_comm_f2c(*comm), status); } void mpi_start_(int* request, int* ierr) { MPI_Request req = smpi_request_f2c(*request); *ierr = MPI_Start(&req); } void mpi_startall_(int* count, int* requests, int* ierr) { MPI_Request* reqs; int i; reqs = xbt_new(MPI_Request, *count); for(i = 0; i < *count; i++) { reqs[i] = smpi_request_f2c(requests[i]); } *ierr = MPI_Startall(*count, reqs); xbt_free(reqs); } void mpi_wait_(int* request, MPI_Status* status, int* ierr) { MPI_Request req = smpi_request_f2c(*request); *ierr = MPI_Wait(&req, FORT_STATUS_IGNORE(status)); if(req==MPI_REQUEST_NULL){ free_request(*request); *request=MPI_FORTRAN_REQUEST_NULL; } } void mpi_waitany_(int* count, int* requests, int* index, MPI_Status* status, int* ierr) { MPI_Request* reqs; int i; reqs = xbt_new(MPI_Request, *count); for(i = 0; i < *count; i++) { reqs[i] = smpi_request_f2c(requests[i]); } *ierr = MPI_Waitany(*count, reqs, index, status); if(reqs[*index]==MPI_REQUEST_NULL){ free_request(requests[*index]); requests[*index]=MPI_FORTRAN_REQUEST_NULL; } xbt_free(reqs); } void mpi_waitall_(int* count, int* requests, MPI_Status* status, int* ierr) { MPI_Request* reqs; int i; reqs = xbt_new(MPI_Request, *count); for(i = 0; i < *count; i++) { reqs[i] = smpi_request_f2c(requests[i]); } *ierr = MPI_Waitall(*count, reqs, FORT_STATUSES_IGNORE(status)); for(i = 0; i < *count; i++) { if(reqs[i]==MPI_REQUEST_NULL){ free_request(requests[i]); requests[i]=MPI_FORTRAN_REQUEST_NULL; } } xbt_free(reqs); } void mpi_barrier_(int* comm, int* ierr) { *ierr = MPI_Barrier(smpi_comm_f2c(*comm)); } void mpi_bcast_(void *buf, int* count, int* datatype, int* root, int* comm, int* ierr) { *ierr = MPI_Bcast(buf, *count, smpi_type_f2c(*datatype), *root, smpi_comm_f2c(*comm)); } void mpi_reduce_(void* sendbuf, void* recvbuf, int* count, int* datatype, int* op, int* root, int* comm, int* ierr) { sendbuf = static_cast( FORT_IN_PLACE(sendbuf)); sendbuf = static_cast( FORT_BOTTOM(sendbuf)); recvbuf = static_cast( FORT_BOTTOM(recvbuf)); *ierr = MPI_Reduce(sendbuf, recvbuf, *count, smpi_type_f2c(*datatype), smpi_op_f2c(*op), *root, smpi_comm_f2c(*comm)); } void mpi_allreduce_(void* sendbuf, void* recvbuf, int* count, int* datatype, int* op, int* comm, int* ierr) { sendbuf = static_cast( FORT_IN_PLACE(sendbuf)); *ierr = MPI_Allreduce(sendbuf, recvbuf, *count, smpi_type_f2c(*datatype), smpi_op_f2c(*op), smpi_comm_f2c(*comm)); } void mpi_reduce_scatter_(void* sendbuf, void* recvbuf, int* recvcounts, int* datatype, int* op, int* comm, int* ierr) { sendbuf = static_cast( FORT_IN_PLACE(sendbuf)); *ierr = MPI_Reduce_scatter(sendbuf, recvbuf, recvcounts, smpi_type_f2c(*datatype), smpi_op_f2c(*op), smpi_comm_f2c(*comm)); } void mpi_scatter_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcount, int* recvtype, int* root, int* comm, int* ierr) { recvbuf = static_cast( FORT_IN_PLACE(recvbuf)); *ierr = MPI_Scatter(sendbuf, *sendcount, smpi_type_f2c(*sendtype), recvbuf, *recvcount, smpi_type_f2c(*recvtype), *root, smpi_comm_f2c(*comm)); } void mpi_scatterv_(void* sendbuf, int* sendcounts, int* displs, int* sendtype, void* recvbuf, int* recvcount, int* recvtype, int* root, int* comm, int* ierr) { recvbuf = static_cast( FORT_IN_PLACE(recvbuf)); *ierr = MPI_Scatterv(sendbuf, sendcounts, displs, smpi_type_f2c(*sendtype), recvbuf, *recvcount, smpi_type_f2c(*recvtype), *root, smpi_comm_f2c(*comm)); } void mpi_gather_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcount, int* recvtype, int* root, int* comm, int* ierr) { sendbuf = static_cast( FORT_IN_PLACE(sendbuf)); sendbuf = static_cast( FORT_BOTTOM(sendbuf)); recvbuf = static_cast( FORT_BOTTOM(recvbuf)); *ierr = MPI_Gather(sendbuf, *sendcount, smpi_type_f2c(*sendtype), recvbuf, *recvcount, smpi_type_f2c(*recvtype), *root, smpi_comm_f2c(*comm)); } void mpi_gatherv_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcounts, int* displs, int* recvtype, int* root, int* comm, int* ierr) { sendbuf = static_cast( FORT_IN_PLACE(sendbuf)); sendbuf = static_cast( FORT_BOTTOM(sendbuf)); recvbuf = static_cast( FORT_BOTTOM(recvbuf)); *ierr = MPI_Gatherv(sendbuf, *sendcount, smpi_type_f2c(*sendtype), recvbuf, recvcounts, displs, smpi_type_f2c(*recvtype), *root, smpi_comm_f2c(*comm)); } void mpi_allgather_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcount, int* recvtype, int* comm, int* ierr) { sendbuf = static_cast( FORT_IN_PLACE(sendbuf)); *ierr = MPI_Allgather(sendbuf, *sendcount, smpi_type_f2c(*sendtype), recvbuf, *recvcount, smpi_type_f2c(*recvtype), smpi_comm_f2c(*comm)); } void mpi_allgatherv_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcounts,int* displs, int* recvtype, int* comm, int* ierr) { sendbuf = static_cast( FORT_IN_PLACE(sendbuf)); *ierr = MPI_Allgatherv(sendbuf, *sendcount, smpi_type_f2c(*sendtype), recvbuf, recvcounts, displs, smpi_type_f2c(*recvtype), smpi_comm_f2c(*comm)); } void mpi_scan_(void* sendbuf, void* recvbuf, int* count, int* datatype, int* op, int* comm, int* ierr) { *ierr = MPI_Scan(sendbuf, recvbuf, *count, smpi_type_f2c(*datatype), smpi_op_f2c(*op), smpi_comm_f2c(*comm)); } void mpi_alltoall_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcount, int* recvtype, int* comm, int* ierr) { *ierr = MPI_Alltoall(sendbuf, *sendcount, smpi_type_f2c(*sendtype), recvbuf, *recvcount, smpi_type_f2c(*recvtype), smpi_comm_f2c(*comm)); } void mpi_alltoallv_(void* sendbuf, int* sendcounts, int* senddisps, int* sendtype, void* recvbuf, int* recvcounts, int* recvdisps, int* recvtype, int* comm, int* ierr) { *ierr = MPI_Alltoallv(sendbuf, sendcounts, senddisps, smpi_type_f2c(*sendtype), recvbuf, recvcounts, recvdisps, smpi_type_f2c(*recvtype), smpi_comm_f2c(*comm)); } void mpi_test_ (int * request, int *flag, MPI_Status * status, int* ierr){ MPI_Request req = smpi_request_f2c(*request); *ierr= MPI_Test(&req, flag, FORT_STATUS_IGNORE(status)); if(req==MPI_REQUEST_NULL){ free_request(*request); *request=MPI_FORTRAN_REQUEST_NULL; } } void mpi_testall_ (int* count, int * requests, int *flag, MPI_Status * statuses, int* ierr){ int i; MPI_Request* reqs = xbt_new(MPI_Request, *count); for(i = 0; i < *count; i++) { reqs[i] = smpi_request_f2c(requests[i]); } *ierr= MPI_Testall(*count, reqs, flag, FORT_STATUSES_IGNORE(statuses)); for(i = 0; i < *count; i++) { if(reqs[i]==MPI_REQUEST_NULL){ free_request(requests[i]); requests[i]=MPI_FORTRAN_REQUEST_NULL; } } xbt_free(reqs); } void mpi_get_processor_name_(char *name, int *resultlen, int* ierr){ *ierr = MPI_Get_processor_name(name, resultlen); } void mpi_get_count_(MPI_Status * status, int* datatype, int *count, int* ierr){ *ierr = MPI_Get_count(FORT_STATUS_IGNORE(status), smpi_type_f2c(*datatype), count); } void mpi_attr_get_(int* comm, int* keyval, void* attr_value, int* flag, int* ierr ){ *ierr = MPI_Attr_get(smpi_comm_f2c(*comm), *keyval, attr_value, flag); } void mpi_type_extent_(int* datatype, MPI_Aint * extent, int* ierr){ *ierr= MPI_Type_extent(smpi_type_f2c(*datatype), extent); } void mpi_type_commit_(int* datatype, int* ierr){ MPI_Datatype tmp= smpi_type_f2c(*datatype); *ierr= MPI_Type_commit(&tmp); } void mpi_type_vector_(int* count, int* blocklen, int* stride, int* old_type, int* newtype, int* ierr){ MPI_Datatype tmp; *ierr= MPI_Type_vector(*count, *blocklen, *stride, smpi_type_f2c(*old_type), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = smpi_type_c2f(tmp); } } void mpi_type_create_vector_(int* count, int* blocklen, int* stride, int* old_type, int* newtype, int* ierr){ MPI_Datatype tmp; *ierr= MPI_Type_vector(*count, *blocklen, *stride, smpi_type_f2c(*old_type), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = smpi_type_c2f(tmp); } } void mpi_type_hvector_(int* count, int* blocklen, MPI_Aint* stride, int* old_type, int* newtype, int* ierr){ MPI_Datatype tmp; *ierr= MPI_Type_hvector (*count, *blocklen, *stride, smpi_type_f2c(*old_type), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = smpi_type_c2f(tmp); } } void mpi_type_create_hvector_(int* count, int* blocklen, MPI_Aint* stride, int* old_type, int* newtype, int* ierr){ MPI_Datatype tmp; *ierr= MPI_Type_hvector(*count, *blocklen, *stride, smpi_type_f2c(*old_type), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = smpi_type_c2f(tmp); } } void mpi_type_free_(int* datatype, int* ierr){ MPI_Datatype tmp= smpi_type_f2c(*datatype); *ierr= MPI_Type_free (&tmp); if(*ierr == MPI_SUCCESS) { free_datatype(*datatype); } } void mpi_type_ub_(int* datatype, MPI_Aint * disp, int* ierr){ *ierr= MPI_Type_ub(smpi_type_f2c(*datatype), disp); } void mpi_type_lb_(int* datatype, MPI_Aint * extent, int* ierr){ *ierr= MPI_Type_extent(smpi_type_f2c(*datatype), extent); } void mpi_type_size_(int* datatype, int *size, int* ierr) { *ierr = MPI_Type_size(smpi_type_f2c(*datatype), size); } void mpi_error_string_(int* errorcode, char* string, int* resultlen, int* ierr){ *ierr = MPI_Error_string(*errorcode, string, resultlen); } void mpi_win_fence_( int* assert, int* win, int* ierr){ *ierr = MPI_Win_fence(* assert, smpi_win_f2c(*win)); } void mpi_win_free_( int* win, int* ierr){ MPI_Win tmp = smpi_win_f2c(*win); *ierr = MPI_Win_free(&tmp); if(*ierr == MPI_SUCCESS) { free_win(*win); } } void mpi_win_create_( int *base, MPI_Aint* size, int* disp_unit, int* info, int* comm, int *win, int* ierr){ MPI_Win tmp; *ierr = MPI_Win_create( static_cast(base), *size, *disp_unit, smpi_info_f2c(*info), smpi_comm_f2c(*comm),&tmp); if(*ierr == MPI_SUCCESS) { *win = smpi_win_c2f(tmp); } } void mpi_win_post_(int* group, int assert, int* win, int* ierr){ *ierr = MPI_Win_post(smpi_group_f2c(*group), assert, smpi_win_f2c(*win)); } void mpi_win_start_(int* group, int assert, int* win, int* ierr){ *ierr = MPI_Win_start(smpi_group_f2c(*group), assert, smpi_win_f2c(*win)); } void mpi_win_complete_(int* win, int* ierr){ *ierr = MPI_Win_complete(smpi_win_f2c(*win)); } void mpi_win_wait_(int* win, int* ierr){ *ierr = MPI_Win_wait(smpi_win_f2c(*win)); } void mpi_win_set_name_ (int* win, char * name, int* ierr, int size){ //handle trailing blanks while(name[size-1]==' ') size--; while(*name==' '){//handle leading blanks size--; name++; } char* tname = xbt_new(char,size+1); strncpy(tname, name, size); tname[size]='\0'; *ierr = MPI_Win_set_name(smpi_win_f2c(*win), tname); xbt_free(tname); } void mpi_win_get_name_ (int* win, char * name, int* len, int* ierr){ *ierr = MPI_Win_get_name(smpi_win_f2c(*win),name,len); if(*len>0) name[*len]=' ';//blank padding, not \0 } void mpi_info_create_( int *info, int* ierr){ MPI_Info tmp; *ierr = MPI_Info_create(&tmp); if(*ierr == MPI_SUCCESS) { *info = smpi_info_c2f(tmp); } } void mpi_info_set_( int *info, char *key, char *value, int* ierr, unsigned int keylen, unsigned int valuelen){ //handle trailing blanks while(key[keylen-1]==' ') keylen--; while(*key==' '){//handle leading blanks keylen--; key++; } char* tkey = xbt_new(char,keylen+1); strncpy(tkey, key, keylen); tkey[keylen]='\0'; while(value[valuelen-1]==' ') valuelen--; while(*value==' '){//handle leading blanks valuelen--; value++; } char* tvalue = xbt_new(char,valuelen+1); strncpy(tvalue, value, valuelen); tvalue[valuelen]='\0'; *ierr = MPI_Info_set( smpi_info_f2c(*info), tkey, tvalue); xbt_free(tkey); xbt_free(tvalue); } void mpi_info_get_ (int* info,char *key,int* valuelen, char *value, int *flag, int* ierr, unsigned int keylen ){ while(key[keylen-1]==' ') keylen--; while(*key==' '){//handle leading blanks keylen--; key++; } char* tkey = xbt_new(char,keylen+1); strncpy(tkey, key, keylen); tkey[keylen]='\0'; *ierr = MPI_Info_get(smpi_info_f2c(*info),tkey,*valuelen, value, flag); xbt_free(tkey); if(*flag!=0){ int replace=0; int i=0; for (i=0; i<*valuelen; i++){ if(value[i]=='\0') replace=1; if(replace) value[i]=' '; } } } void mpi_info_free_(int* info, int* ierr){ MPI_Info tmp = smpi_info_f2c(*info); *ierr = MPI_Info_free(&tmp); if(*ierr == MPI_SUCCESS) { free_info(*info); } } void mpi_get_( int *origin_addr, int* origin_count, int* origin_datatype, int *target_rank, MPI_Aint* target_disp, int *target_count, int* tarsmpi_type_f2c, int* win, int* ierr){ *ierr = MPI_Get( static_cast(origin_addr),*origin_count, smpi_type_f2c(*origin_datatype),*target_rank, *target_disp, *target_count,smpi_type_f2c(*tarsmpi_type_f2c), smpi_win_f2c(*win)); } void mpi_accumulate_( int *origin_addr, int* origin_count, int* origin_datatype, int *target_rank, MPI_Aint* target_disp, int *target_count, int* tarsmpi_type_f2c, int* op, int* win, int* ierr){ *ierr = MPI_Accumulate( static_cast(origin_addr),*origin_count, smpi_type_f2c(*origin_datatype),*target_rank, *target_disp, *target_count,smpi_type_f2c(*tarsmpi_type_f2c), smpi_op_f2c(*op), smpi_win_f2c(*win)); } void mpi_put_( int *origin_addr, int* origin_count, int* origin_datatype, int *target_rank, MPI_Aint* target_disp, int *target_count, int* tarsmpi_type_f2c, int* win, int* ierr){ *ierr = MPI_Put( static_cast(origin_addr),*origin_count, smpi_type_f2c(*origin_datatype),*target_rank, *target_disp, *target_count,smpi_type_f2c(*tarsmpi_type_f2c), smpi_win_f2c(*win)); } //following are automatically generated, and have to be checked void mpi_finalized_ (int * flag, int* ierr){ *ierr = MPI_Finalized(flag); } void mpi_init_thread_ (int* required, int *provided, int* ierr){ smpi_init_fortran_types(); *ierr = MPI_Init_thread(nullptr, nullptr,*required, provided); running_processes++; } void mpi_query_thread_ (int *provided, int* ierr){ *ierr = MPI_Query_thread(provided); } void mpi_is_thread_main_ (int *flag, int* ierr){ *ierr = MPI_Is_thread_main(flag); } void mpi_address_ (void *location, MPI_Aint * address, int* ierr){ *ierr = MPI_Address(location, address); } void mpi_get_address_ (void *location, MPI_Aint * address, int* ierr){ *ierr = MPI_Get_address(location, address); } void mpi_type_dup_ (int* datatype, int* newdatatype, int* ierr){ MPI_Datatype tmp; *ierr = MPI_Type_dup(smpi_type_f2c(*datatype), &tmp); if(*ierr == MPI_SUCCESS) { *newdatatype = smpi_type_c2f(tmp); } } void mpi_type_set_name_ (int* datatype, char * name, int* ierr, int size){ char* tname = xbt_new(char, size+1); strncpy(tname, name, size); tname[size]='\0'; *ierr = MPI_Type_set_name(smpi_type_f2c(*datatype), tname); xbt_free(tname); } void mpi_type_get_name_ (int* datatype, char * name, int* len, int* ierr){ *ierr = MPI_Type_get_name(smpi_type_f2c(*datatype),name,len); if(*len>0) name[*len]=' '; } void mpi_type_get_attr_ (int* type, int* type_keyval, void *attribute_val, int* flag, int* ierr){ *ierr = MPI_Type_get_attr ( smpi_type_f2c(*type), *type_keyval, attribute_val,flag); } void mpi_type_set_attr_ (int* type, int* type_keyval, void *attribute_val, int* ierr){ *ierr = MPI_Type_set_attr ( smpi_type_f2c(*type), *type_keyval, attribute_val); } void mpi_type_delete_attr_ (int* type, int* type_keyval, int* ierr){ *ierr = MPI_Type_delete_attr ( smpi_type_f2c(*type), *type_keyval); } void mpi_type_create_keyval_ (void* copy_fn, void* delete_fn, int* keyval, void* extra_state, int* ierr){ *ierr = MPI_Type_create_keyval(reinterpret_cast(copy_fn), reinterpret_cast(delete_fn), keyval, extra_state) ; } void mpi_type_free_keyval_ (int* keyval, int* ierr) { *ierr = MPI_Type_free_keyval( keyval); } void mpi_pcontrol_ (int* level , int* ierr){ *ierr = MPI_Pcontrol(*static_cast(level)); } void mpi_type_get_extent_ (int* datatype, MPI_Aint * lb, MPI_Aint * extent, int* ierr){ *ierr = MPI_Type_get_extent(smpi_type_f2c(*datatype), lb, extent); } void mpi_type_get_true_extent_ (int* datatype, MPI_Aint * lb, MPI_Aint * extent, int* ierr){ *ierr = MPI_Type_get_true_extent(smpi_type_f2c(*datatype), lb, extent); } void mpi_op_create_ (void * function, int* commute, int* op, int* ierr){ MPI_Op tmp; *ierr = MPI_Op_create(reinterpret_cast(function),*commute, &tmp); if(*ierr == MPI_SUCCESS) { *op = smpi_op_c2f(tmp); } } void mpi_op_free_ (int* op, int* ierr){ MPI_Op tmp=smpi_op_f2c(*op); *ierr = MPI_Op_free(& tmp); if(*ierr == MPI_SUCCESS) { free_op(*op); } } void mpi_group_free_ (int* group, int* ierr){ MPI_Group tmp=smpi_group_f2c(*group); *ierr = MPI_Group_free(&tmp); if(*ierr == MPI_SUCCESS) { free_group(*group); } } void mpi_group_size_ (int* group, int *size, int* ierr){ *ierr = MPI_Group_size(smpi_group_f2c(*group), size); } void mpi_group_rank_ (int* group, int *rank, int* ierr){ *ierr = MPI_Group_rank(smpi_group_f2c(*group), rank); } void mpi_group_translate_ranks_ (int* group1, int* n, int *ranks1, int* group2, int *ranks2, int* ierr) { *ierr = MPI_Group_translate_ranks(smpi_group_f2c(*group1), *n, ranks1, smpi_group_f2c(*group2), ranks2); } void mpi_group_compare_ (int* group1, int* group2, int *result, int* ierr){ *ierr = MPI_Group_compare(smpi_group_f2c(*group1), smpi_group_f2c(*group2), result); } void mpi_group_union_ (int* group1, int* group2, int* newgroup, int* ierr){ MPI_Group tmp; *ierr = MPI_Group_union(smpi_group_f2c(*group1), smpi_group_f2c(*group2), &tmp); if(*ierr == MPI_SUCCESS) { *newgroup = smpi_group_c2f(tmp); } } void mpi_group_intersection_ (int* group1, int* group2, int* newgroup, int* ierr){ MPI_Group tmp; *ierr = MPI_Group_intersection(smpi_group_f2c(*group1), smpi_group_f2c(*group2), &tmp); if(*ierr == MPI_SUCCESS) { *newgroup = smpi_group_c2f(tmp); } } void mpi_group_difference_ (int* group1, int* group2, int* newgroup, int* ierr){ MPI_Group tmp; *ierr = MPI_Group_difference(smpi_group_f2c(*group1), smpi_group_f2c(*group2), &tmp); if(*ierr == MPI_SUCCESS) { *newgroup = smpi_group_c2f(tmp); } } void mpi_group_excl_ (int* group, int* n, int *ranks, int* newgroup, int* ierr){ MPI_Group tmp; *ierr = MPI_Group_excl(smpi_group_f2c(*group), *n, ranks, &tmp); if(*ierr == MPI_SUCCESS) { *newgroup = smpi_group_c2f(tmp); } } void mpi_group_range_incl_ (int* group, int* n, int ranges[][3], int* newgroup, int* ierr) { MPI_Group tmp; *ierr = MPI_Group_range_incl(smpi_group_f2c(*group), *n, ranges, &tmp); if(*ierr == MPI_SUCCESS) { *newgroup = smpi_group_c2f(tmp); } } void mpi_group_range_excl_ (int* group, int* n, int ranges[][3], int* newgroup, int* ierr) { MPI_Group tmp; *ierr = MPI_Group_range_excl(smpi_group_f2c(*group), *n, ranges, &tmp); if(*ierr == MPI_SUCCESS) { *newgroup = smpi_group_c2f(tmp); } } void mpi_comm_get_attr_ (int* comm, int* comm_keyval, void *attribute_val, int *flag, int* ierr){ *ierr = MPI_Comm_get_attr (smpi_comm_f2c(*comm), *comm_keyval, attribute_val, flag); } void mpi_comm_set_attr_ (int* comm, int* comm_keyval, void *attribute_val, int* ierr){ *ierr = MPI_Comm_set_attr ( smpi_comm_f2c(*comm), *comm_keyval, attribute_val); } void mpi_comm_delete_attr_ (int* comm, int* comm_keyval, int* ierr){ *ierr = MPI_Comm_delete_attr (smpi_comm_f2c(*comm), *comm_keyval); } void mpi_comm_create_keyval_ (void* copy_fn, void* delete_fn, int* keyval, void* extra_state, int* ierr){ *ierr = MPI_Comm_create_keyval(reinterpret_cast(copy_fn), reinterpret_cast(delete_fn), keyval, extra_state) ; } void mpi_comm_free_keyval_ (int* keyval, int* ierr) { *ierr = MPI_Comm_free_keyval( keyval); } void mpi_comm_get_name_ (int* comm, char* name, int* len, int* ierr){ *ierr = MPI_Comm_get_name(smpi_comm_f2c(*comm), name, len); if(*len>0) name[*len]=' '; } void mpi_comm_compare_ (int* comm1, int* comm2, int *result, int* ierr){ *ierr = MPI_Comm_compare(smpi_comm_f2c(*comm1), smpi_comm_f2c(*comm2), result); } void mpi_comm_disconnect_ (int* comm, int* ierr){ MPI_Comm tmp=smpi_comm_f2c(*comm); *ierr = MPI_Comm_disconnect(&tmp); if(*ierr == MPI_SUCCESS) { free_comm(*comm); } } void mpi_request_free_ (int* request, int* ierr){ MPI_Request tmp=smpi_request_f2c(*request); *ierr = MPI_Request_free(&tmp); if(*ierr == MPI_SUCCESS) { free_request(*request); } } void mpi_sendrecv_replace_ (void *buf, int* count, int* datatype, int* dst, int* sendtag, int* src, int* recvtag, int* comm, MPI_Status* status, int* ierr) { *ierr = MPI_Sendrecv_replace(buf, *count, smpi_type_f2c(*datatype), *dst, *sendtag, *src, *recvtag, smpi_comm_f2c(*comm), FORT_STATUS_IGNORE(status)); } void mpi_testany_ (int* count, int* requests, int *index, int *flag, MPI_Status* status, int* ierr) { MPI_Request* reqs; int i; reqs = xbt_new(MPI_Request, *count); for(i = 0; i < *count; i++) { reqs[i] = smpi_request_f2c(requests[i]); } *ierr = MPI_Testany(*count, reqs, index, flag, FORT_STATUS_IGNORE(status)); if(*index!=MPI_UNDEFINED && reqs[*index]==MPI_REQUEST_NULL){ free_request(requests[*index]); requests[*index]=MPI_FORTRAN_REQUEST_NULL; } xbt_free(reqs); } void mpi_waitsome_ (int* incount, int* requests, int *outcount, int *indices, MPI_Status* status, int* ierr) { MPI_Request* reqs; int i; reqs = xbt_new(MPI_Request, *incount); for(i = 0; i < *incount; i++) { reqs[i] = smpi_request_f2c(requests[i]); } *ierr = MPI_Waitsome(*incount, reqs, outcount, indices, status); for(i=0;i<*outcount;i++){ if(reqs[indices[i]]==MPI_REQUEST_NULL){ free_request(requests[indices[i]]); requests[indices[i]]=MPI_FORTRAN_REQUEST_NULL; } } xbt_free(reqs); } void mpi_reduce_local_ (void *inbuf, void *inoutbuf, int* count, int* datatype, int* op, int* ierr){ *ierr = MPI_Reduce_local(inbuf, inoutbuf, *count, smpi_type_f2c(*datatype), smpi_op_f2c(*op)); } void mpi_reduce_scatter_block_ (void *sendbuf, void *recvbuf, int* recvcount, int* datatype, int* op, int* comm, int* ierr) { sendbuf = static_cast( FORT_IN_PLACE(sendbuf)); *ierr = MPI_Reduce_scatter_block(sendbuf, recvbuf, *recvcount, smpi_type_f2c(*datatype), smpi_op_f2c(*op), smpi_comm_f2c(*comm)); } void mpi_pack_size_ (int* incount, int* datatype, int* comm, int* size, int* ierr) { *ierr = MPI_Pack_size(*incount, smpi_type_f2c(*datatype), smpi_comm_f2c(*comm), size); } void mpi_cart_coords_ (int* comm, int* rank, int* maxdims, int* coords, int* ierr) { *ierr = MPI_Cart_coords(smpi_comm_f2c(*comm), *rank, *maxdims, coords); } void mpi_cart_create_ (int* comm_old, int* ndims, int* dims, int* periods, int* reorder, int* comm_cart, int* ierr) { MPI_Comm tmp; *ierr = MPI_Cart_create(smpi_comm_f2c(*comm_old), *ndims, dims, periods, *reorder, &tmp); if(*ierr == MPI_SUCCESS) { *comm_cart = smpi_comm_c2f(tmp); } } void mpi_cart_get_ (int* comm, int* maxdims, int* dims, int* periods, int* coords, int* ierr) { *ierr = MPI_Cart_get(smpi_comm_f2c(*comm), *maxdims, dims, periods, coords); } void mpi_cart_map_ (int* comm_old, int* ndims, int* dims, int* periods, int* newrank, int* ierr) { *ierr = MPI_Cart_map(smpi_comm_f2c(*comm_old), *ndims, dims, periods, newrank); } void mpi_cart_rank_ (int* comm, int* coords, int* rank, int* ierr) { *ierr = MPI_Cart_rank(smpi_comm_f2c(*comm), coords, rank); } void mpi_cart_shift_ (int* comm, int* direction, int* displ, int* source, int* dest, int* ierr) { *ierr = MPI_Cart_shift(smpi_comm_f2c(*comm), *direction, *displ, source, dest); } void mpi_cart_sub_ (int* comm, int* remain_dims, int* comm_new, int* ierr) { MPI_Comm tmp; *ierr = MPI_Cart_sub(smpi_comm_f2c(*comm), remain_dims, &tmp); if(*ierr == MPI_SUCCESS) { *comm_new = smpi_comm_c2f(tmp); } } void mpi_cartdim_get_ (int* comm, int* ndims, int* ierr) { *ierr = MPI_Cartdim_get(smpi_comm_f2c(*comm), ndims); } void mpi_graph_create_ (int* comm_old, int* nnodes, int* index, int* edges, int* reorder, int* comm_graph, int* ierr) { MPI_Comm tmp; *ierr = MPI_Graph_create(smpi_comm_f2c(*comm_old), *nnodes, index, edges, *reorder, &tmp); if(*ierr == MPI_SUCCESS) { *comm_graph = smpi_comm_c2f(tmp); } } void mpi_graph_get_ (int* comm, int* maxindex, int* maxedges, int* index, int* edges, int* ierr) { *ierr = MPI_Graph_get(smpi_comm_f2c(*comm), *maxindex, *maxedges, index, edges); } void mpi_graph_map_ (int* comm_old, int* nnodes, int* index, int* edges, int* newrank, int* ierr) { *ierr = MPI_Graph_map(smpi_comm_f2c(*comm_old), *nnodes, index, edges, newrank); } void mpi_graph_neighbors_ (int* comm, int* rank, int* maxneighbors, int* neighbors, int* ierr) { *ierr = MPI_Graph_neighbors(smpi_comm_f2c(*comm), *rank, *maxneighbors, neighbors); } void mpi_graph_neighbors_count_ (int* comm, int* rank, int* nneighbors, int* ierr) { *ierr = MPI_Graph_neighbors_count(smpi_comm_f2c(*comm), *rank, nneighbors); } void mpi_graphdims_get_ (int* comm, int* nnodes, int* nedges, int* ierr) { *ierr = MPI_Graphdims_get(smpi_comm_f2c(*comm), nnodes, nedges); } void mpi_topo_test_ (int* comm, int* top_type, int* ierr) { *ierr = MPI_Topo_test(smpi_comm_f2c(*comm), top_type); } void mpi_error_class_ (int* errorcode, int* errorclass, int* ierr) { *ierr = MPI_Error_class(*errorcode, errorclass); } void mpi_errhandler_create_ (void* function, void* errhandler, int* ierr) { *ierr = MPI_Errhandler_create(reinterpret_cast(function), static_cast(errhandler)); } void mpi_errhandler_free_ (void* errhandler, int* ierr) { *ierr = MPI_Errhandler_free(static_cast(errhandler)); } void mpi_errhandler_get_ (int* comm, void* errhandler, int* ierr) { *ierr = MPI_Errhandler_get(smpi_comm_f2c(*comm), static_cast(errhandler)); } void mpi_errhandler_set_ (int* comm, void* errhandler, int* ierr) { *ierr = MPI_Errhandler_set(smpi_comm_f2c(*comm), *static_cast(errhandler)); } void mpi_comm_set_errhandler_ (int* comm, void* errhandler, int* ierr) { *ierr = MPI_Errhandler_set(smpi_comm_f2c(*comm), *static_cast(errhandler)); } void mpi_comm_get_errhandler_ (int* comm, void* errhandler, int* ierr) { *ierr = MPI_Errhandler_set(smpi_comm_f2c(*comm), static_cast(errhandler)); } void mpi_type_contiguous_ (int* count, int* old_type, int* newtype, int* ierr) { MPI_Datatype tmp; *ierr = MPI_Type_contiguous(*count, smpi_type_f2c(*old_type), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = smpi_type_c2f(tmp); } } void mpi_cancel_ (int* request, int* ierr) { MPI_Request tmp=smpi_request_f2c(*request); *ierr = MPI_Cancel(&tmp); } void mpi_buffer_attach_ (void* buffer, int* size, int* ierr) { *ierr = MPI_Buffer_attach(buffer, *size); } void mpi_buffer_detach_ (void* buffer, int* size, int* ierr) { *ierr = MPI_Buffer_detach(buffer, size); } void mpi_testsome_ (int* incount, int* requests, int* outcount, int* indices, MPI_Status* statuses, int* ierr) { MPI_Request* reqs; int i; reqs = xbt_new(MPI_Request, *incount); for(i = 0; i < *incount; i++) { reqs[i] = smpi_request_f2c(requests[i]); indices[i]=0; } *ierr = MPI_Testsome(*incount, reqs, outcount, indices, FORT_STATUSES_IGNORE(statuses)); for(i=0;i<*incount;i++){ if(indices[i] && reqs[indices[i]]==MPI_REQUEST_NULL){ free_request(requests[indices[i]]); requests[indices[i]]=MPI_FORTRAN_REQUEST_NULL; } } xbt_free(reqs); } void mpi_comm_test_inter_ (int* comm, int* flag, int* ierr) { *ierr = MPI_Comm_test_inter(smpi_comm_f2c(*comm), flag); } void mpi_unpack_ (void* inbuf, int* insize, int* position, void* outbuf, int* outcount, int* type, int* comm, int* ierr) { *ierr = MPI_Unpack(inbuf, *insize, position, outbuf, *outcount, smpi_type_f2c(*type), smpi_comm_f2c(*comm)); } void mpi_pack_external_size_ (char *datarep, int* incount, int* datatype, MPI_Aint *size, int* ierr){ *ierr = MPI_Pack_external_size(datarep, *incount, smpi_type_f2c(*datatype), size); } void mpi_pack_external_ (char *datarep, void *inbuf, int* incount, int* datatype, void *outbuf, MPI_Aint* outcount, MPI_Aint *position, int* ierr){ *ierr = MPI_Pack_external(datarep, inbuf, *incount, smpi_type_f2c(*datatype), outbuf, *outcount, position); } void mpi_unpack_external_ ( char *datarep, void *inbuf, MPI_Aint* insize, MPI_Aint *position, void *outbuf, int* outcount, int* datatype, int* ierr){ *ierr = MPI_Unpack_external( datarep, inbuf, *insize, position, outbuf, *outcount, smpi_type_f2c(*datatype)); } void mpi_type_hindexed_ (int* count, int* blocklens, MPI_Aint* indices, int* old_type, int* newtype, int* ierr) { MPI_Datatype tmp; *ierr = MPI_Type_hindexed(*count, blocklens, indices, smpi_type_f2c(*old_type), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = smpi_type_c2f(tmp); } } void mpi_type_create_hindexed_(int* count, int* blocklens, MPI_Aint* indices, int* old_type, int* newtype, int* ierr){ MPI_Datatype tmp; *ierr = MPI_Type_create_hindexed(*count, blocklens, indices, smpi_type_f2c(*old_type), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = smpi_type_c2f(tmp); } } void mpi_type_create_hindexed_block_ (int* count, int* blocklength, MPI_Aint* indices, int* old_type, int* newtype, int* ierr) { MPI_Datatype tmp; *ierr = MPI_Type_create_hindexed_block(*count, *blocklength, indices, smpi_type_f2c(*old_type), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = smpi_type_c2f(tmp); } } void mpi_type_indexed_ (int* count, int* blocklens, int* indices, int* old_type, int* newtype, int* ierr) { MPI_Datatype tmp; *ierr = MPI_Type_indexed(*count, blocklens, indices, smpi_type_f2c(*old_type), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = smpi_type_c2f(tmp); } } void mpi_type_create_indexed_block_ (int* count, int* blocklength, int* indices, int* old_type, int*newtype, int* ierr){ MPI_Datatype tmp; *ierr = MPI_Type_create_indexed_block(*count, *blocklength, indices, smpi_type_f2c(*old_type), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = smpi_type_c2f(tmp); } } void mpi_type_struct_ (int* count, int* blocklens, MPI_Aint* indices, int* old_types, int* newtype, int* ierr) { MPI_Datatype tmp; int i=0; MPI_Datatype* types = static_cast(xbt_malloc(*count*sizeof(MPI_Datatype))); for(i=0; i< *count; i++){ types[i] = smpi_type_f2c(old_types[i]); } *ierr = MPI_Type_struct(*count, blocklens, indices, types, &tmp); if(*ierr == MPI_SUCCESS) { *newtype = smpi_type_c2f(tmp); } xbt_free(types); } void mpi_type_create_struct_(int* count, int* blocklens, MPI_Aint* indices, int* old_types, int* newtype, int* ierr){ MPI_Datatype tmp; int i=0; MPI_Datatype* types = static_cast(xbt_malloc(*count*sizeof(MPI_Datatype))); for(i=0; i< *count; i++){ types[i] = smpi_type_f2c(old_types[i]); } *ierr = MPI_Type_create_struct(*count, blocklens, indices, types, &tmp); if(*ierr == MPI_SUCCESS) { *newtype = smpi_type_c2f(tmp); } xbt_free(types); } void mpi_ssend_ (void* buf, int* count, int* datatype, int* dest, int* tag, int* comm, int* ierr) { *ierr = MPI_Ssend(buf, *count, smpi_type_f2c(*datatype), *dest, *tag, smpi_comm_f2c(*comm)); } void mpi_ssend_init_ (void* buf, int* count, int* datatype, int* dest, int* tag, int* comm, int* request, int* ierr) { MPI_Request tmp; *ierr = MPI_Ssend_init(buf, *count, smpi_type_f2c(*datatype), *dest, *tag, smpi_comm_f2c(*comm), &tmp); if(*ierr == MPI_SUCCESS) { *request = smpi_request_c2f(tmp); } } void mpi_intercomm_create_ (int* local_comm, int *local_leader, int* peer_comm, int* remote_leader, int* tag, int* comm_out, int* ierr) { MPI_Comm tmp; *ierr = MPI_Intercomm_create(smpi_comm_f2c(*local_comm), *local_leader,smpi_comm_f2c(*peer_comm), *remote_leader, *tag, &tmp); if(*ierr == MPI_SUCCESS) { *comm_out = smpi_comm_c2f(tmp); } } void mpi_intercomm_merge_ (int* comm, int* high, int* comm_out, int* ierr) { MPI_Comm tmp; *ierr = MPI_Intercomm_merge(smpi_comm_f2c(*comm), *high, &tmp); if(*ierr == MPI_SUCCESS) { *comm_out = smpi_comm_c2f(tmp); } } void mpi_bsend_ (void* buf, int* count, int* datatype, int *dest, int* tag, int* comm, int* ierr) { *ierr = MPI_Bsend(buf, *count, smpi_type_f2c(*datatype), *dest, *tag, smpi_comm_f2c(*comm)); } void mpi_bsend_init_ (void* buf, int* count, int* datatype, int *dest, int* tag, int* comm, int* request, int* ierr) { MPI_Request tmp; *ierr = MPI_Bsend_init(buf, *count, smpi_type_f2c(*datatype), *dest, *tag, smpi_comm_f2c(*comm), &tmp); if(*ierr == MPI_SUCCESS) { *request = smpi_request_c2f(tmp); } } void mpi_ibsend_ (void* buf, int* count, int* datatype, int *dest, int* tag, int* comm, int* request, int* ierr) { MPI_Request tmp; *ierr = MPI_Ibsend(buf, *count, smpi_type_f2c(*datatype), *dest, *tag, smpi_comm_f2c(*comm), &tmp); if(*ierr == MPI_SUCCESS) { *request = smpi_request_c2f(tmp); } } void mpi_comm_remote_group_ (int* comm, int* group, int* ierr) { MPI_Group tmp; *ierr = MPI_Comm_remote_group(smpi_comm_f2c(*comm), &tmp); if(*ierr == MPI_SUCCESS) { *group = smpi_group_c2f(tmp); } } void mpi_comm_remote_size_ (int* comm, int* size, int* ierr) { *ierr = MPI_Comm_remote_size(smpi_comm_f2c(*comm), size); } void mpi_issend_ (void* buf, int* count, int* datatype, int *dest, int* tag, int* comm, int* request, int* ierr) { MPI_Request tmp; *ierr = MPI_Issend(buf, *count, smpi_type_f2c(*datatype), *dest, *tag, smpi_comm_f2c(*comm), &tmp); if(*ierr == MPI_SUCCESS) { *request = smpi_request_c2f(tmp); } } void mpi_probe_ (int* source, int* tag, int* comm, MPI_Status* status, int* ierr) { *ierr = MPI_Probe(*source, *tag, smpi_comm_f2c(*comm), FORT_STATUS_IGNORE(status)); } void mpi_attr_delete_ (int* comm, int* keyval, int* ierr) { *ierr = MPI_Attr_delete(smpi_comm_f2c(*comm), *keyval); } void mpi_attr_put_ (int* comm, int* keyval, void* attr_value, int* ierr) { *ierr = MPI_Attr_put(smpi_comm_f2c(*comm), *keyval, attr_value); } void mpi_rsend_init_ (void* buf, int* count, int* datatype, int *dest, int* tag, int* comm, int* request, int* ierr) { MPI_Request tmp; *ierr = MPI_Rsend_init(buf, *count, smpi_type_f2c(*datatype), *dest, *tag, smpi_comm_f2c(*comm), &tmp); if(*ierr == MPI_SUCCESS) { *request = smpi_request_c2f(tmp); } } void mpi_keyval_create_ (void* copy_fn, void* delete_fn, int* keyval, void* extra_state, int* ierr) { *ierr = MPI_Keyval_create(reinterpret_cast(copy_fn),reinterpret_cast(delete_fn), keyval, extra_state); } void mpi_keyval_free_ (int* keyval, int* ierr) { *ierr = MPI_Keyval_free(keyval); } void mpi_test_cancelled_ (MPI_Status* status, int* flag, int* ierr) { *ierr = MPI_Test_cancelled(status, flag); } void mpi_pack_ (void* inbuf, int* incount, int* type, void* outbuf, int* outcount, int* position, int* comm, int* ierr) { *ierr = MPI_Pack(inbuf, *incount, smpi_type_f2c(*type), outbuf, *outcount, position, smpi_comm_f2c(*comm)); } void mpi_get_elements_ (MPI_Status* status, int* datatype, int* elements, int* ierr) { *ierr = MPI_Get_elements(status, smpi_type_f2c(*datatype), elements); } void mpi_dims_create_ (int* nnodes, int* ndims, int* dims, int* ierr) { *ierr = MPI_Dims_create(*nnodes, *ndims, dims); } void mpi_iprobe_ (int* source, int* tag, int* comm, int* flag, MPI_Status* status, int* ierr) { *ierr = MPI_Iprobe(*source, *tag, smpi_comm_f2c(*comm), flag, status); } void mpi_type_get_envelope_ ( int* datatype, int *num_integers, int *num_addresses, int *num_datatypes, int *combiner, int* ierr){ *ierr = MPI_Type_get_envelope( smpi_type_f2c(*datatype), num_integers, num_addresses, num_datatypes, combiner); } void mpi_type_get_contents_ (int* datatype, int* max_integers, int* max_addresses, int* max_datatypes, int* array_of_integers, MPI_Aint* array_of_addresses, int* array_of_datatypes, int* ierr){ *ierr = MPI_Type_get_contents(smpi_type_f2c(*datatype), *max_integers, *max_addresses,*max_datatypes, array_of_integers, array_of_addresses, reinterpret_cast(array_of_datatypes)); } void mpi_type_create_darray_ (int* size, int* rank, int* ndims, int* array_of_gsizes, int* array_of_distribs, int* array_of_dargs, int* array_of_psizes, int* order, int* oldtype, int*newtype, int* ierr) { MPI_Datatype tmp; *ierr = MPI_Type_create_darray(*size, *rank, *ndims, array_of_gsizes, array_of_distribs, array_of_dargs, array_of_psizes, *order, smpi_type_f2c(*oldtype), &tmp) ; if(*ierr == MPI_SUCCESS) { *newtype = smpi_type_c2f(tmp); } } void mpi_type_create_resized_ (int* oldtype,MPI_Aint* lb, MPI_Aint* extent, int*newtype, int* ierr){ MPI_Datatype tmp; *ierr = MPI_Type_create_resized(smpi_type_f2c(*oldtype),*lb, *extent, &tmp); if(*ierr == MPI_SUCCESS) { *newtype = smpi_type_c2f(tmp); } } void mpi_type_create_subarray_ (int* ndims,int *array_of_sizes, int *array_of_subsizes, int *array_of_starts, int* order, int* oldtype, int*newtype, int* ierr){ MPI_Datatype tmp; *ierr = MPI_Type_create_subarray(*ndims,array_of_sizes, array_of_subsizes, array_of_starts, *order, smpi_type_f2c(*oldtype), &tmp); if(*ierr == MPI_SUCCESS) { *newtype = smpi_type_c2f(tmp); } } void mpi_type_match_size_ (int* typeclass,int* size,int* datatype, int* ierr){ MPI_Datatype tmp; *ierr = MPI_Type_match_size(*typeclass,*size,&tmp); if(*ierr == MPI_SUCCESS) { *datatype = smpi_type_c2f(tmp); } } void mpi_alltoallw_ ( void *sendbuf, int *sendcnts, int *sdispls, int* sendtypes, void *recvbuf, int *recvcnts, int *rdispls, int* recvtypes, int* comm, int* ierr){ *ierr = MPI_Alltoallw( sendbuf, sendcnts, sdispls, reinterpret_cast(sendtypes), recvbuf, recvcnts, rdispls, reinterpret_cast(recvtypes), smpi_comm_f2c(*comm)); } void mpi_exscan_ (void *sendbuf, void *recvbuf, int* count, int* datatype, int* op, int* comm, int* ierr){ *ierr = MPI_Exscan(sendbuf, recvbuf, *count, smpi_type_f2c(*datatype), smpi_op_f2c(*op), smpi_comm_f2c(*comm)); } void mpi_comm_set_name_ (int* comm, char* name, int* ierr, int size){ char* tname = xbt_new(char, size+1); strncpy(tname, name, size); tname[size]='\0'; *ierr = MPI_Comm_set_name (smpi_comm_f2c(*comm), tname); xbt_free(tname); } void mpi_comm_dup_with_info_ (int* comm, int* info, int* newcomm, int* ierr){ MPI_Comm tmp; *ierr = MPI_Comm_dup_with_info(smpi_comm_f2c(*comm),smpi_info_f2c(*info),&tmp); if(*ierr == MPI_SUCCESS) { *newcomm = smpi_comm_c2f(tmp); } } void mpi_comm_split_type_ (int* comm, int* split_type, int* key, int* info, int* newcomm, int* ierr){ MPI_Comm tmp; *ierr = MPI_Comm_split_type(smpi_comm_f2c(*comm), *split_type, *key, smpi_info_f2c(*info), &tmp); if(*ierr == MPI_SUCCESS) { *newcomm = smpi_comm_c2f(tmp); } } void mpi_comm_set_info_ (int* comm, int* info, int* ierr){ *ierr = MPI_Comm_set_info (smpi_comm_f2c(*comm), smpi_info_f2c(*info)); } void mpi_comm_get_info_ (int* comm, int* info, int* ierr){ MPI_Info tmp; *ierr = MPI_Comm_get_info (smpi_comm_f2c(*comm), &tmp); if(*ierr==MPI_SUCCESS){ *info = smpi_info_c2f(tmp); } } void mpi_comm_create_errhandler_ ( void *function, void *errhandler, int* ierr){ *ierr = MPI_Comm_create_errhandler( reinterpret_cast(function), static_cast(errhandler)); } void mpi_add_error_class_ ( int *errorclass, int* ierr){ *ierr = MPI_Add_error_class( errorclass); } void mpi_add_error_code_ ( int* errorclass, int *errorcode, int* ierr){ *ierr = MPI_Add_error_code(*errorclass, errorcode); } void mpi_add_error_string_ ( int* errorcode, char *string, int* ierr){ *ierr = MPI_Add_error_string(*errorcode, string); } void mpi_comm_call_errhandler_ (int* comm,int* errorcode, int* ierr){ *ierr = MPI_Comm_call_errhandler(smpi_comm_f2c(*comm), *errorcode); } void mpi_info_dup_ (int* info, int* newinfo, int* ierr){ MPI_Info tmp; *ierr = MPI_Info_dup(smpi_info_f2c(*info), &tmp); if(*ierr==MPI_SUCCESS){ *newinfo= smpi_info_c2f(tmp); } } void mpi_info_get_valuelen_ ( int* info, char *key, int *valuelen, int *flag, int* ierr, unsigned int keylen){ while(key[keylen-1]==' ') keylen--; while(*key==' '){//handle leading blanks keylen--; key++; } char* tkey = xbt_new(char, keylen+1); strncpy(tkey, key, keylen); tkey[keylen]='\0'; *ierr = MPI_Info_get_valuelen( smpi_info_f2c(*info), tkey, valuelen, flag); xbt_free(tkey); } void mpi_info_delete_ (int* info, char *key, int* ierr, unsigned int keylen){ while(key[keylen-1]==' ') keylen--; while(*key==' '){//handle leading blanks keylen--; key++; } char* tkey = xbt_new(char, keylen+1); strncpy(tkey, key, keylen); tkey[keylen]='\0'; *ierr = MPI_Info_delete(smpi_info_f2c(*info), tkey); xbt_free(tkey); } void mpi_info_get_nkeys_ ( int* info, int *nkeys, int* ierr){ *ierr = MPI_Info_get_nkeys( smpi_info_f2c(*info), nkeys); } void mpi_info_get_nthkey_ ( int* info, int* n, char *key, int* ierr, unsigned int keylen){ *ierr = MPI_Info_get_nthkey( smpi_info_f2c(*info), *n, key); unsigned int i = 0; for (i=strlen(key); i(query_fn), reinterpret_cast(free_fn), reinterpret_cast(cancel_fn), extra_state, &tmp); if(*ierr == MPI_SUCCESS) { *request = smpi_request_c2f(tmp); } } void mpi_grequest_complete_ ( int* request, int* ierr){ *ierr = MPI_Grequest_complete( smpi_request_f2c(*request)); } void mpi_status_set_cancelled_ (MPI_Status* status,int* flag, int* ierr){ *ierr = MPI_Status_set_cancelled(status,*flag); } void mpi_status_set_elements_ ( MPI_Status* status, int* datatype, int* count, int* ierr){ *ierr = MPI_Status_set_elements( status, smpi_type_f2c(*datatype), *count); } void mpi_comm_connect_ ( char *port_name, int* info, int* root, int* comm, int*newcomm, int* ierr){ MPI_Comm tmp; *ierr = MPI_Comm_connect( port_name, *reinterpret_cast(info), *root, smpi_comm_f2c(*comm), &tmp); if(*ierr == MPI_SUCCESS) { *newcomm = smpi_comm_c2f(tmp); } } void mpi_publish_name_ ( char *service_name, int* info, char *port_name, int* ierr){ *ierr = MPI_Publish_name( service_name, *reinterpret_cast(info), port_name); } void mpi_unpublish_name_ ( char *service_name, int* info, char *port_name, int* ierr){ *ierr = MPI_Unpublish_name( service_name, *reinterpret_cast(info), port_name); } void mpi_lookup_name_ ( char *service_name, int* info, char *port_name, int* ierr){ *ierr = MPI_Lookup_name( service_name, *reinterpret_cast(info), port_name); } void mpi_comm_join_ ( int* fd, int* intercomm, int* ierr){ MPI_Comm tmp; *ierr = MPI_Comm_join( *fd, &tmp); if(*ierr == MPI_SUCCESS) { *intercomm = smpi_comm_c2f(tmp); } } void mpi_open_port_ ( int* info, char *port_name, int* ierr){ *ierr = MPI_Open_port( *reinterpret_cast(info),port_name); } void mpi_close_port_ ( char *port_name, int* ierr){ *ierr = MPI_Close_port( port_name); } void mpi_comm_accept_ ( char *port_name, int* info, int* root, int* comm, int*newcomm, int* ierr){ MPI_Comm tmp; *ierr = MPI_Comm_accept( port_name, *reinterpret_cast(info), *root, smpi_comm_f2c(*comm), &tmp); if(*ierr == MPI_SUCCESS) { *newcomm = smpi_comm_c2f(tmp); } } void mpi_comm_spawn_ ( char *command, char *argv, int* maxprocs, int* info, int* root, int* comm, int* intercomm, int* array_of_errcodes, int* ierr){ MPI_Comm tmp; *ierr = MPI_Comm_spawn( command, nullptr, *maxprocs, *reinterpret_cast(info), *root, smpi_comm_f2c(*comm), &tmp, array_of_errcodes); if(*ierr == MPI_SUCCESS) { *intercomm = smpi_comm_c2f(tmp); } } void mpi_comm_spawn_multiple_ ( int* count, char *array_of_commands, char** array_of_argv, int* array_of_maxprocs, int* array_of_info, int* root, int* comm, int* intercomm, int* array_of_errcodes, int* ierr){ MPI_Comm tmp; *ierr = MPI_Comm_spawn_multiple(* count, &array_of_commands, &array_of_argv, array_of_maxprocs, reinterpret_cast(array_of_info), *root, smpi_comm_f2c(*comm), &tmp, array_of_errcodes); if(*ierr == MPI_SUCCESS) { *intercomm = smpi_comm_c2f(tmp); } } void mpi_comm_get_parent_ ( int* parent, int* ierr){ MPI_Comm tmp; *ierr = MPI_Comm_get_parent( &tmp); if(*ierr == MPI_SUCCESS) { *parent = smpi_comm_c2f(tmp); } } void mpi_file_close_ ( int* file, int* ierr){ *ierr= MPI_File_close(reinterpret_cast(*file)); } void mpi_file_delete_ ( char* filename, int* info, int* ierr){ *ierr= MPI_File_delete(filename, smpi_info_f2c(*info)); } void mpi_file_open_ ( int* comm, char* filename, int* amode, int* info, int* fh, int* ierr){ *ierr= MPI_File_open(smpi_comm_f2c(*comm), filename, *amode, smpi_info_f2c(*info), reinterpret_cast(*fh)); } void mpi_file_set_view_ ( int* fh, long long int* offset, int* etype, int* filetype, char* datarep, int* info, int* ierr){ *ierr= MPI_File_set_view(reinterpret_cast(*fh) , reinterpret_cast(*offset), smpi_type_f2c(*etype), smpi_type_f2c(*filetype), datarep, smpi_info_f2c(*info)); } void mpi_file_read_ ( int* fh, void* buf, int* count, int* datatype, MPI_Status* status, int* ierr){ *ierr= MPI_File_read(reinterpret_cast(*fh), buf, *count, smpi_type_f2c(*datatype), status); } void mpi_file_write_ ( int* fh, void* buf, int* count, int* datatype, MPI_Status* status, int* ierr){ *ierr= MPI_File_write(reinterpret_cast(*fh), buf, *count, smpi_type_f2c(*datatype), status); } SimGrid-3.14.159/src/smpi/colls/000755 001750 001750 00000000000 13031031273 017744 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/src/smpi/colls/allgather-2dmesh.c000644 001750 001750 00000014647 13031031271 023245 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** Copyright (c) 2006, Ahmad Faraj & Xin Yuan, All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Florida State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ************************************************************************* * Any results obtained from executing this software require the * * acknowledgment and citation of the software and its owners. * * The full citation is given below: * * * * A. Faraj and X. Yuan. "Automatic Generation and Tuning of MPI * * Collective Communication Routines." The 19th ACM International * * Conference on Supercomputing (ICS), Cambridge, Massachusetts, * * June 20-22, 2005. * ************************************************************************* *****************************************************************************/ /***************************************************************************** * Function: is_2dmesh * Return: int * Inputs: num: the number of processors in a communicator i: x dimension j: y dimension * Descp: takes a number and tries to find a factoring of x, y mesh out of it * Auther: Ahmad Faraj ****************************************************************************/ #ifndef TWOD #define TWOD static int is_2dmesh(int num, int *i, int *j) { int x, max = num / 2; x = sqrt(num); while (x <= max) { if ((num % x) == 0) { *i = x; *j = num / x; if (*i > *j) { x = *i; *i = *j; *j = x; } return 1; } x++; } return 0; } #endif /***************************************************************************** * Function: allgather_2dmesh_shoot * return: int * send_buff: send input buffer * send_count: number of elements to send * send_type: data type of elements being sent * recv_buff: receive output buffer * recv_count: number of elements to received * recv_type: data type of elements being received * comm: communication * Descrp: Function realizes the allgather operation using the 2dmesh * algorithm. Allgather ommunication occurs first in the x dimension then in * the y dimension. The communication in each dimension follows * "simple" * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_allgather_2dmesh(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Request *req, *req_ptr; MPI_Aint extent; int i, src, dst, rank, num_procs; int X, Y, send_offset, recv_offset; int my_row_base, my_col_base, src_row_base, block_size, num_reqs; int tag = COLL_TAG_ALLGATHER; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); extent = smpi_datatype_get_extent(send_type); block_size = extent * send_count; if (!is_2dmesh(num_procs, &X, &Y)) THROWF(arg_error,0, "allgather_2dmesh algorithm can't be used with this number of processes! "); my_row_base = (rank / Y) * Y; my_col_base = rank % Y; num_reqs = X; if (Y > X) num_reqs = Y; req = (MPI_Request *) xbt_malloc(num_reqs * sizeof(MPI_Request)); req_ptr = req; // do local allgather/local copy recv_offset = rank * block_size; smpi_datatype_copy(send_buff, send_count, send_type, (char *)recv_buff + recv_offset, recv_count, recv_type); // do row-wise comm for (i = 0; i < Y; i++) { src = i + my_row_base; if (src == rank) continue; recv_offset = src * block_size; *(req_ptr++) = smpi_mpi_irecv((char *)recv_buff + recv_offset, recv_count, recv_type, src, tag, comm); } for (i = 0; i < Y; i++) { dst = i + my_row_base; if (dst == rank) continue; smpi_mpi_send(send_buff, send_count, send_type, dst, tag, comm); } smpi_mpi_waitall(Y - 1, req, MPI_STATUSES_IGNORE); req_ptr = req; // do colwise comm for (i = 0; i < X; i++) { src = (i * Y + my_col_base); if (src == rank) continue; src_row_base = (src / Y) * Y; recv_offset = src_row_base * block_size; *(req_ptr++) = smpi_mpi_irecv((char *)recv_buff + recv_offset, recv_count * Y, recv_type, src, tag, comm); } for (i = 0; i < X; i++) { dst = (i * Y + my_col_base); if (dst == rank) continue; send_offset = my_row_base * block_size; smpi_mpi_send((char *)recv_buff + send_offset, send_count * Y, send_type, dst, tag, comm); } smpi_mpi_waitall(X - 1, req, MPI_STATUSES_IGNORE); free(req); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/alltoallv-ring-one-barrier.c000644 001750 001750 00000003745 13031031271 025251 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** * Function: alltoall_ring * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works in P - 1 steps. In step i, node j - i -> j -> j + i. * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_alltoallv_ring_one_barrier(void *send_buff, int *send_counts, int *send_disps, MPI_Datatype send_type, void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Status s; MPI_Aint send_chunk, recv_chunk; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALLV; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); send_chunk = smpi_datatype_get_extent(send_type); recv_chunk = smpi_datatype_get_extent(recv_type); smpi_mpi_barrier(comm); for (i = 0; i < num_procs; i++) { src = (rank - i + num_procs) % num_procs; dst = (rank + i) % num_procs; smpi_mpi_sendrecv(send_ptr + send_disps[dst] * send_chunk, send_counts[dst], send_type, dst, tag, recv_ptr + recv_disps[src] * recv_chunk, recv_counts[src], recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allgather-bruck.c000644 001750 001750 00000013154 13031031271 023161 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** Copyright (c) 2006, Ahmad Faraj & Xin Yuan, All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Florida State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ************************************************************************* * Any results obtained from executing this software require the * * acknowledgment and citation of the software and its owners. * * The full citation is given below: * * * * A. Faraj and X. Yuan. "Automatic Generation and Tuning of MPI * * Collective Communication Routines." The 19th ACM International * * Conference on Supercomputing (ICS), Cambridge, Massachusetts, * * June 20-22, 2005. * ************************************************************************* *****************************************************************************/ /***************************************************************************** * Function: allgather_bruck * return: int * inputs: * send_buff: send input buffer * send_count: number of elements to send * send_type: data type of elements being sent * recv_buff: receive output buffer * recv_count: number of elements to received * recv_type: data type of elements being received * comm: communication * Descrp: Function realizes the allgather operation using the bruck * algorithm. * Auther: MPICH * Comment: Original bruck algorithm from MPICH is slightly modified by * Ahmad Faraj. ****************************************************************************/ int smpi_coll_tuned_allgather_bruck(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { // MPI variables MPI_Status status; MPI_Aint recv_extent; // local int variables int src, dst, rank, num_procs, count, remainder; int tag = COLL_TAG_ALLGATHER; int pof2 = 1; // local string variables char *tmp_buff; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; // get size of the communicator, followed by rank num_procs = smpi_comm_size(comm); rank = smpi_comm_rank(comm); // get size of single element's type for recv buffer recv_extent = smpi_datatype_get_extent(recv_type); count = recv_count; tmp_buff = (char *) smpi_get_tmp_sendbuffer(num_procs * recv_count * recv_extent); // perform a local copy smpi_datatype_copy(send_ptr, send_count, send_type, tmp_buff, recv_count, recv_type); while (pof2 <= (num_procs / 2)) { src = (rank + pof2) % num_procs; dst = (rank - pof2 + num_procs) % num_procs; smpi_mpi_sendrecv(tmp_buff, count, recv_type, dst, tag, tmp_buff + count * recv_extent, count, recv_type, src, tag, comm, &status); count *= 2; pof2 *= 2; } remainder = num_procs - pof2; if (remainder) { src = (rank + pof2) % num_procs; dst = (rank - pof2 + num_procs) % num_procs; smpi_mpi_sendrecv(tmp_buff, remainder * recv_count, recv_type, dst, tag, tmp_buff + count * recv_extent, remainder * recv_count, recv_type, src, tag, comm, &status); } smpi_mpi_sendrecv(tmp_buff, (num_procs - rank) * recv_count, recv_type, rank, tag, recv_ptr + rank * recv_count * recv_extent, (num_procs - rank) * recv_count, recv_type, rank, tag, comm, &status); if (rank) smpi_mpi_sendrecv(tmp_buff + (num_procs - rank) * recv_count * recv_extent, rank * recv_count, recv_type, rank, tag, recv_ptr, rank * recv_count, recv_type, rank, tag, comm, &status); smpi_free_tmp_buffer(tmp_buff); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/reduce-mvapich-knomial.c000644 001750 001750 00000015576 13031031272 024451 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2012 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * Copyright (c) 2008 Sun Microsystems, Inc. All rights reserved. * Copyright (c) 2009 University of Houston. All rights reserved. * * Additional copyrights may follow */ /* * * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ /* Copyright (c) 2001-2014, The Ohio State University. All rights * reserved. * * This file is part of the MVAPICH2 software package developed by the * team members of The Ohio State University's Network-Based Computing * Laboratory (NBCL), headed by Professor Dhabaleswar K. (DK) Panda. * * For detailed copyright and licensing information, please refer to the * copyright file COPYRIGHT in the top level MVAPICH2 directory. * */ #include "colls_private.h" extern int mv2_reduce_intra_knomial_factor; extern int mv2_reduce_inter_knomial_factor; #define SMPI_DEFAULT_KNOMIAL_FACTOR 4 // int mv2_reduce_knomial_factor = 2; static int MPIR_Reduce_knomial_trace(int root, int reduce_knomial_factor, MPI_Comm comm, int *dst, int *expected_send_count, int *expected_recv_count, int **src_array) { int mask=0x1, k, comm_size, src, rank, relative_rank, lroot=0; int orig_mask=0x1; int recv_iter=0, send_iter=0; int *knomial_reduce_src_array=NULL; comm_size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); lroot = root; relative_rank = (rank - lroot + comm_size) % comm_size; /* First compute to whom we need to send data */ while (mask < comm_size) { if (relative_rank % (reduce_knomial_factor*mask)) { *dst = relative_rank/(reduce_knomial_factor*mask)* (reduce_knomial_factor*mask)+root; if (*dst >= comm_size) { *dst -= comm_size; } send_iter++; break; } mask *= reduce_knomial_factor; } mask /= reduce_knomial_factor; /* Now compute how many children we have in the knomial-tree */ orig_mask = mask; while (mask > 0) { for(k=1;k 0) { knomial_reduce_src_array = smpi_get_tmp_sendbuffer(sizeof(int)*recv_iter); } mask = orig_mask; recv_iter=0; while (mask > 0) { for(k=1;k= comm_size) { src -= comm_size; } knomial_reduce_src_array[recv_iter++] = src; } } mask /= reduce_knomial_factor; } *expected_recv_count = recv_iter; *expected_send_count = send_iter; *src_array = knomial_reduce_src_array; return 0; } int smpi_coll_tuned_reduce_mvapich2_knomial ( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { int mpi_errno = MPI_SUCCESS; int rank, is_commutative; int src, k; MPI_Request send_request; int index=0; MPI_Aint true_lb, true_extent, extent; MPI_Status status; int recv_iter=0, dst=-1, expected_send_count, expected_recv_count; int *src_array=NULL; void **tmp_buf=NULL; MPI_Request *requests=NULL; if (count == 0) return MPI_SUCCESS; rank = smpi_comm_rank(comm); /* Create a temporary buffer */ smpi_datatype_extent(datatype, &true_lb, &true_extent); extent = smpi_datatype_get_extent(datatype); is_commutative = smpi_op_is_commute(op); if (rank != root) { recvbuf=(void *)smpi_get_tmp_recvbuffer(count*(MAX(extent,true_extent))); recvbuf = (void *)((char*)recvbuf - true_lb); } if ((rank != root) || (sendbuf != MPI_IN_PLACE)) { mpi_errno = smpi_datatype_copy(sendbuf, count, datatype, recvbuf, count, datatype); } if(mv2_reduce_intra_knomial_factor<0) { mv2_reduce_intra_knomial_factor = SMPI_DEFAULT_KNOMIAL_FACTOR; } if(mv2_reduce_inter_knomial_factor<0) { mv2_reduce_inter_knomial_factor = SMPI_DEFAULT_KNOMIAL_FACTOR; } MPIR_Reduce_knomial_trace(root, mv2_reduce_intra_knomial_factor, comm, &dst, &expected_send_count, &expected_recv_count, &src_array); if(expected_recv_count > 0 ) { tmp_buf = xbt_malloc(sizeof(void *)*expected_recv_count); requests = xbt_malloc(sizeof(MPI_Request)*expected_recv_count); for(k=0; k < expected_recv_count; k++ ) { tmp_buf[k] = smpi_get_tmp_sendbuffer(count*(MAX(extent,true_extent))); tmp_buf[k] = (void *)((char*)tmp_buf[k] - true_lb); } while(recv_iter < expected_recv_count) { src = src_array[expected_recv_count - (recv_iter+1)]; requests[recv_iter]=smpi_mpi_irecv (tmp_buf[recv_iter], count, datatype ,src, COLL_TAG_REDUCE, comm); recv_iter++; } recv_iter=0; while(recv_iter < expected_recv_count) { index=smpi_mpi_waitany(expected_recv_count, requests, &status); recv_iter++; if (is_commutative) { smpi_op_apply(op, tmp_buf[index], recvbuf, &count, &datatype); } } for(k=0; k < expected_recv_count; k++ ) { smpi_free_tmp_buffer(tmp_buf[k]); } xbt_free(tmp_buf); xbt_free(requests); } if(src_array != NULL) { xbt_free(src_array); } if(rank != root) { send_request=smpi_mpi_isend(recvbuf,count, datatype, dst, COLL_TAG_REDUCE,comm); smpi_mpi_waitall(1, &send_request, &status); smpi_free_tmp_buffer((void *)((char*)recvbuf + true_lb)); } /* --END ERROR HANDLING-- */ return mpi_errno; } SimGrid-3.14.159/src/smpi/colls/allreduce-smp-rsag.c000644 001750 001750 00000015241 13031031271 023600 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /* This fucntion performs all-reduce operation as follow. 1) binomial_tree reduce inside each SMP node 2) reduce-scatter -inter between root of each SMP node 3) allgather - inter between root of each SMP node 4) binomial_tree bcast inside each SMP node */ int smpi_coll_tuned_allreduce_smp_rsag(void *send_buf, void *recv_buf, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { int comm_size, rank; void *tmp_buf; int tag = COLL_TAG_ALLREDUCE; int mask, src, dst; MPI_Status status; if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } int num_core=1; if (smpi_comm_is_uniform(comm)){ num_core = smpi_comm_size(smpi_comm_get_intra_comm(comm)); } /* #ifdef MPICH2_REDUCTION MPI_User_function * uop = MPIR_Op_table[op % 16 - 1]; #else MPI_User_function *uop; struct MPIR_OP *op_ptr; op_ptr = MPIR_ToPointer(op); uop = op_ptr->op; #endif */ comm_size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); MPI_Aint extent; extent = smpi_datatype_get_extent(dtype); tmp_buf = (void *) smpi_get_tmp_sendbuffer(count * extent); int intra_rank, inter_rank; intra_rank = rank % num_core; inter_rank = rank / num_core; //printf("node %d intra_rank = %d, inter_rank = %d\n", rank, intra_rank, inter_rank); int inter_comm_size = (comm_size + num_core - 1) / num_core; if (!rank) { //printf("intra com size = %d\n",num_core); //printf("inter com size = %d\n",inter_comm_size); } smpi_mpi_sendrecv(send_buf, count, dtype, rank, tag, recv_buf, count, dtype, rank, tag, comm, &status); // SMP_binomial_reduce mask = 1; while (mask < num_core) { if ((mask & intra_rank) == 0) { src = (inter_rank * num_core) + (intra_rank | mask); // if (src < ((inter_rank + 1) * num_core)) { if (src < comm_size) { smpi_mpi_recv(tmp_buf, count, dtype, src, tag, comm, &status); smpi_op_apply(op, tmp_buf, recv_buf, &count, &dtype); //printf("Node %d recv from node %d when mask is %d\n", rank, src, mask); } } else { dst = (inter_rank * num_core) + (intra_rank & (~mask)); smpi_mpi_send(recv_buf, count, dtype, dst, tag, comm); //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); break; } mask <<= 1; } // INTER: reduce-scatter if (intra_rank == 0) { int send_offset, recv_offset; int seg_count = count / inter_comm_size; int to = ((inter_rank + 1) % inter_comm_size) * num_core; int from = ((inter_rank + inter_comm_size - 1) % inter_comm_size) * num_core; int i; //printf("node %d to %d from %d\n",rank,to,from); for (i = 0; i < (inter_comm_size - 1); i++) { send_offset = ((inter_rank - 1 - i + inter_comm_size) % inter_comm_size) * seg_count * extent; recv_offset = ((inter_rank - 2 - i + inter_comm_size) % inter_comm_size) * seg_count * extent; smpi_mpi_sendrecv((char *) recv_buf + send_offset, seg_count, dtype, to, tag + i, tmp_buf, seg_count, dtype, from, tag + i, comm, &status); // result is in rbuf smpi_op_apply(op, tmp_buf, (char *) recv_buf + recv_offset, &seg_count, &dtype); } // INTER: allgather for (i = 0; i < (inter_comm_size - 1); i++) { send_offset = ((inter_rank - i + inter_comm_size) % inter_comm_size) * seg_count * extent; recv_offset = ((inter_rank - 1 - i + inter_comm_size) % inter_comm_size) * seg_count * extent; smpi_mpi_sendrecv((char *) recv_buf + send_offset, seg_count, dtype, to, tag + i, (char *) recv_buf + recv_offset, seg_count, dtype, from, tag + i, comm, &status); } } // INTER_binomial_reduce // only root node for each SMP // if (intra_rank == 0) { // // mask = 1; // while (mask < inter_comm_size) { // if ((mask & inter_rank) == 0) { // src = (inter_rank | mask) * num_core; // if (src < comm_size) { // smpi_mpi_recv(tmp_buf, count, dtype, src, tag, comm, &status); // (* uop) (tmp_buf, recv_buf, &count, &dtype); //printf("Node %d recv from node %d when mask is %d\n", rank, src, mask); // } // } // else { // dst = (inter_rank & (~mask)) * num_core; // smpi_mpi_send(recv_buf, count, dtype, dst, tag, comm); //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); // break; // } // mask <<=1; // } // } // INTER_binomial_bcast // if (intra_rank == 0) { // mask = 1; // while (mask < inter_comm_size) { // if (inter_rank & mask) { // src = (inter_rank - mask) * num_core; //printf("Node %d recv from node %d when mask is %d\n", rank, src, mask); // smpi_mpi_recv(recv_buf, count, dtype, src, tag, comm, &status); // break; // } // mask <<= 1; // } // // mask >>= 1; //printf("My rank = %d my mask = %d\n", rank,mask); // while (mask > 0) { // if (inter_rank < inter_comm_size) { // dst = (inter_rank + mask) * num_core; // if (dst < comm_size) { //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); // smpi_mpi_send(recv_buf, count, dtype, dst, tag, comm); // } // } // mask >>= 1; // } // } // INTRA_binomial_bcast int num_core_in_current_smp = num_core; if (inter_rank == (inter_comm_size - 1)) { num_core_in_current_smp = comm_size - (inter_rank * num_core); } // printf("Node %d num_core = %d\n",rank, num_core_in_current_smp); mask = 1; while (mask < num_core_in_current_smp) { if (intra_rank & mask) { src = (inter_rank * num_core) + (intra_rank - mask); //printf("Node %d recv from node %d when mask is %d\n", rank, src, mask); smpi_mpi_recv(recv_buf, count, dtype, src, tag, comm, &status); break; } mask <<= 1; } mask >>= 1; //printf("My rank = %d my mask = %d\n", rank,mask); while (mask > 0) { dst = (inter_rank * num_core) + (intra_rank + mask); if (dst < comm_size) { //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); smpi_mpi_send(recv_buf, count, dtype, dst, tag, comm); } mask >>= 1; } smpi_free_tmp_buffer(tmp_buf); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allreduce-smp-rdb.c000644 001750 001750 00000013762 13031031271 023421 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /* IMPLEMENTED BY PITCH PATARASUK Non-topoloty-specific (however, number of cores/node need to be changed) all-reduce operation designed for smp clusters It uses 2-layer communication: binomial for intra-communication and rdb for inter-communication*/ /* ** NOTE ** Use -DMPICH2 if this code does not compile. MPICH1 code also work on MPICH2 on our cluster and the performance are similar. This code assume commutative and associative reduce operator (MPI_SUM, MPI_MAX, etc). */ //#include /* This fucntion performs all-reduce operation as follow. 1) binomial_tree reduce inside each SMP node 2) Recursive doubling intra-communication between root of each SMP node 3) binomial_tree bcast inside each SMP node */ int smpi_coll_tuned_allreduce_smp_rdb(void *send_buf, void *recv_buf, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { int comm_size, rank; void *tmp_buf; int tag = COLL_TAG_ALLREDUCE; int mask, src, dst; MPI_Status status; if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } int num_core=1; if (smpi_comm_is_uniform(comm)){ num_core = smpi_comm_size(smpi_comm_get_intra_comm(comm)); } /* #ifdef MPICH2_REDUCTION MPI_User_function * uop = MPIR_Op_table[op % 16 - 1]; #else MPI_User_function *uop; struct MPIR_OP *op_ptr; op_ptr = MPIR_ToPointer(op); uop = op_ptr->op; #endif */ comm_size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); MPI_Aint extent; extent = smpi_datatype_get_extent(dtype); tmp_buf = (void *) smpi_get_tmp_sendbuffer(count * extent); /* compute intra and inter ranking */ int intra_rank, inter_rank; intra_rank = rank % num_core; inter_rank = rank / num_core; /* size of processes participate in intra communications => should be equal to number of machines */ int inter_comm_size = (comm_size + num_core - 1) / num_core; /* copy input buffer to output buffer */ smpi_mpi_sendrecv(send_buf, count, dtype, rank, tag, recv_buf, count, dtype, rank, tag, comm, &status); /* start binomial reduce intra communication inside each SMP node */ mask = 1; while (mask < num_core) { if ((mask & intra_rank) == 0) { src = (inter_rank * num_core) + (intra_rank | mask); if (src < comm_size) { smpi_mpi_recv(tmp_buf, count, dtype, src, tag, comm, &status); smpi_op_apply(op, tmp_buf, recv_buf, &count, &dtype); } } else { dst = (inter_rank * num_core) + (intra_rank & (~mask)); smpi_mpi_send(recv_buf, count, dtype, dst, tag, comm); break; } mask <<= 1; } /* end binomial reduce intra-communication */ /* start rdb (recursive doubling) all-reduce inter-communication between each SMP nodes : each node only have one process that can communicate to other nodes */ if (intra_rank == 0) { /* find nearest power-of-two less than or equal to inter_comm_size */ int pof2, rem, newrank, newdst; pof2 = 1; while (pof2 <= inter_comm_size) pof2 <<= 1; pof2 >>= 1; rem = inter_comm_size - pof2; /* In the non-power-of-two case, all even-numbered processes of rank < 2*rem send their data to (rank+1). These even-numbered processes no longer participate in the algorithm until the very end. */ if (inter_rank < 2 * rem) { if (inter_rank % 2 == 0) { dst = rank + num_core; smpi_mpi_send(recv_buf, count, dtype, dst, tag, comm); newrank = -1; } else { src = rank - num_core; smpi_mpi_recv(tmp_buf, count, dtype, src, tag, comm, &status); smpi_op_apply(op, tmp_buf, recv_buf, &count, &dtype); newrank = inter_rank / 2; } } else { newrank = inter_rank - rem; } /* example inter-communication RDB rank change algorithm 0,4,8,12..36 <= true rank (assume 4 core per SMP) 0123 4567 89 <= inter_rank 1 3 4567 89 (1,3 got data from 0,2 : 0,2 will be idle until the end) 0 1 4567 89 0 1 2345 67 => newrank */ if (newrank != -1) { mask = 1; while (mask < pof2) { newdst = newrank ^ mask; /* find real rank of dest */ dst = (newdst < rem) ? newdst * 2 + 1 : newdst + rem; dst *= num_core; /* exchange data in rdb manner */ smpi_mpi_sendrecv(recv_buf, count, dtype, dst, tag, tmp_buf, count, dtype, dst, tag, comm, &status); smpi_op_apply(op, tmp_buf, recv_buf, &count, &dtype); mask <<= 1; } } /* non pof2 case left-over processes (all even ranks: < 2 * rem) get the result */ if (inter_rank < 2 * rem) { if (inter_rank % 2) { smpi_mpi_send(recv_buf, count, dtype, rank - num_core, tag, comm); } else { smpi_mpi_recv(recv_buf, count, dtype, rank + num_core, tag, comm, &status); } } } /* start binomial broadcast intra-communication inside each SMP nodes */ int num_core_in_current_smp = num_core; if (inter_rank == (inter_comm_size - 1)) { num_core_in_current_smp = comm_size - (inter_rank * num_core); } mask = 1; while (mask < num_core_in_current_smp) { if (intra_rank & mask) { src = (inter_rank * num_core) + (intra_rank - mask); smpi_mpi_recv(recv_buf, count, dtype, src, tag, comm, &status); break; } mask <<= 1; } mask >>= 1; while (mask > 0) { dst = (inter_rank * num_core) + (intra_rank + mask); if (dst < comm_size) { smpi_mpi_send(recv_buf, count, dtype, dst, tag, comm); } mask >>= 1; } smpi_free_tmp_buffer(tmp_buf); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/bcast-arrival-pattern-aware-wait.c000644 001750 001750 00000017224 13031031271 026360 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" int bcast_arrival_pattern_aware_wait_segment_size_in_byte = 8192; #ifndef BCAST_ARRIVAL_PATTERN_AWARE_HEADER_SIZE #define BCAST_ARRIVAL_PATTERN_AWARE_HEADER_SIZE 1024 #endif #ifndef BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE #define BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE 128 #endif /* Non-topology-specific pipelined linear-bcast function */ int smpi_coll_tuned_bcast_arrival_pattern_aware_wait(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { MPI_Status status; MPI_Request request; MPI_Request *send_request_array; MPI_Request *recv_request_array; MPI_Status *send_status_array; MPI_Status *recv_status_array; MPI_Status temp_status_array[BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE]; int rank, size; int i, j, k; int tag = -COLL_TAG_BCAST; int will_send[BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE]; int sent_count; int header_index; int flag_array[BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE]; int already_sent[BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE]; int header_buf[BCAST_ARRIVAL_PATTERN_AWARE_HEADER_SIZE]; char temp_buf[BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE]; int max_node = BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE; int header_size = BCAST_ARRIVAL_PATTERN_AWARE_HEADER_SIZE; MPI_Aint extent; extent = smpi_datatype_get_extent(datatype); /* source and destination */ int to, from; rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); /* segment is segment size in number of elements (not bytes) */ int segment = bcast_arrival_pattern_aware_wait_segment_size_in_byte / extent; segment = segment == 0 ? 1 :segment; /* pipeline length */ int pipe_length = count / segment; /* use for buffer offset for sending and receiving data = segment size in byte */ int increment = segment * extent; /* if the input size is not divisible by segment size => the small remainder will be done with native implementation */ int remainder = count % segment; /* if root is not zero send to rank zero first this can be modified to make it faster by using logical src, dst. */ if (root != 0) { if (rank == root) { smpi_mpi_send(buf, count, datatype, 0, tag, comm); } else if (rank == 0) { smpi_mpi_recv(buf, count, datatype, root, tag, comm, &status); } } /* value == 0 means root has not send data (or header) to the node yet */ for (i = 0; i < max_node; i++) { already_sent[i] = 0; } /* when a message is smaller than a block size => no pipeline */ if (count <= segment) { segment = count; pipe_length = 1; } /* start pipeline bcast */ send_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); recv_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); send_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); recv_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); /* root */ if (rank == 0) { sent_count = 0; int iteration = 0; for (i = 0; i < BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE; i++) will_send[i] = 0; while (sent_count < (size - 1)) { iteration++; /* loop k times to let more processes arrive before start sending data */ for (k = 0; k < 3; k++) { for (i = 1; i < size; i++) { if ((already_sent[i] == 0) && (will_send[i] == 0)) { smpi_mpi_iprobe(i, MPI_ANY_TAG, comm, &flag_array[i], &temp_status_array[i]); if (flag_array[i] == 1) { will_send[i] = 1; smpi_mpi_recv(&temp_buf[i], 1, MPI_CHAR, i, tag, comm, &status); i = 0; } } } } header_index = 0; /* recv 1-byte message */ for (i = 1; i < size; i++) { /* message arrive */ if ((will_send[i] == 1) && (already_sent[i] == 0)) { header_buf[header_index] = i; header_index++; sent_count++; /* will send in the next step */ already_sent[i] = 1; } } /* send header followed by data */ if (header_index != 0) { header_buf[header_index] = -1; to = header_buf[0]; /* send header */ smpi_mpi_send(header_buf, header_size, MPI_INT, to, tag, comm); /* send data - pipeline */ for (i = 0; i < pipe_length; i++) { send_request_array[i] = smpi_mpi_isend((char *)buf + (i * increment), segment, datatype, to, tag, comm); } smpi_mpi_waitall((pipe_length), send_request_array, send_status_array); } /* end - send header followed by data */ /* randomly MPI_Send to one node */ /* this part has been commented out - performance-wise */ else if (2 == 3) { /* search for the first node that never received data before */ for (i = 0; i < size; i++) { if (i == root) continue; if (already_sent[i] == 0) { header_buf[0] = i; header_buf[1] = -1; to = i; smpi_mpi_send(header_buf, header_size, MPI_INT, to, tag, comm); /* still need to chop data so that we can use the same non-root code */ for (j = 0; j < pipe_length; j++) { smpi_mpi_send((char *)buf + (j * increment), segment, datatype, to, tag, comm); } } } } } /* end - while (send_count < size-1) loop */ } /* end - root */ /* none root */ else { /* send 1-byte message to root */ smpi_mpi_send(temp_buf, 1, MPI_CHAR, 0, tag, comm); /* wait for header forward when required */ request = smpi_mpi_irecv(header_buf, header_size, MPI_INT, MPI_ANY_SOURCE, tag, comm); smpi_mpi_wait(&request, MPI_STATUS_IGNORE); /* search for where it is */ int myordering = 0; while (rank != header_buf[myordering]) { myordering++; } to = header_buf[myordering + 1]; if (myordering == 0) { from = 0; } else { from = header_buf[myordering - 1]; } /* send header when required */ if (to != -1) { smpi_mpi_send(header_buf, header_size, MPI_INT, to, tag, comm); } /* receive data */ for (i = 0; i < pipe_length; i++) { recv_request_array[i] = smpi_mpi_irecv((char *)buf + (i * increment), segment, datatype, from, tag, comm); } /* forward data */ if (to != -1) { for (i = 0; i < pipe_length; i++) { smpi_mpi_wait(&recv_request_array[i], MPI_STATUS_IGNORE); send_request_array[i] = smpi_mpi_isend((char *)buf + (i * increment), segment, datatype, to, tag, comm); } smpi_mpi_waitall((pipe_length), send_request_array, send_status_array); } /* recv only */ else { smpi_mpi_waitall((pipe_length), recv_request_array, recv_status_array); } } free(send_request_array); free(recv_request_array); free(send_status_array); free(recv_status_array); /* end pipeline */ /* when count is not divisible by block size, use default BCAST for the remainder */ if ((remainder != 0) && (count > segment)) { XBT_WARN("MPI_bcast_arrival_pattern_aware_wait use default MPI_bcast."); smpi_mpi_bcast((char *)buf + (pipe_length * increment), remainder, datatype, root, comm); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/alltoallv-bruck.c000644 001750 001750 00000006564 13031031271 023217 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /** * Alltoall Bruck * * Openmpi calls this routine when the message size sent to each rank < 2000 bytes and size < 12 * FIXME: uh, check smpi_pmpi again, but this routine is called for > 12, not * less... **/ int smpi_coll_tuned_alltoallv_bruck(void *sendbuf, int *sendcounts, int *senddisps, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *recvdisps, MPI_Datatype recvtype, MPI_Comm comm) { int system_tag = COLL_TAG_ALLTOALLV; int i, rank, size, err, count; MPI_Aint lb; MPI_Aint sendext = 0; MPI_Aint recvext = 0; MPI_Request *requests; // FIXME: check implementation rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); XBT_DEBUG("<%d> algorithm alltoall_bruck() called.", rank); smpi_datatype_extent(sendtype, &lb, &sendext); smpi_datatype_extent(recvtype, &lb, &recvext); /* Local copy from self */ err = smpi_datatype_copy((char *)sendbuf + senddisps[rank] * sendext, sendcounts[rank], sendtype, (char *)recvbuf + recvdisps[rank] * recvext, recvcounts[rank], recvtype); if (err == MPI_SUCCESS && size > 1) { /* Initiate all send/recv to/from others. */ int bblock = 4;//MPIR_PARAM_ALLTOALL_THROTTLE //if (bblock == 0) bblock = comm_size; // requests = xbt_new(MPI_Request, 2 * (bblock - 1)); int ii, ss, dst; /* post only bblock isends/irecvs at a time as suggested by Tony Ladd */ for (ii=0; ii skip request creation [src = %d, recvcount = %d]", rank, i, recvcounts[dst]); continue; } requests[count]=smpi_mpi_irecv((char *)recvbuf + recvdisps[dst] * recvext, recvcounts[dst], recvtype, dst, system_tag, comm ); count++; } /* Now create all sends */ for ( i=0; i skip request creation [dst = %d, sendcount = %d]", rank, i, sendcounts[dst]); continue; } requests[count]=smpi_mpi_isend((char *)sendbuf + senddisps[dst] * sendext, sendcounts[dst], sendtype, dst, system_tag, comm); count++; } /* Wait for them all. */ //smpi_mpi_startall(count, requests); XBT_DEBUG("<%d> wait for %d requests", rank, count); smpi_mpi_waitall(count, requests, MPI_STATUSES_IGNORE); xbt_free(requests); } } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/alltoallv-ompi-basic-linear.c000644 001750 001750 00000007046 13031031271 025400 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /* * Linear functions are copied from the basic coll module. For * some small number of nodes and/or small data sizes they are just as * fast as tuned/tree based segmenting operations and as such may be * selected by the decision functions. These are copied into this module * due to the way we select modules in V1. i.e. in V2 we will handle this * differently and so will not have to duplicate code. * GEF Oct05 after asking Jeff. */ int smpi_coll_tuned_alltoallv_ompi_basic_linear(void *sbuf, int *scounts, int *sdisps, MPI_Datatype sdtype, void *rbuf, int *rcounts, int *rdisps, MPI_Datatype rdtype, MPI_Comm comm) { int i, size, rank; char *psnd, *prcv; int nreqs; ptrdiff_t sext, rext; MPI_Request *preq; size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); MPI_Request *ireqs= xbt_malloc(sizeof(MPI_Request) * size * 2); XBT_DEBUG( "coll:tuned:alltoallv_intra_basic_linear rank %d", rank); sext=smpi_datatype_get_extent(sdtype); rext=smpi_datatype_get_extent(rdtype); /* Simple optimization - handle send to self first */ psnd = ((char *) sbuf) + (sdisps[rank] * sext); prcv = ((char *) rbuf) + (rdisps[rank] * rext); if (0 != scounts[rank]) { smpi_datatype_copy(psnd, scounts[rank], sdtype, prcv, rcounts[rank], rdtype); } /* If only one process, we're done. */ if (1 == size) { return MPI_SUCCESS; } /* Now, initiate all send/recv to/from others. */ nreqs = 0; preq = ireqs; /* Post all receives first */ for (i = 0; i < size; ++i) { if (i == rank || 0 == rcounts[i]) { continue; } prcv = ((char *) rbuf) + (rdisps[i] * rext); *preq = smpi_irecv_init(prcv, rcounts[i], rdtype, i, COLL_TAG_ALLTOALLV, comm ); preq++; ++nreqs; } /* Now post all sends */ for (i = 0; i < size; ++i) { if (i == rank || 0 == scounts[i]) { continue; } psnd = ((char *) sbuf) + (sdisps[i] * sext); *preq=smpi_isend_init(psnd, scounts[i], sdtype, i, COLL_TAG_ALLTOALLV, comm ); preq++; ++nreqs; } /* Start your engines. This will never return an error. */ smpi_mpi_startall(nreqs, ireqs); /* Wait for them all. If there's an error, note that we don't care * what the error was -- just that there *was* an error. The PML * will finish all requests, even if one or more of them fail. * i.e., by the end of this call, all the requests are free-able. * So free them anyway -- even if there was an error, and return the * error after we free everything. */ smpi_mpi_waitall(nreqs, ireqs, MPI_STATUSES_IGNORE); /* Free the requests. */ for (i = 0; i < nreqs; ++i) { if(ireqs[i]!=MPI_REQUEST_NULL)smpi_mpi_request_free(&ireqs[i]); } free(ireqs); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allgather-NTSLR-NB.c000644 001750 001750 00000004463 13031031271 023255 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" // Allgather-Non-Topoloty-Scecific-Logical-Ring algorithm int smpi_coll_tuned_allgather_NTSLR_NB(void *sbuf, int scount, MPI_Datatype stype, void *rbuf, int rcount, MPI_Datatype rtype, MPI_Comm comm) { MPI_Aint rextent, sextent; MPI_Status status, status2; int i, to, from, rank, size; int send_offset, recv_offset; int tag = COLL_TAG_ALLGATHER; rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); rextent = smpi_datatype_get_extent(rtype); sextent = smpi_datatype_get_extent(stype); MPI_Request *rrequest_array; MPI_Request *srequest_array; rrequest_array = (MPI_Request *) xbt_malloc(size * sizeof(MPI_Request)); srequest_array = (MPI_Request *) xbt_malloc(size * sizeof(MPI_Request)); // irregular case use default MPI fucntions if (scount * sextent != rcount * rextent) { XBT_WARN("MPI_allgather_NTSLR_NB use default MPI_allgather."); smpi_mpi_allgather(sbuf, scount, stype, rbuf, rcount, rtype, comm); return MPI_SUCCESS; } // topo non-specific to = (rank + 1) % size; from = (rank + size - 1) % size; //copy a single segment from sbuf to rbuf send_offset = rank * scount * sextent; smpi_mpi_sendrecv(sbuf, scount, stype, rank, tag, (char *)rbuf + send_offset, rcount, rtype, rank, tag, comm, &status); //start sending logical ring message int increment = scount * sextent; //post all irecv first for (i = 0; i < size - 1; i++) { recv_offset = ((rank - i - 1 + size) % size) * increment; rrequest_array[i] = smpi_mpi_irecv((char *)rbuf + recv_offset, rcount, rtype, from, tag + i, comm); } for (i = 0; i < size - 1; i++) { send_offset = ((rank - i + size) % size) * increment; srequest_array[i] = smpi_mpi_isend((char *)rbuf + send_offset, scount, stype, to, tag + i, comm); smpi_mpi_wait(&rrequest_array[i], &status); smpi_mpi_wait(&srequest_array[i], &status2); } free(rrequest_array); free(srequest_array); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/bcast-ompi-split-bintree.c000644 001750 001750 00000031147 13031031272 024732 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2009 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * Copyright (c) 2009 University of Houston. All rights reserved. * * Additional copyrights may follow * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are * met: * - Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * - Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer listed * in this license in the documentation and/or other materials * provided with the distribution. * - Neither the name of the copyright holders nor the names of its * contributors may be used to endorse or promote products derived from * this software without specific prior written permission. * The copyright holders provide no reassurances that the source code * provided does not infringe any patent, copyright, or any other * intellectual property rights of third parties. The copyright holders * disclaim any liability to any recipient for claims brought against * recipient by any third party for infringement of that parties * intellectual property rights. * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #include "colls_private.h" #include "coll_tuned_topo.h" #define MAXTREEFANOUT 32 int smpi_coll_tuned_bcast_ompi_split_bintree ( void* buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int segsize ; int rank, size; int segindex, i, lr, pair; int segcount[2]; /* Number ompi_request_wait_allof elements sent with each segment */ uint32_t counts[2]; int num_segments[2]; /* Number of segmenets */ int sendcount[2]; /* the same like segcount, except for the last segment */ size_t realsegsize[2]; char *tmpbuf[2]; size_t type_size; ptrdiff_t type_extent; MPI_Request base_req, new_req; ompi_coll_tree_t *tree; // mca_coll_tuned_module_t *tuned_module = (mca_coll_tuned_module_t*) module; // mca_coll_tuned_comm_t *data = tuned_module->tuned_data; size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); //compute again segsize const size_t intermediate_message_size = 370728; size_t message_size = smpi_datatype_size(datatype) * (unsigned long)count; if(message_size < intermediate_message_size) segsize = 1024 ; else segsize = 1024 << 3; XBT_DEBUG("ompi_coll_tuned_bcast_intra_split_bintree rank %d root %d ss %5d", rank, root, segsize); if (size == 1) { return MPI_SUCCESS; } /* setup the binary tree topology. */ tree = ompi_coll_tuned_topo_build_tree(2,comm,root); type_size = smpi_datatype_size( datatype ); /* Determine number of segments and number of elements per segment */ counts[0] = count/2; if (count % 2 != 0) counts[0]++; counts[1] = count - counts[0]; if ( segsize > 0 ) { /* Note that ompi_datatype_type_size() will never return a negative value in typelng; it returns an int [vs. an unsigned type] because of the MPI spec. */ if (segsize < ((uint32_t) type_size)) { segsize = type_size; /* push segsize up to hold one type */ } segcount[0] = segcount[1] = segsize / type_size; num_segments[0] = counts[0]/segcount[0]; if ((counts[0] % segcount[0]) != 0) num_segments[0]++; num_segments[1] = counts[1]/segcount[1]; if ((counts[1] % segcount[1]) != 0) num_segments[1]++; } else { segcount[0] = counts[0]; segcount[1] = counts[1]; num_segments[0] = num_segments[1] = 1; } /* if the message is too small to be split into segments */ if( (counts[0] == 0 || counts[1] == 0) || (segsize > counts[0] * type_size) || (segsize > counts[1] * type_size) ) { /* call linear version here ! */ return (smpi_coll_tuned_bcast_SMP_linear ( buffer, count, datatype, root, comm)); } type_extent = smpi_datatype_get_extent(datatype); /* Determine real segment size */ realsegsize[0] = segcount[0] * type_extent; realsegsize[1] = segcount[1] * type_extent; /* set the buffer pointers */ tmpbuf[0] = (char *) buffer; tmpbuf[1] = (char *) buffer+counts[0] * type_extent; /* Step 1: Root splits the buffer in 2 and sends segmented message down the branches. Left subtree of the tree receives first half of the buffer, while right subtree receives the remaining message. */ /* determine if I am left (0) or right (1), (root is right) */ lr = ((rank + size - root)%size + 1)%2; /* root code */ if( rank == root ) { /* determine segment count */ sendcount[0] = segcount[0]; sendcount[1] = segcount[1]; /* for each segment */ for (segindex = 0; segindex < num_segments[0]; segindex++) { /* for each child */ for( i = 0; i < tree->tree_nextsize && i < 2; i++ ) { if (segindex >= num_segments[i]) { /* no more segments */ continue; } /* determine how many elements are being sent in this round */ if(segindex == (num_segments[i] - 1)) sendcount[i] = counts[i] - segindex*segcount[i]; /* send data */ smpi_mpi_send(tmpbuf[i], sendcount[i], datatype, tree->tree_next[i], COLL_TAG_BCAST, comm); /* update tmp buffer */ tmpbuf[i] += realsegsize[i]; } } } /* intermediate nodes code */ else if( tree->tree_nextsize > 0 ) { /* Intermediate nodes: * It will receive segments only from one half of the data. * Which one is determined by whether the node belongs to the "left" or "right" * subtree. Topoloby building function builds binary tree such that * odd "shifted ranks" ((rank + size - root)%size) are on the left subtree, * and even on the right subtree. * * Create the pipeline. We first post the first receive, then in the loop we * post the next receive and after that wait for the previous receive to complete * and we disseminating the data to all children. */ sendcount[lr] = segcount[lr]; base_req=smpi_mpi_irecv(tmpbuf[lr], sendcount[lr], datatype, tree->tree_prev, COLL_TAG_BCAST, comm); for( segindex = 1; segindex < num_segments[lr]; segindex++ ) { /* determine how many elements to expect in this round */ if( segindex == (num_segments[lr] - 1)) sendcount[lr] = counts[lr] - segindex*segcount[lr]; /* post new irecv */ new_req = smpi_mpi_irecv( tmpbuf[lr] + realsegsize[lr], sendcount[lr], datatype, tree->tree_prev, COLL_TAG_BCAST, comm); /* wait for and forward current segment */ smpi_mpi_waitall( 1, &base_req, MPI_STATUSES_IGNORE ); for( i = 0; i < tree->tree_nextsize; i++ ) { /* send data to children (segcount[lr]) */ smpi_mpi_send( tmpbuf[lr], segcount[lr], datatype, tree->tree_next[i], COLL_TAG_BCAST, comm); } /* end of for each child */ /* upate the base request */ base_req = new_req; /* go to the next buffer (ie. the one corresponding to the next recv) */ tmpbuf[lr] += realsegsize[lr]; } /* end of for segindex */ /* wait for the last segment and forward current segment */ smpi_mpi_waitall( 1, &base_req, MPI_STATUSES_IGNORE ); for( i = 0; i < tree->tree_nextsize; i++ ) { /* send data to children */ smpi_mpi_send(tmpbuf[lr], sendcount[lr], datatype, tree->tree_next[i], COLL_TAG_BCAST, comm); } /* end of for each child */ } /* leaf nodes */ else { /* Just consume segments as fast as possible */ sendcount[lr] = segcount[lr]; for (segindex = 0; segindex < num_segments[lr]; segindex++) { /* determine how many elements to expect in this round */ if (segindex == (num_segments[lr] - 1)) sendcount[lr] = counts[lr] - segindex*segcount[lr]; /* receive segments */ smpi_mpi_recv(tmpbuf[lr], sendcount[lr], datatype, tree->tree_prev, COLL_TAG_BCAST, comm, MPI_STATUS_IGNORE); /* update the initial pointer to the buffer */ tmpbuf[lr] += realsegsize[lr]; } } /* reset the buffer pointers */ tmpbuf[0] = (char *) buffer; tmpbuf[1] = (char *) buffer+counts[0] * type_extent; /* Step 2: Find your immediate pair (identical node in opposite subtree) and SendRecv data buffer with them. The tree building function ensures that if (we are not root) if we are in the left subtree (lr == 0) our pair is (rank+1)%size. if we are in the right subtree (lr == 1) our pair is (rank-1)%size If we have even number of nodes the rank (size-1) will pair up with root. */ if (lr == 0) { pair = (rank+1)%size; } else { pair = (rank+size-1)%size; } if ( (size%2) != 0 && rank != root) { smpi_mpi_sendrecv( tmpbuf[lr], counts[lr], datatype, pair, COLL_TAG_BCAST, tmpbuf[(lr+1)%2], counts[(lr+1)%2], datatype, pair, COLL_TAG_BCAST, comm, MPI_STATUS_IGNORE); } else if ( (size%2) == 0 ) { /* root sends right buffer to the last node */ if( rank == root ) { smpi_mpi_send(tmpbuf[1], counts[1], datatype, (root+size-1)%size, COLL_TAG_BCAST, comm); } /* last node receives right buffer from the root */ else if (rank == (root+size-1)%size) { smpi_mpi_recv(tmpbuf[1], counts[1], datatype, root, COLL_TAG_BCAST, comm, MPI_STATUS_IGNORE); } /* everyone else exchanges buffers */ else { smpi_mpi_sendrecv( tmpbuf[lr], counts[lr], datatype, pair, COLL_TAG_BCAST, tmpbuf[(lr+1)%2], counts[(lr+1)%2], datatype, pair, COLL_TAG_BCAST, comm, MPI_STATUS_IGNORE); } } xbt_free(tree); return (MPI_SUCCESS); } SimGrid-3.14.159/src/smpi/colls/coll_tuned_topo.h000644 001750 001750 00000005621 13031031273 023312 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2005 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * * Additional copyrights may follow */ #ifndef MCA_COLL_TUNED_TOPO_H_HAS_BEEN_INCLUDED #define MCA_COLL_TUNED_TOPO_H_HAS_BEEN_INCLUDED #include "colls_private.h" #define MAXTREEFANOUT 32 #define COLL_TUNED_COMPUTED_SEGCOUNT(SEGSIZE, TYPELNG, SEGCOUNT) \ if( ((SEGSIZE) >= (TYPELNG)) && \ ((SEGSIZE) < ((TYPELNG) * (SEGCOUNT))) ) { \ size_t residual; \ (SEGCOUNT) = (int)((SEGSIZE) / (TYPELNG)); \ residual = (SEGSIZE) - (SEGCOUNT) * (TYPELNG); \ if( residual > ((TYPELNG) >> 1) ) \ (SEGCOUNT)++; \ } \ typedef struct ompi_coll_tree_t { int32_t tree_root; int32_t tree_fanout; int32_t tree_bmtree; int32_t tree_prev; int32_t tree_next[MAXTREEFANOUT]; int32_t tree_nextsize; } ompi_coll_tree_t; ompi_coll_tree_t* ompi_coll_tuned_topo_build_tree( int fanout, MPI_Comm com, int root ); ompi_coll_tree_t* ompi_coll_tuned_topo_build_in_order_bintree( MPI_Comm comm ); ompi_coll_tree_t* ompi_coll_tuned_topo_build_bmtree( MPI_Comm comm, int root ); ompi_coll_tree_t* ompi_coll_tuned_topo_build_in_order_bmtree( MPI_Comm comm, int root ); ompi_coll_tree_t* ompi_coll_tuned_topo_build_chain( int fanout, MPI_Comm com, int root ); int ompi_coll_tuned_topo_destroy_tree( ompi_coll_tree_t** tree ); /* debugging stuff, will be removed later */ int ompi_coll_tuned_topo_dump_tree (ompi_coll_tree_t* tree, int rank); #endif /* MCA_COLL_TUNED_TOPO_H_HAS_BEEN_INCLUDED */ SimGrid-3.14.159/src/smpi/colls/alltoall-ring-one-barrier.c000644 001750 001750 00000003726 13031031271 025062 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** * Function: alltoall_ring * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works in P - 1 steps. In step i, node j - i -> j -> j + i. * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_alltoall_ring_one_barrier(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Status s; MPI_Aint send_chunk, recv_chunk; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALL; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); send_chunk = smpi_datatype_get_extent(send_type); recv_chunk = smpi_datatype_get_extent(recv_type); send_chunk *= send_count; recv_chunk *= recv_count; mpi_coll_barrier_fun(comm); for (i = 0; i < num_procs; i++) { src = (rank - i + num_procs) % num_procs; dst = (rank + i) % num_procs; smpi_mpi_sendrecv(send_ptr + dst * send_chunk, send_count, send_type, dst, tag, recv_ptr + src * recv_chunk, recv_count, recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/bcast-scatter-LR-allgather.c000644 001750 001750 00000015115 13031031272 025125 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** Copyright (c) 2006, Ahmad Faraj & Xin Yuan, All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Florida State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ************************************************************************* * Any results obtained from executing this software require the * * acknowledgment and citation of the software and its owners. * * The full citation is given below: * * * * A. Faraj and X. Yuan. "Automatic Generation and Tuning of MPI * * Collective Communication Routines." The 19th ACM International * * Conference on Supercomputing (ICS), Cambridge, Massachusetts, * * June 20-22, 2005. * ************************************************************************* *****************************************************************************/ /***************************************************************************** * Function: bcast_scatter_LR_allgather * Return: int * Inputs: buff: send input buffer count: number of elements to send data_type: data type of elements being sent root: source of data comm: communicator * Descrp: broadcasts using a scatter followed by LR allgather. * Auther: MPIH / modified by Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_bcast_scatter_LR_allgather(void *buff, int count, MPI_Datatype data_type, int root, MPI_Comm comm) { MPI_Aint extent; MPI_Status status; int i, src, dst, rank, num_procs; int mask, relative_rank, curr_size, recv_size, send_size, nbytes; int scatter_size, left, right, next_src, *recv_counts, *disps; int tag = COLL_TAG_BCAST; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); extent = smpi_datatype_get_extent(data_type); nbytes = extent * count; scatter_size = (nbytes + num_procs - 1) / num_procs; // ceiling division curr_size = (rank == root) ? nbytes : 0; // root starts with all the data relative_rank = (rank >= root) ? rank - root : rank - root + num_procs; mask = 0x1; while (mask < num_procs) { if (relative_rank & mask) { src = rank - mask; if (src < 0) src += num_procs; recv_size = nbytes - relative_rank * scatter_size; // recv_size is larger than what might actually be sent by the // sender. We don't need compute the exact value because MPI // allows you to post a larger recv. if (recv_size <= 0) curr_size = 0; // this process doesn't receive any data // because of uneven division else { smpi_mpi_recv((char *) buff + relative_rank * scatter_size, recv_size, MPI_BYTE, src, tag, comm, &status); curr_size = smpi_mpi_get_count(&status, MPI_BYTE); } break; } mask <<= 1; } // This process is responsible for all processes that have bits // set from the LSB upto (but not including) mask. Because of // the "not including", we start by shifting mask back down // one. mask >>= 1; while (mask > 0) { if (relative_rank + mask < num_procs) { send_size = curr_size - scatter_size * mask; // mask is also the size of this process's subtree if (send_size > 0) { dst = rank + mask; if (dst >= num_procs) dst -= num_procs; smpi_mpi_send((char *) buff + scatter_size * (relative_rank + mask), send_size, MPI_BYTE, dst, tag, comm); curr_size -= send_size; } } mask >>= 1; } // done scatter now do allgather recv_counts = (int *) xbt_malloc(sizeof(int) * num_procs); disps = (int *) xbt_malloc(sizeof(int) * num_procs); for (i = 0; i < num_procs; i++) { recv_counts[i] = nbytes - i * scatter_size; if (recv_counts[i] > scatter_size) recv_counts[i] = scatter_size; if (recv_counts[i] < 0) recv_counts[i] = 0; } disps[0] = 0; for (i = 1; i < num_procs; i++) disps[i] = disps[i - 1] + recv_counts[i - 1]; left = (num_procs + rank - 1) % num_procs; right = (rank + 1) % num_procs; src = rank; next_src = left; for (i = 1; i < num_procs; i++) { smpi_mpi_sendrecv((char *) buff + disps[(src - root + num_procs) % num_procs], recv_counts[(src - root + num_procs) % num_procs], MPI_BYTE, right, tag, (char *) buff + disps[(next_src - root + num_procs) % num_procs], recv_counts[(next_src - root + num_procs) % num_procs], MPI_BYTE, left, tag, comm, &status); src = next_src; next_src = (num_procs + next_src - 1) % num_procs; } free(recv_counts); free(disps); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/bcast-NTSL-Isend.c000644 001750 001750 00000011200 13031031271 023012 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" static int bcast_NTSL_segment_size_in_byte = 8192; /* Non-topology-specific pipelined linear-bcast function 0->1, 1->2 ,2->3, ....., ->last node : in a pipeline fashion */ int smpi_coll_tuned_bcast_NTSL_Isend(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int tag = COLL_TAG_BCAST; MPI_Status status; MPI_Request request; MPI_Request *send_request_array; MPI_Request *recv_request_array; MPI_Status *send_status_array; MPI_Status *recv_status_array; int rank, size; int i; MPI_Aint extent; extent = smpi_datatype_get_extent(datatype); rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); /* source node and destination nodes (same through out the functions) */ int to = (rank + 1) % size; int from = (rank + size - 1) % size; /* segment is segment size in number of elements (not bytes) */ int segment = bcast_NTSL_segment_size_in_byte / extent; segment = segment == 0 ? 1 :segment; /* pipeline length */ int pipe_length = count / segment; /* use for buffer offset for sending and receiving data = segment size in byte */ int increment = segment * extent; /* if the input size is not divisible by segment size => the small remainder will be done with native implementation */ int remainder = count % segment; /* if root is not zero send to rank zero first this can be modified to make it faster by using logical src, dst. */ if (root != 0) { if (rank == root) { smpi_mpi_send(buf, count, datatype, 0, tag, comm); } else if (rank == 0) { smpi_mpi_recv(buf, count, datatype, root, tag, comm, &status); } } /* when a message is smaller than a block size => no pipeline */ if (count <= segment) { if (rank == 0) { smpi_mpi_send(buf, count, datatype, to, tag, comm); } else if (rank == (size - 1)) { request = smpi_mpi_irecv(buf, count, datatype, from, tag, comm); smpi_mpi_wait(&request, &status); } else { request = smpi_mpi_irecv(buf, count, datatype, from, tag, comm); smpi_mpi_wait(&request, &status); smpi_mpi_send(buf, count, datatype, to, tag, comm); } return MPI_SUCCESS; } /* pipeline bcast */ else { send_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); recv_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); send_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); recv_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); /* root send data */ if (rank == 0) { for (i = 0; i < pipe_length; i++) { send_request_array[i] = smpi_mpi_isend((char *) buf + (i * increment), segment, datatype, to, (tag + i), comm); } smpi_mpi_waitall((pipe_length), send_request_array, send_status_array); } /* last node only receive data */ else if (rank == (size - 1)) { for (i = 0; i < pipe_length; i++) { recv_request_array[i] = smpi_mpi_irecv((char *) buf + (i * increment), segment, datatype, from, (tag + i), comm); } smpi_mpi_waitall((pipe_length), recv_request_array, recv_status_array); } /* intermediate nodes relay (receive, then send) data */ else { for (i = 0; i < pipe_length; i++) { recv_request_array[i] = smpi_mpi_irecv((char *) buf + (i * increment), segment, datatype, from, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { smpi_mpi_wait(&recv_request_array[i], &status); send_request_array[i] = smpi_mpi_isend((char *) buf + (i * increment), segment, datatype, to, (tag + i), comm); } smpi_mpi_waitall((pipe_length), send_request_array, send_status_array); } free(send_request_array); free(recv_request_array); free(send_status_array); free(recv_status_array); } /* end pipeline */ /* when count is not divisible by block size, use default BCAST for the remainder */ if ((remainder != 0) && (count > segment)) { XBT_WARN("MPI_bcast_NTSL_Isend_nb use default MPI_bcast."); smpi_mpi_bcast((char *) buf + (pipe_length * increment), remainder, datatype, root, comm); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/alltoall-ring-light-barrier.c000644 001750 001750 00000005167 13031031271 025411 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** * Function: alltoall_ring_light_barrier * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works in P - 1 steps. In step i, node j - i -> j -> j + i. Light barriers are inserted between communications in different phases. * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_alltoall_ring_light_barrier(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint send_chunk, recv_chunk; MPI_Status s; int i, src, dst, rank, num_procs, next_dst, next_src; int tag = COLL_TAG_ALLTOALL; char send_sync = 'a', recv_sync = 'b'; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); send_chunk = smpi_datatype_get_extent(send_type); recv_chunk = smpi_datatype_get_extent(recv_type); send_chunk *= send_count; recv_chunk *= recv_count; smpi_mpi_sendrecv(send_ptr + rank * send_chunk, send_count, send_type, rank, tag, recv_ptr + rank * recv_chunk, recv_count, recv_type, rank, tag, comm, &s); for (i = 1; i < num_procs; i++) { src = (rank - i + num_procs) % num_procs; dst = (rank + i) % num_procs; smpi_mpi_sendrecv(send_ptr + dst * send_chunk, send_count, send_type, dst, tag, recv_ptr + src * recv_chunk, recv_count, recv_type, src, tag, comm, &s); if ((i + 1) < num_procs) { next_src = (rank - (i + 1) + num_procs) % num_procs; next_dst = (rank + (i + 1) + num_procs) % num_procs; smpi_mpi_sendrecv(&send_sync, 1, MPI_CHAR, next_src, tag, &recv_sync, 1, MPI_CHAR, next_dst, tag, comm, &s); } } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allgather-ring.c000644 001750 001750 00000010545 13031031271 023013 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** Copyright (c) 2006, Ahmad Faraj & Xin Yuan, All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Florida State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ************************************************************************* * Any results obtained from executing this software require the * * acknowledgment and citation of the software and its owners. * * The full citation is given below: * * * * A. Faraj and X. Yuan. "Automatic Generation and Tuning of MPI * * Collective Communication Routines." The 19th ACM International * * Conference on Supercomputing (ICS), Cambridge, Massachusetts, * * June 20-22, 2005. * ************************************************************************* *****************************************************************************/ /***************************************************************************** * Function: allgather_ring * return: int * inputs: * send_buff: send input buffer * send_count: number of elements to send * send_type: data type of elements being sent * recv_buff: receive output buffer * recv_count: number of elements to received * recv_type: data type of elements being received * comm: communication * Descrp: Function works in P - 1 steps. In step i, node j - i -> j -> j+ i. * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_allgather_ring(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint extent; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLGATHER; MPI_Status status; char *sendptr = (char *) send_buff; char *recvptr = (char *) recv_buff; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); extent = smpi_datatype_get_extent(send_type); // local send/recv smpi_mpi_sendrecv(sendptr, send_count, send_type, rank, tag, recvptr + rank * recv_count * extent, recv_count, recv_type, rank, tag, comm, &status); for (i = 1; i < num_procs; i++) { src = (rank - i + num_procs) % num_procs; dst = (rank + i) % num_procs; smpi_mpi_sendrecv(sendptr, send_count, send_type, dst, tag, recvptr + src * recv_count * extent, recv_count, recv_type, src, tag, comm, &status); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/alltoallv-pair-mpi-barrier.c000644 001750 001750 00000004276 13031031271 025251 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** * Function: alltoall_pair_mpi_barrier * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works when P is power of two. In each phase of P - 1 phases, nodes in pair communicate their data. MPI barriers are inserted between each two phases. * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_alltoallv_pair_mpi_barrier(void *send_buff, int *send_counts, int *send_disps, MPI_Datatype send_type, void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Status s; MPI_Aint send_chunk, recv_chunk; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALLV; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "alltoallv pair algorithm can't be used with non power of two number of processes ! "); send_chunk = smpi_datatype_get_extent(send_type); recv_chunk = smpi_datatype_get_extent(recv_type); for (i = 0; i < num_procs; i++) { src = dst = rank ^ i; smpi_mpi_barrier(comm); smpi_mpi_sendrecv(send_ptr + send_disps[dst] * send_chunk, send_counts[dst], send_type, dst, tag, recv_ptr + recv_disps[src] * recv_chunk, recv_counts[src], recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allgatherv-ompi-neighborexchange.c000644 001750 001750 00000020676 13031031271 026512 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * ompi_coll_tuned_allgatherv_intra_neighborexchange * * Function: allgatherv using N/2 steps (O(N)) * Accepts: Same arguments as MPI_Allgatherv * Returns: MPI_SUCCESS or error code * * Description: Neighbor Exchange algorithm for allgather adapted for * allgatherv. * Described by Chen et.al. in * "Performance Evaluation of Allgather Algorithms on * Terascale Linux Cluster with Fast Ethernet", * Proceedings of the Eighth International Conference on * High-Performance Computing inn Asia-Pacific Region * (HPCASIA'05), 2005 * * Rank r exchanges message with one of its neighbors and * forwards the data further in the next step. * * No additional memory requirements. * * Limitations: Algorithm works only on even number of processes. * For odd number of processes we switch to ring algorithm. * * Example on 6 nodes: * Initial state * # 0 1 2 3 4 5 * [0] [ ] [ ] [ ] [ ] [ ] * [ ] [1] [ ] [ ] [ ] [ ] * [ ] [ ] [2] [ ] [ ] [ ] * [ ] [ ] [ ] [3] [ ] [ ] * [ ] [ ] [ ] [ ] [4] [ ] * [ ] [ ] [ ] [ ] [ ] [5] * Step 0: * # 0 1 2 3 4 5 * [0] [0] [ ] [ ] [ ] [ ] * [1] [1] [ ] [ ] [ ] [ ] * [ ] [ ] [2] [2] [ ] [ ] * [ ] [ ] [3] [3] [ ] [ ] * [ ] [ ] [ ] [ ] [4] [4] * [ ] [ ] [ ] [ ] [5] [5] * Step 1: * # 0 1 2 3 4 5 * [0] [0] [0] [ ] [ ] [0] * [1] [1] [1] [ ] [ ] [1] * [ ] [2] [2] [2] [2] [ ] * [ ] [3] [3] [3] [3] [ ] * [4] [ ] [ ] [4] [4] [4] * [5] [ ] [ ] [5] [5] [5] * Step 2: * # 0 1 2 3 4 5 * [0] [0] [0] [0] [0] [0] * [1] [1] [1] [1] [1] [1] * [2] [2] [2] [2] [2] [2] * [3] [3] [3] [3] [3] [3] * [4] [4] [4] [4] [4] [4] * [5] [5] [5] [5] [5] [5] */ #include "colls_private.h" int smpi_coll_tuned_allgatherv_ompi_neighborexchange(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int *rcounts, int *rdispls, MPI_Datatype rdtype, MPI_Comm comm) { int line = -1; int rank, size; int neighbor[2], offset_at_step[2], recv_data_from[2], send_data_from; int i, even_rank; int err = 0; ptrdiff_t slb, rlb, sext, rext; char *tmpsend = NULL, *tmprecv = NULL; size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); if (size % 2) { XBT_DEBUG( "coll:tuned:allgatherv_ompi_neighborexchange WARNING: odd size %d, switching to ring algorithm", size); return smpi_coll_tuned_allgatherv_ring(sbuf, scount, sdtype, rbuf, rcounts, rdispls, rdtype, comm); } XBT_DEBUG( "coll:tuned:allgatherv_ompi_neighborexchange rank %d", rank); err = smpi_datatype_extent (sdtype, &slb, &sext); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } err = smpi_datatype_extent (rdtype, &rlb, &rext); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } /* Initialization step: - if send buffer is not MPI_IN_PLACE, copy send buffer to the appropriate block of receive buffer */ tmprecv = (char*) rbuf + rdispls[rank] * rext; if (MPI_IN_PLACE != sbuf) { tmpsend = (char*) sbuf; err = smpi_datatype_copy(tmpsend, scount, sdtype, tmprecv, rcounts[rank], rdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } } /* Determine neighbors, order in which blocks will arrive, etc. */ even_rank = !(rank % 2); if (even_rank) { neighbor[0] = (rank + 1) % size; neighbor[1] = (rank - 1 + size) % size; recv_data_from[0] = rank; recv_data_from[1] = rank; offset_at_step[0] = (+2); offset_at_step[1] = (-2); } else { neighbor[0] = (rank - 1 + size) % size; neighbor[1] = (rank + 1) % size; recv_data_from[0] = neighbor[0]; recv_data_from[1] = neighbor[0]; offset_at_step[0] = (-2); offset_at_step[1] = (+2); } /* Communication loop: - First step is special: exchange a single block with neighbor[0]. - Rest of the steps: update recv_data_from according to offset, and exchange two blocks with appropriate neighbor. the send location becomes previous receve location. Note, we need to create indexed datatype to send and receive these blocks properly. */ tmprecv = (char*)rbuf + rdispls[neighbor[0]] * rext; tmpsend = (char*)rbuf + rdispls[rank] * rext; smpi_mpi_sendrecv(tmpsend, rcounts[rank], rdtype, neighbor[0], COLL_TAG_ALLGATHERV, tmprecv, rcounts[neighbor[0]], rdtype, neighbor[0], COLL_TAG_ALLGATHERV, comm, MPI_STATUS_IGNORE); /* Determine initial sending counts and displacements*/ if (even_rank) { send_data_from = rank; } else { send_data_from = recv_data_from[0]; } for (i = 1; i < (size / 2); i++) { MPI_Datatype new_rdtype, new_sdtype; int new_scounts[2], new_sdispls[2], new_rcounts[2], new_rdispls[2]; const int i_parity = i % 2; recv_data_from[i_parity] = (recv_data_from[i_parity] + offset_at_step[i_parity] + size) % size; /* Create new indexed types for sending and receiving. We are sending data from ranks (send_data_from) and (send_data_from+1) We are receiving data from ranks (recv_data_from[i_parity]) and (recv_data_from[i_parity]+1). */ new_scounts[0] = rcounts[send_data_from]; new_scounts[1] = rcounts[(send_data_from + 1)]; new_sdispls[0] = rdispls[send_data_from]; new_sdispls[1] = rdispls[(send_data_from + 1)]; err = smpi_datatype_indexed(2, new_scounts, new_sdispls, rdtype, &new_sdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } smpi_datatype_commit(&new_sdtype); new_rcounts[0] = rcounts[recv_data_from[i_parity]]; new_rcounts[1] = rcounts[(recv_data_from[i_parity] + 1)]; new_rdispls[0] = rdispls[recv_data_from[i_parity]]; new_rdispls[1] = rdispls[(recv_data_from[i_parity] + 1)]; err = smpi_datatype_indexed(2, new_rcounts, new_rdispls, rdtype, &new_rdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } smpi_datatype_commit(&new_rdtype); tmprecv = (char*)rbuf; tmpsend = (char*)rbuf; /* Sendreceive */ smpi_mpi_sendrecv(tmpsend, 1, new_sdtype, neighbor[i_parity], COLL_TAG_ALLGATHERV, tmprecv, 1, new_rdtype, neighbor[i_parity], COLL_TAG_ALLGATHERV, comm, MPI_STATUS_IGNORE); send_data_from = recv_data_from[i_parity]; smpi_datatype_unuse(new_sdtype); smpi_datatype_unuse(new_rdtype); } return MPI_SUCCESS; err_hndl: XBT_DEBUG( "%s:%4d\tError occurred %d, rank %2d", __FILE__, line, err, rank); return err; } SimGrid-3.14.159/src/smpi/colls/bcast-SMP-binomial.c000644 001750 001750 00000006524 13031031271 023436 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" int smpi_coll_tuned_bcast_SMP_binomial(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int mask = 1; int size; int rank; MPI_Status status; int tag = COLL_TAG_BCAST; size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } int num_core=1; if (smpi_comm_is_uniform(comm)){ num_core = smpi_comm_size(smpi_comm_get_intra_comm(comm)); }else{ //implementation buggy in this case return smpi_coll_tuned_bcast_mpich( buf , count, datatype, root, comm); } int to_intra, to_inter; int from_intra, from_inter; int inter_rank = rank / num_core; int inter_size = (size - 1) / num_core + 1; int intra_rank = rank % num_core; int intra_size = num_core; if (((rank / num_core) * num_core) == ((size / num_core) * num_core)) intra_size = size - (rank / num_core) * num_core; // if root is not zero send to rank zero first if (root != 0) { if (rank == root) smpi_mpi_send(buf, count, datatype, 0, tag, comm); else if (rank == 0) smpi_mpi_recv(buf, count, datatype, root, tag, comm, &status); } //FIRST STEP node 0 send to every root-of-each-SMP with binomial tree //printf("node %d inter_rank = %d, inter_size = %d\n",rank,inter_rank, inter_size); if (intra_rank == 0) { mask = 1; while (mask < inter_size) { if (inter_rank & mask) { from_inter = (inter_rank - mask) * num_core; //printf("Node %d recv from node %d when mask is %d\n", rank, from_inter, mask); smpi_mpi_recv(buf, count, datatype, from_inter, tag, comm, &status); break; } mask <<= 1; } mask >>= 1; //printf("My rank = %d my mask = %d\n", rank,mask); while (mask > 0) { if (inter_rank < inter_size) { to_inter = (inter_rank + mask) * num_core; if (to_inter < size) { //printf("Node %d send to node %d when mask is %d\n", rank, to_inter, mask); smpi_mpi_send(buf, count, datatype, to_inter, tag, comm); } } mask >>= 1; } } // SECOND STEP every root-of-each-SMP send to all children with binomial tree // base is a rank of root-of-each-SMP int base = (rank / num_core) * num_core; mask = 1; while (mask < intra_size) { if (intra_rank & mask) { from_intra = base + (intra_rank - mask); //printf("Node %d recv from node %d when mask is %d\n", rank, from_inter, mask); smpi_mpi_recv(buf, count, datatype, from_intra, tag, comm, &status); break; } mask <<= 1; } mask >>= 1; //printf("My rank = %d my mask = %d\n", rank,mask); while (mask > 0) { if (intra_rank < intra_size) { to_intra = base + (intra_rank + mask); if (to_intra < size) { //printf("Node %d send to node %d when mask is %d\n", rank, to_inter, mask); smpi_mpi_send(buf, count, datatype, to_intra, tag, comm); } } mask >>= 1; } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allgather-pair.c000644 001750 001750 00000011016 13031031271 023001 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** Copyright (c) 2006, Ahmad Faraj & Xin Yuan, All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Florida State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ************************************************************************* * Any results obtained from executing this software require the * * acknowledgment and citation of the software and its owners. * * The full citation is given below: * * * * A. Faraj and X. Yuan. "Automatic Generation and Tuning of MPI * * Collective Communication Routines." The 19th ACM International * * Conference on Supercomputing (ICS), Cambridge, Massachusetts, * * June 20-22, 2005. * ************************************************************************* *****************************************************************************/ /***************************************************************************** * Function: allgather_pair * return: int * inputs: * send_buff: send input buffer * send_count: number of elements to send * send_type: data type of elements being sent * recv_buff: receive output buffer * recv_count: number of elements to received * recv_type: data type of elements being received * comm: communication * Descrp: Function works when P is power of two. In each phase of P - 1 * phases, nodes in pair communicate their data. * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_allgather_pair(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint extent; unsigned int i, src, dst; int tag = COLL_TAG_ALLGATHER; MPI_Status status; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; unsigned int rank = smpi_comm_rank(comm); unsigned int num_procs = smpi_comm_size(comm); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "allgather pair algorithm can't be used with non power of two number of processes ! "); extent = smpi_datatype_get_extent(send_type); // local send/recv smpi_mpi_sendrecv(send_ptr, send_count, send_type, rank, tag, recv_ptr + rank * recv_count * extent, recv_count, recv_type, rank, tag, comm, &status); for (i = 1; i < num_procs; i++) { src = dst = rank ^ i; smpi_mpi_sendrecv(send_ptr, send_count, send_type, dst, tag, recv_ptr + src * recv_count * extent, recv_count, recv_type, src, tag, comm, &status); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/barrier-ompi.c000644 001750 001750 00000025721 13031031271 022505 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2006 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * Copyright (c) 2008 Sun Microsystems, Inc. All rights reserved. * * Additional copyrights may follow */ #include "colls_private.h" #include "coll_tuned_topo.h" /* * Barrier is ment to be a synchronous operation, as some BTLs can mark * a request done before its passed to the NIC and progress might not be made * elsewhere we cannot allow a process to exit the barrier until its last * [round of] sends are completed. * * It is last round of sends rather than 'last' individual send as each pair of * peers can use different channels/devices/btls and the receiver of one of * these sends might be forced to wait as the sender * leaves the collective and does not make progress until the next mpi call * */ /* * Simple double ring version of barrier * * synchronous gurantee made by last ring of sends are synchronous * */ int smpi_coll_tuned_barrier_ompi_doublering(MPI_Comm comm ) { int rank, size; int left, right; rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); XBT_DEBUG("ompi_coll_tuned_barrier_ompi_doublering rank %d", rank); left = ((rank-1+size)%size); right = ((rank+1)%size); if (rank > 0) { /* receive message from the left */ smpi_mpi_recv((void*)NULL, 0, MPI_BYTE, left, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } /* Send message to the right */ smpi_mpi_send((void*)NULL, 0, MPI_BYTE, right, COLL_TAG_BARRIER, comm); /* root needs to receive from the last node */ if (rank == 0) { smpi_mpi_recv((void*)NULL, 0, MPI_BYTE, left, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } /* Allow nodes to exit */ if (rank > 0) { /* post Receive from left */ smpi_mpi_recv((void*)NULL, 0, MPI_BYTE, left, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } /* send message to the right one */ smpi_mpi_send((void*)NULL, 0, MPI_BYTE, right, COLL_TAG_BARRIER, comm); /* rank 0 post receive from the last node */ if (rank == 0) { smpi_mpi_recv((void*)NULL, 0, MPI_BYTE, left, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } return MPI_SUCCESS; } /* * To make synchronous, uses sync sends and sync sendrecvs */ int smpi_coll_tuned_barrier_ompi_recursivedoubling(MPI_Comm comm ) { int rank, size, adjsize; int mask, remote; rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); XBT_DEBUG( "ompi_coll_tuned_barrier_ompi_recursivedoubling rank %d", rank); /* do nearest power of 2 less than size calc */ for( adjsize = 1; adjsize <= size; adjsize <<= 1 ); adjsize >>= 1; /* if size is not exact power of two, perform an extra step */ if (adjsize != size) { if (rank >= adjsize) { /* send message to lower ranked node */ remote = rank - adjsize; smpi_mpi_sendrecv(NULL, 0, MPI_BYTE, remote, COLL_TAG_BARRIER, NULL, 0, MPI_BYTE, remote, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } else if (rank < (size - adjsize)) { /* receive message from high level rank */ smpi_mpi_recv((void*)NULL, 0, MPI_BYTE, rank+adjsize, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } } /* exchange messages */ if ( rank < adjsize ) { mask = 0x1; while ( mask < adjsize ) { remote = rank ^ mask; mask <<= 1; if (remote >= adjsize) continue; /* post receive from the remote node */ smpi_mpi_sendrecv(NULL, 0, MPI_BYTE, remote, COLL_TAG_BARRIER, NULL, 0, MPI_BYTE, remote, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } } /* non-power of 2 case */ if (adjsize != size) { if (rank < (size - adjsize)) { /* send enter message to higher ranked node */ remote = rank + adjsize; smpi_mpi_send((void*)NULL, 0, MPI_BYTE, remote, COLL_TAG_BARRIER, comm); } } return MPI_SUCCESS; } /* * To make synchronous, uses sync sends and sync sendrecvs */ int smpi_coll_tuned_barrier_ompi_bruck(MPI_Comm comm ) { int rank, size; int distance, to, from; rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); XBT_DEBUG( "ompi_coll_tuned_barrier_ompi_bruck rank %d", rank); /* exchange data with rank-2^k and rank+2^k */ for (distance = 1; distance < size; distance <<= 1) { from = (rank + size - distance) % size; to = (rank + distance) % size; /* send message to lower ranked node */ smpi_mpi_sendrecv(NULL, 0, MPI_BYTE, to, COLL_TAG_BARRIER, NULL, 0, MPI_BYTE, from, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } return MPI_SUCCESS; } /* * To make synchronous, uses sync sends and sync sendrecvs */ /* special case for two processes */ int smpi_coll_tuned_barrier_ompi_two_procs(MPI_Comm comm ) { int remote; remote = smpi_comm_rank(comm); XBT_DEBUG( "ompi_coll_tuned_barrier_ompi_two_procs rank %d", remote); remote = (remote + 1) & 0x1; smpi_mpi_sendrecv(NULL, 0, MPI_BYTE, remote, COLL_TAG_BARRIER, NULL, 0, MPI_BYTE, remote, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); return (MPI_SUCCESS); } /* * Linear functions are copied from the BASIC coll module * they do not segment the message and are simple implementations * but for some small number of nodes and/or small data sizes they * are just as fast as tuned/tree based segmenting operations * and as such may be selected by the decision functions * These are copied into this module due to the way we select modules * in V1. i.e. in V2 we will handle this differently and so will not * have to duplicate code. * GEF Oct05 after asking Jeff. */ /* copied function (with appropriate renaming) starts here */ int smpi_coll_tuned_barrier_ompi_basic_linear(MPI_Comm comm) { int i; int size = smpi_comm_size(comm); int rank = smpi_comm_rank(comm); /* All non-root send & receive zero-length message. */ if (rank > 0) { smpi_mpi_send (NULL, 0, MPI_BYTE, 0, COLL_TAG_BARRIER, comm); smpi_mpi_recv (NULL, 0, MPI_BYTE, 0, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } /* The root collects and broadcasts the messages. */ else { MPI_Request* requests; requests = (MPI_Request*)malloc( size * sizeof(MPI_Request) ); for (i = 1; i < size; ++i) { requests[i] = smpi_mpi_irecv(NULL, 0, MPI_BYTE, MPI_ANY_SOURCE, COLL_TAG_BARRIER, comm ); } smpi_mpi_waitall( size-1, requests+1, MPI_STATUSES_IGNORE ); for (i = 1; i < size; ++i) { requests[i] = smpi_mpi_isend(NULL, 0, MPI_BYTE, i, COLL_TAG_BARRIER, comm ); } smpi_mpi_waitall( size-1, requests+1, MPI_STATUSES_IGNORE ); free( requests ); } /* All done */ return MPI_SUCCESS; } /* copied function (with appropriate renaming) ends here */ /* * Another recursive doubling type algorithm, but in this case * we go up the tree and back down the tree. */ int smpi_coll_tuned_barrier_ompi_tree(MPI_Comm comm) { int rank, size, depth; int jump, partner; rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); XBT_DEBUG( "ompi_coll_tuned_barrier_ompi_tree %d", rank); /* Find the nearest power of 2 of the communicator size. */ for(depth = 1; depth < size; depth <<= 1 ); for (jump=1; jump rank) { smpi_mpi_recv (NULL, 0, MPI_BYTE, partner, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } else if (partner < rank) { smpi_mpi_send (NULL, 0, MPI_BYTE, partner, COLL_TAG_BARRIER, comm); } } } depth>>=1; for (jump = depth; jump>0; jump>>=1) { partner = rank ^ jump; if (!(partner & (jump-1)) && partner < size) { if (partner > rank) { smpi_mpi_send (NULL, 0, MPI_BYTE, partner, COLL_TAG_BARRIER, comm); } else if (partner < rank) { smpi_mpi_recv (NULL, 0, MPI_BYTE, partner, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } } } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/smpi_intel_mpi_selector.c000644 001750 001750 00000115053 13031031272 025024 0ustar00mquinsonmquinson000000 000000 /* selector for collective algorithms based on openmpi's default coll_tuned_decision_fixed selector */ /* Copyright (c) 2009-2010, 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" // This selector is based on information gathered on the Stampede cluster, with Intel MPI 4.1.3.049, and from the intel reference manual. The data was gathered launching runs with 1,2,4,8,16 processes per node. #define INTEL_MAX_NB_THRESHOLDS 32 #define INTEL_MAX_NB_NUMPROCS 12 #define INTEL_MAX_NB_PPN 5 /* 1 2 4 8 16 ppn */ typedef struct { int max_size; int algo; } intel_tuning_table_size_element; typedef struct { int max_num_proc; int num_elems; intel_tuning_table_size_element elems[INTEL_MAX_NB_THRESHOLDS]; } intel_tuning_table_numproc_element; typedef struct { int ppn; intel_tuning_table_numproc_element elems[INTEL_MAX_NB_NUMPROCS]; } intel_tuning_table_element; /* I_MPI_ADJUST_ALLREDUCE MPI_Allreduce 1 - Recursive doubling algorithm 2 - Rabenseifner's algorithm 3 - Reduce + Bcast algorithm 4 - Topology aware Reduce + Bcast algorithm 5 - Binomial gather + scatter algorithm 6 - Topology aware binominal gather + scatter algorithm 7 - Shumilin's ring algorithm 8 - Ring algorithm as Shumilin's ring algorithm is unknown, default to ring' */ int (*intel_allreduce_functions_table[])(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) ={ smpi_coll_tuned_allreduce_rdb, smpi_coll_tuned_allreduce_rab1, smpi_coll_tuned_allreduce_redbcast, smpi_coll_tuned_allreduce_mvapich2_two_level, smpi_coll_tuned_allreduce_smp_binomial, smpi_coll_tuned_allreduce_mvapich2_two_level, smpi_coll_tuned_allreduce_ompi_ring_segmented, smpi_coll_tuned_allreduce_ompi_ring_segmented }; intel_tuning_table_element intel_allreduce_table[] = { {1,{ { 2,9,{ {6,7}, {85,1}, {192,7}, {853,1}, {1279,7}, {16684,1}, {34279,8}, {1681224,3}, {2147483647,7} } }, { 4, 8,{ {16,7}, {47,1}, {2062,7}, {16699,1}, {33627,7}, {70732,8}, {1300705,3}, {2147483647,8} } }, {8,8,{ {118,1}, {146,4}, {16760,1}, {36364,6}, {136239,8}, {315710,7}, {3220366,3}, {2147483647,8} } }, {16,7,{ {934,1}, {1160,6}, {15505,1}, {52730,2}, {300705,8}, {563680,7}, {2147483647,3} } }, {2147483647,11,{ {5,6}, {11,4}, {182,1}, {700,6}, {1450,4}, {11146,1}, {25539,6}, {37634,4}, {93784,6}, {817658,2}, {2147483647,3} } } } }, {2,{ { 4,6,{ {2084,7}, {15216,1}, {99715,7}, {168666,3}, {363889,2}, {2147483647,7} } }, { 8,6,{ {14978,1}, {66879,2}, {179296,8}, {304801,3}, {704509,7}, {2147483647,2} } }, { 16,6,{ {16405,1}, {81784,2}, {346385,8}, {807546,7}, {1259854,2}, {2147483647,3} } }, { 32,4,{ {8913,1}, {103578,2}, {615876,8}, {2147483647,2} } }, { 64,7,{ {1000,1}, {2249,2}, {6029,1}, {325357,2}, {1470976,8}, {2556670,7}, {2147483647,3} } }, { 128,5,{ {664,1}, {754706,2}, {1663862,4}, {3269097,2}, {2147483647,7} } }, { 2147483647,3,{ {789,1}, {2247589,2}, {2147483647,8} } } } }, {4,{ { 4,4,{ {5738,1}, {197433,2}, {593742,7}, {2147483647,2} } }, { 8,7,{ {5655,1}, {75166,2}, {177639,8}, {988014,3}, {1643869,2}, {2494859,8}, {2147483647,2} } }, { 16,7,{ {587,1}, {3941,2}, {9003,1}, {101469,2}, {355768,8}, {3341814,3}, {2147483647,8} } }, { 32,4,{ {795,1}, {146567,2}, {732118,8}, {2147483647,3} } }, { 64,4,{ {528,1}, {221277,2}, {1440737,8}, {2147483647,3} } }, { 128,4,{ {481,1}, {593833,2}, {2962021,8}, {2147483647,7} } }, { 256,2,{ {584,1}, {2147483647,2} } }, { 2147483647,3,{ {604,1}, {2997006,2}, {2147483647,8} } } } }, {8,{ { 8,6,{ {2560,1}, {114230,6}, {288510,8}, {664038,2}, {1339913,6}, {2147483647,4} } }, { 16,5,{ {497,1}, {54201,2}, {356217,8}, {3413609,3}, {2147483647,8} } }, { 32,5,{ {377,1}, {109745,2}, {716514,8}, {3976768,3}, {2147483647,8} } }, { 64,6,{ {109,1}, {649,5}, {266080,2}, {1493331,8}, {2541403,7}, {2147483647,3} } }, { 128,4,{ {7,1}, {751,5}, {408808,2}, {2147483647,8} } }, { 256,3,{ {828,5}, {909676,2}, {2147483647,8} } }, { 512,5,{ {847,5}, {1007066,2}, {1068775,4}, {2803389,2}, {2147483647,8} } }, { 2147483647,3,{ {1974,5}, {4007876,2}, {2147483647,8} } } } }, {16,{ { 16,12,{ {409,1}, {768,6}, {1365,4}, {3071,6}, {11299,2}, {21746,6}, {55629,2}, {86065,4}, {153867,2}, {590560,6}, {1448760,2}, {2147483647,8}, } }, { 32,8,{ {6,1}, {24,5}, {86,1}, {875,5}, {74528,2}, {813050,8}, {1725981,7}, {2147483647,8}, } }, { 64,6,{ {1018,5}, {1217,6}, {2370,5}, {160654,2}, {1885487,8}, {2147483647,3}, } }, { 128,4,{ {2291,5}, {434465,2}, {3525103,8}, {2147483647,7}, } }, { 256,3,{ {2189,5}, {713154,2}, {2147483647,8}, } }, { 512,3,{ {2140,5}, {1235056,2}, {2147483647,8}, } }, { 2147483647,3,{ {2153,5}, {2629855,2}, {2147483647,8}, } } } } }; /*I_MPI_ADJUST_ALLTOALL MPI_Alltoall 1. Bruck's algorithm 2. Isend/Irecv + waitall algorithm 3. Pair wise exchange algorithm 4. Plum's algorithm */ intel_tuning_table_element intel_alltoall_table[] = { {1,{ { 2,1, { {2147483647,3} } }, { 4,2, { {0,4}, {2147483647,2} } }, {8,1, { {2147483647,2} } }, {16,5, { {0,3}, {84645,2}, {167570,3}, {413152,4}, {2147483647,2} } }, {32,6, { {61,1}, {164,2}, {696,1}, {143254,2}, {387024,3}, {2147483647,2} }, }, {64,4, { {523,1}, {146088,2}, {488989,4}, {2147483647,2} } }, {2147483647,3, { {270,1}, {628,4}, {2147483647,2} } } } }, {2, { { 4,4,{ {1,2}, {75,3}, {131072,2}, {2147483647,2} } }, { 8,3,{ {709,1}, {131072,2}, {2147483647,2} } }, { 16,4,{ {40048,2}, {131072,3}, {155927,3}, {2147483647,4} } }, { 32,7,{ {105,1}, {130,2}, {1030,1}, {58893,2}, {131072,2}, {271838,3}, {2147483647,2} } }, { 2147483647,8,{ {521,1}, {2032,4}, {2412,2}, {4112,4}, {61620,2}, {131072,3}, {427408,3}, {2147483647,4} } } } }, {4,{ { 8,3,{ {512,1}, {32768,2}, {2147483647,2} } }, { 64,8,{ {7,1}, {199,4}, {764,1}, {6409,4}, {20026,2}, {32768,3}, {221643,4}, {2147483647,3} } }, { 2147483647,7,{ {262,1}, {7592,4}, {22871,2}, {32768,3}, {47538,3}, {101559,4}, {2147483647,3} } } } }, {8,{ { 16,6,{ {973,1}, {5126,4}, {16898,2}, {32768,4}, {65456,4}, {2147483647,2} } }, { 32,7,{ {874,1}, {6727,4}, {17912,2}, {32768,3}, {41513,3}, {199604,4}, {2147483647,3} } }, { 64,8,{ {5,1}, {114,4}, {552,1}, {8130,4}, {32768,3}, {34486,3}, {160113,4}, {2147483647,3} } }, { 128,6,{ {270,1}, {3679,4}, {32768,3}, {64367,3}, {146595,4}, {2147483647,3} } }, { 2147483647,4,{ {133,1}, {4017,4}, {32768,3}, {76351,4}, {2147483647,3} } } } }, {16,{ { 32,7,{ {963,1}, {1818,4}, {20007,2}, {32768,4}, {54296,4}, {169735,3}, {2147483647,2} } }, { 64,11,{ {17,1}, {42,4}, {592,1}, {2015,4}, {2753,2}, {6496,3}, {20402,4}, {32768,3}, {36246,3}, {93229,4}, {2147483647,3} } }, { 128,9,{ {18,1}, {40,4}, {287,1}, {1308,4}, {6842,1}, {32768,3}, {36986,3}, {129081,4}, {2147483647,3} } }, { 256,7,{ {135,1}, {1538,4}, {3267,1}, {4132,3}, {31469,4}, {32768,3}, {2147483647,3} } }, { 2147483647,8,{ {66,1}, {1637,4}, {2626,1}, {4842,4}, {32768,3}, {33963,3}, {72978,4}, {2147483647,3} } } } } }; int (*intel_alltoall_functions_table[])(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, MPI_Comm comm) ={ smpi_coll_tuned_alltoall_bruck, smpi_coll_tuned_alltoall_mvapich2_scatter_dest, smpi_coll_tuned_alltoall_pair, smpi_coll_tuned_alltoall_mvapich2//Plum is proprietary ? (and super efficient) }; /*I_MPI_ADJUST_BARRIER MPI_Barrier 1. Dissemination algorithm 2. Recursive doubling algorithm 3. Topology aware dissemination algorithm 4. Topology aware recursive doubling algorithm 5. Binominal gather + scatter algorithm 6. Topology aware binominal gather + scatter algorithm */ static int intel_barrier_gather_scatter(MPI_Comm comm){ //our default barrier performs a antibcast/bcast smpi_mpi_barrier(comm); return MPI_SUCCESS; } int (*intel_barrier_functions_table[])(MPI_Comm comm) ={ smpi_coll_tuned_barrier_ompi_basic_linear, smpi_coll_tuned_barrier_ompi_recursivedoubling, smpi_coll_tuned_barrier_ompi_basic_linear, smpi_coll_tuned_barrier_ompi_recursivedoubling, intel_barrier_gather_scatter, intel_barrier_gather_scatter }; intel_tuning_table_element intel_barrier_table[] = { {1,{ {2,1, { {2147483647,2} } }, {4,1, { {2147483647,6} } }, {8,1, { {2147483647,1} } }, {64,1, { {2147483647,2} } }, {2147483647,1, { {2147483647,6} } } } }, {2,{ { 2,1,{ {2147483647,1} } }, { 4,1,{ {2147483647,3} } }, { 8,1,{ {2147483647,5} } }, { 32,1,{ {2147483647,2} } }, { 128,1,{ {2147483647,3} } }, { 2147483647,1,{ {2147483647,4} } } } }, {4,{ { 4,1,{ {2147483647,2} } }, { 8,1,{ {2147483647,5} } }, { 32,1,{ {2147483647,2} } }, { 2147483647,1,{ {2147483647,4} } } } }, {8,{ { 8,1,{ {2147483647,1} } }, { 32,1,{ {2147483647,2} } }, { 2147483647,1,{ {2147483647,4} } } } }, {16,{ { 4,1,{ {2147483647,2} } }, { 8,1,{ {2147483647,5} } }, { 32,1,{ {2147483647,2} } }, { 2147483647,1,{ {2147483647,4} } } } } }; /*I_MPI_ADJUST_BCAST MPI_Bcast 1. Binomial algorithm 2. Recursive doubling algorithm 3. Ring algorithm 4. Topology aware binomial algorithm 5. Topology aware recursive doubling algorithm 6. Topology aware ring algorithm 7. Shumilin's bcast algorithm */ int (*intel_bcast_functions_table[])(void *buff, int count, MPI_Datatype datatype, int root, MPI_Comm comm) ={ smpi_coll_tuned_bcast_binomial_tree, //smpi_coll_tuned_bcast_scatter_rdb_allgather, smpi_coll_tuned_bcast_NTSL, smpi_coll_tuned_bcast_NTSL, smpi_coll_tuned_bcast_SMP_binomial, //smpi_coll_tuned_bcast_scatter_rdb_allgather, smpi_coll_tuned_bcast_NTSL, smpi_coll_tuned_bcast_SMP_linear, smpi_coll_tuned_bcast_mvapich2,//we don't know shumilin's algo' }; intel_tuning_table_element intel_bcast_table[] = { {1,{ {2,9, { {1,2}, {402,7}, {682,5}, {1433,4}, {5734,7}, {21845,1}, {95963,6}, {409897,5}, {2147483647,1} } }, {4,1, { {2147483647,7} } }, {8,11, { {3,6}, {4,7}, {25,6}, {256,1}, {682,6}, {1264,1}, {2234,6}, {6655,5}, {16336,1}, {3998434,7}, {2147483647,6} } }, {2147483647,1, { {2147483647,7} } } } }, {2,{ { 4,6,{ {806,4}, {18093,7}, {51366,6}, {182526,4}, {618390,1}, {2147483647,7} } }, { 8,6,{ {24,1}, {74,4}, {18137,1}, {614661,7}, {1284626,1}, {2147483647,2} } }, { 16,4,{ {1,1}, {158,7}, {16955,1}, {2147483647,7} } }, { 32,3,{ {242,7}, {10345,1}, {2147483647,7} } }, { 2147483647,4,{ {1,1}, {737,7}, {5340,1}, {2147483647,7} } } } }, {4,{ { 8,4,{ {256,4}, {17181,1}, {1048576,7}, {2147483647,7} } }, { 2147483647,1,{ {2147483647,7} } } } }, {8,{ { 16,5,{ {3,1}, {318,7}, {1505,1}, {1048576,7}, {2147483647,7} } }, { 32,3,{ {422,7}, {851,1}, {2147483647,7} } }, { 64,3,{ {468,7}, {699,1}, {2147483647,7} } }, { 2147483647,1,{ {2147483647,7} } } } }, {16,{ { 8,4,{ {256,4}, {17181,1}, {1048576,7}, {2147483647,7} } }, { 2147483647,1,{ {2147483647,7} } } } } }; /*I_MPI_ADJUST_REDUCE MPI_Reduce 1. Shumilin's algorithm 2. Binomial algorithm 3. Topology aware Shumilin's algorithm 4. Topology aware binomial algorithm 5. Rabenseifner's algorithm 6. Topology aware Rabenseifner's algorithm */ int (*intel_reduce_functions_table[])(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) ={ smpi_coll_tuned_reduce_mvapich2, smpi_coll_tuned_reduce_binomial, smpi_coll_tuned_reduce_mvapich2, smpi_coll_tuned_reduce_mvapich2_two_level, smpi_coll_tuned_reduce_rab, smpi_coll_tuned_reduce_rab }; intel_tuning_table_element intel_reduce_table[] = { {1,{ {2147483647,1, { {2147483647,1} } } } }, {2,{ { 2,1,{ {2147483647,1} } }, { 4,2,{ {10541,3}, {2147483647,1} } }, { 2147483647,1,{ {2147483647,1} } } } }, {4,{ { 256,1,{ {2147483647,1} } }, { 2147483647,2,{ {45,3}, {2147483647,1} } } } }, {8,{ { 512,1,{ {2147483647,1} } }, { 2147483647,3,{ {5,1}, {11882,3}, {2147483647,1} } } } }, {16,{ { 256,1,{ {2147483647,1} } }, { 2147483647,2,{ {45,3}, {2147483647,1} } } } } }; /* I_MPI_ADJUST_REDUCE_SCATTER MPI_Reduce_scatter 1. Recursive having algorithm 2. Pair wise exchange algorithm 3. Recursive doubling algorithm 4. Reduce + Scatterv algorithm 5. Topology aware Reduce + Scatterv algorithm */ static int intel_reduce_scatter_reduce_scatterv(void *sbuf, void *rbuf, int *rcounts, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { smpi_mpi_reduce_scatter(sbuf, rbuf, rcounts,dtype, op,comm); return MPI_SUCCESS; } static int intel_reduce_scatter_recursivehalving(void *sbuf, void *rbuf, int *rcounts, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { if(smpi_op_is_commute(op)) return smpi_coll_tuned_reduce_scatter_ompi_basic_recursivehalving(sbuf, rbuf, rcounts,dtype, op,comm); else return smpi_coll_tuned_reduce_scatter_mvapich2(sbuf, rbuf, rcounts,dtype, op,comm); } int (*intel_reduce_scatter_functions_table[])( void *sbuf, void *rbuf, int *rcounts, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm ) ={ intel_reduce_scatter_recursivehalving, smpi_coll_tuned_reduce_scatter_mpich_pair, smpi_coll_tuned_reduce_scatter_mpich_rdb, intel_reduce_scatter_reduce_scatterv, intel_reduce_scatter_reduce_scatterv }; intel_tuning_table_element intel_reduce_scatter_table[] = { {1,{ {2,5, { {5,4}, {522429,2}, {1375877,5}, {2932736,2}, {2147483647,5} } }, {4,9, { {4,4}, {15,1}, {120,3}, {651,1}, {12188,3}, {33890,1}, {572117,2}, {1410202,5}, {2147483647,2} } }, {8,7, { {4,4}, {2263,1}, {25007,3}, {34861,1}, {169625,2}, {2734000,4}, {2147483647,2} } }, {16,5, { {4,4}, {14228,1}, {46084,3}, {522139,2}, {2147483647,5} } }, {32,5, { {4,4}, {27516,1}, {61693,3}, {2483469,2}, {2147483647,5} } }, {64,4, { {0,3}, {4,4}, {100396,1}, {2147483647,2} } }, {2147483647,6, { {0,3}, {4,4}, {186926,1}, {278259,3}, {1500100,2}, {2147483647,5} } } } }, {2,{ { 2,2,{ {6,1}, {2147483647,2} } }, { 4,7,{ {5,4}, {13,5}, {59,3}, {76,1}, {91488,3}, {680063,4}, {2147483647,2} } }, { 8,8,{ {4,4}, {11,5}, {31,1}, {69615,3}, {202632,2}, {396082,5}, {1495696,4}, {2147483647,2} } }, { 16,1,{ {4,4}, {345,1}, {79523,3}, {2147483647,2} } }, { 32,5,{ {0,3}, {4,4}, {992,1}, {71417,3}, {2147483647,2} } }, { 64,4,{ {4,4}, {1472,1}, {196592,3}, {2147483647,2} } }, { 128,5,{ {0,3}, {4,4}, {32892,1}, {381072,3}, {2147483647,2} } }, { 2147483647,6,{ {0,2}, {4,4}, {33262,1}, {1571397,3}, {2211398,5}, {2147483647,4} } } } }, {4,{ { 4,7,{ {12,4}, {27,5}, {49,3}, {187,1}, {405673,3}, {594687,4}, {2147483647,2} } }, { 8,5,{ {24,5}, {155,1}, {204501,3}, {274267,5}, {2147483647,4} } }, { 16,6,{ {63,1}, {72,3}, {264,1}, {168421,3}, {168421,4}, {2147483647,2} } }, { 32,10,{ {0,3}, {4,4}, {12,1}, {18,5}, {419,1}, {188739,3}, {716329,4}, {1365841,5}, {2430194,2}, {2147483647,4} } }, { 64,8,{ {0,3}, {4,4}, {17,5}, {635,1}, {202937,3}, {308253,5}, {1389874,4}, {2147483647,2} } }, { 128,8,{ {0,3}, {4,4}, {16,5}, {1238,1}, {280097,3}, {631434,5}, {2605072,4}, {2147483647,2} } }, { 256,5,{ {0,2}, {4,4}, {16,5}, {2418,1}, {2147483647,3} } }, { 2147483647,6,{ {0,2}, {4,4}, {16,5}, {33182,1}, {3763779,3}, {2147483647,4} } } } }, {8,{ { 8,6,{ {5,4}, {494,1}, {97739,3}, {522836,2}, {554174,5}, {2147483647,2} } }, { 16,8,{ {5,4}, {62,1}, {94,3}, {215,1}, {185095,3}, {454784,4}, {607911,5}, {2147483647,4} } }, { 32,7,{ {0,3}, {4,4}, {302,1}, {250841,3}, {665822,4}, {1760980,5}, {2147483647,4} } }, { 64,8,{ {0,3}, {4,4}, {41,5}, {306,1}, {332405,3}, {1269189,4}, {3712421,5}, {2147483647,4} } }, { 128,6,{ {0,3}, {4,4}, {39,5}, {526,1}, {487878,3}, {2147483647,4} } }, { 256,8,{ {0,2}, {4,4}, {36,5}, {1382,1}, {424162,3}, {632881,5}, {1127566,3}, {2147483647,4} } }, { 512,4,{ {4,4}, {34,5}, {5884,1}, {2147483647,3} } }, { 2147483647,4,{ {5,4}, {32,5}, {25105,1}, {2147483647,3} } } } }, {16,{ { 4,7,{ {12,4}, {27,5}, {49,3}, {187,1}, {405673,3}, {594687,4}, {2147483647,2} } }, { 8,5,{ {24,5}, {155,1}, {204501,3}, {274267,5}, {2147483647,4} } }, { 16,6,{ {63,1}, {72,3}, {264,1}, {168421,3}, {168421,4}, {2147483647,2} } }, { 32,10,{ {0,3}, {4,4}, {12,1}, {18,5}, {419,1}, {188739,3}, {716329,4}, {1365841,5}, {2430194,2}, {2147483647,4} } }, { 64,8,{ {0,3}, {4,4}, {17,5}, {635,1}, {202937,3}, {308253,5}, {1389874,4}, {2147483647,2} } }, { 128,8,{ {0,3}, {4,4}, {16,5}, {1238,1}, {280097,3}, {631434,5}, {2605072,4}, {2147483647,2} } }, { 256,5,{ {0,2}, {4,4}, {16,5}, {2418,1}, {2147483647,3} } }, { 2147483647,6,{ {0,2}, {4,4}, {16,5}, {33182,1}, {3763779,3}, {2147483647,4} } } } } }; /* I_MPI_ADJUST_ALLGATHER MPI_Allgather 1. Recursive doubling algorithm 2. Bruck's algorithm 3. Ring algorithm 4. Topology aware Gatherv + Bcast algorithm */ int (*intel_allgather_functions_table[])(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, MPI_Comm comm ) ={ smpi_coll_tuned_allgather_rdb, smpi_coll_tuned_allgather_bruck, smpi_coll_tuned_allgather_ring, smpi_coll_tuned_allgather_GB }; intel_tuning_table_element intel_allgather_table[] = { {1,{ {4,11, { {1,4}, {384,1}, {1533,4}, {3296,1}, {10763,4}, {31816,3}, {193343,4}, {405857,3}, {597626,4}, {1844323,3}, {2147483647,4} } }, {8,10, { {12,4}, {46,1}, {205,4}, {3422,2}, {4200,4}, {8748,1}, {24080,3}, {33244,4}, {371159,1}, {2147483647,3} } }, {16, 8, { {3,4}, {53,1}, {100,4}, {170,1}, {6077,4}, {127644,1}, {143741,4}, {2147483647,3} } }, {2147483647,10, { {184,1}, {320,4}, {759,1}, {1219,4}, {2633,1}, {8259,4}, {123678,1}, {160801,4}, {284341,1}, {2147483647,4} } } } }, {2,{ { 8,6,{ {490,1}, {558,2}, {2319,1}, {46227,3}, {2215101,1}, {2147483647,3} } }, { 16,4,{ {1005,1}, {1042,2}, {2059,1}, {2147483647,3} } }, { 2147483647,2,{ {2454,1}, {2147483647,3} } } } }, {4,{ { 8,2,{ {2861,1}, {2147483647,3} } }, { 2147483647,2,{ {605,1}, {2147483647,3} } } } }, {8,{ { 16,4,{ {66,1}, {213,4}, {514,1}, {2147483647,3} } }, { 32,4,{ {91,1}, {213,4}, {514,1}, {2147483647,3} } }, { 64,4,{ {71,1}, {213,4}, {514,1}, {2147483647,3} } }, { 128,2,{ {305,1}, {2147483647,3} } }, { 2147483647,2,{ {213,1}, {2147483647,3} } } } }, {16,{ { 8,2,{ {2861,1}, {2147483647,3} } }, { 2147483647,2,{ {605,1}, {2147483647,3} } } } } }; /* I_MPI_ADJUST_ALLGATHERV MPI_Allgatherv 1. Recursive doubling algorithm 2. Bruck's algorithm 3. Ring algorithm 4. Topology aware Gatherv + Bcast algorithm */ int (*intel_allgatherv_functions_table[])(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int *rcounts, int *rdispls, MPI_Datatype rdtype, MPI_Comm comm ) ={ smpi_coll_tuned_allgatherv_mpich_rdb, smpi_coll_tuned_allgatherv_ompi_bruck, smpi_coll_tuned_allgatherv_ring, smpi_coll_tuned_allgatherv_GB }; intel_tuning_table_element intel_allgatherv_table[] = { {1,{ {2,3, { {259668,3}, {635750,4}, {2147483647,3} } }, {4,7, { {1,1}, {5,4}, {46,1}, {2590,2}, {1177259,3}, {2767234,4}, {2147483647,3} } }, {8, 6, { {99,2}, {143,1}, {4646,2}, {63522,3}, {2187806,4}, {2147483647,3} } }, {2147483647,7, { {1,1}, {5,4}, {46,1}, {2590,2}, {1177259,3}, {2767234,4}, {2147483647,3} } } } }, {2,{ { 4,3,{ {3147,1}, {5622,2}, {2147483647,3} } }, { 8,3,{ {975,1}, {4158,2}, {2147483647,3} } }, { 16,2,{ {2146,1}, {2147483647,3} } }, { 32,4,{ {81,1}, {414,2}, {1190,1}, {2147483647,3} } }, { 2147483647,5,{ {1,2}, {3,1}, {783,2}, {1782,4}, {2147483647,3} } } } }, {4,{ { 8,2,{ {2554,1}, {2147483647,3} } }, { 16,4,{ {272,1}, {657,2}, {2078,1}, {2147483647,3} } }, { 32,2,{ {1081,1}, {2147483647,3} } }, { 64,2,{ {547,1}, {2147483647,3} } }, { 2147483647,5,{ {19,1}, {239,2}, {327,1}, {821,4}, {2147483647,3} } } } }, {8,{ { 16,3,{ {55,1}, {514,2}, {2147483647,3} } }, { 32,4,{ {53,1}, {167,4}, {514,2}, {2147483647,3} } }, { 64,3,{ {13,1}, {319,4}, {2147483647,3} } }, { 128,7,{ {2,1}, {11,2}, {48,1}, {201,2}, {304,1}, {1048,4}, {2147483647,3} } }, { 2147483647,5,{ {5,1}, {115,4}, {129,1}, {451,4}, {2147483647,3} } } } }, {16,{ { 8,2,{ {2554,1}, {2147483647,3} } }, { 16,4,{ {272,1}, {657,2}, {2078,1}, {2147483647,3} } }, { 32,2,{ {1081,1}, {2147483647,3} } }, { 64,2,{ {547,1}, {2147483647,3} } }, { 2147483647,5,{ {19,1}, {239,2}, {327,1}, {821,4}, {2147483647,3} } } } } }; /* I_MPI_ADJUST_GATHER MPI_Gather 1. Binomial algorithm 2. Topology aware binomial algorithm 3. Shumilin's algorithm */ int (*intel_gather_functions_table[])(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, int root, MPI_Comm comm ) ={ smpi_coll_tuned_gather_ompi_binomial, smpi_coll_tuned_gather_ompi_binomial, smpi_coll_tuned_gather_mvapich2 }; intel_tuning_table_element intel_gather_table[] = { {1,{ {8,3, { {17561,3}, {44791,2}, {2147483647,3} } }, {16,7, { {16932,3}, {84425,2}, {158363,3}, {702801,2}, {1341444,3}, {2413569,2}, {2147483647,3} } }, {2147483647,4, { {47187,3}, {349696,2}, {2147483647,3}, {2147483647,1} } } } }, {2,{ {2147483647,1,{ {2147483647,3} } } } }, {4,{ {2147483647,1,{ {2147483647,3} } } } }, {8,{ { 16,1,{ {2147483647,3} } }, { 32,2,{ {9,2}, {2147483647,3} } }, { 64,2,{ {784,2}, {2147483647,3} } }, { 128,3,{ {160,3}, {655,2}, {2147483647,3} } }, { 2147483647,1,{ {2147483647,3} } } } }, {16,{ {2147483647,1,{ {2147483647,3} } } } } }; /* I_MPI_ADJUST_SCATTER MPI_Scatter 1. Binomial algorithm 2. Topology aware binomial algorithm 3. Shumilin's algorithm */ int (*intel_scatter_functions_table[])(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, int root, MPI_Comm comm ) ={ smpi_coll_tuned_scatter_ompi_binomial, smpi_coll_tuned_scatter_ompi_binomial, smpi_coll_tuned_scatter_mvapich2 }; intel_tuning_table_element intel_scatter_table[] = { {1,{ {2,2, { {16391,1}, {2147483647,3} } }, {4,6, { {16723,3}, {153541,2}, {425631,3}, {794142,2}, {1257027,3}, {2147483647,2} } }, {8,7, { {2633,3}, {6144,2}, {14043,3}, {24576,2}, {107995,3}, {1752729,2}, {2147483647,3} } }, {16,7, { {2043,3}, {2252,2}, {17749,3}, {106020,2}, {628654,3}, {3751354,2}, {2147483647,3} } }, {2147483647,4, { {65907,3}, {245132,2}, {1042439,3}, {2147483647,2}, {2147483647,1} } } } }, {2,{ {2147483647,1,{ {2147483647,3} } } } }, {4,{ { 8,1,{ {2147483647,3} } }, { 16,2,{ {140,3}, {1302,1}, {2147483647,3} } }, { 32,2,{ {159,3}, {486,1}, {2147483647,3} } }, { 64,2,{ {149,1}, {2147483647,3} } }, { 2147483647,2,{ {139,1}, {2147483647,3} } } } }, {8,{ { 16,4,{ {587,1}, {1370,2}, {2102,1}, {2147483647,3} } }, { 32,3,{ {1038,1}, {2065,2}, {2147483647,3} } }, { 64,3,{ {515,1}, {2069,2}, {2147483647,3} } }, { 128,3,{ {284,1}, {796,2}, {2147483647,3} } }, { 2147483647,2,{ {139,1}, {2147483647,3} } } } }, {16,{ { 8,1,{ {2147483647,3} } }, { 16,3,{ {140,3}, {1302,1}, {2147483647,3} } }, { 32,3,{ {159,3}, {486,1}, {2147483647,3} } }, { 64,2,{ {149,1}, {2147483647,3} } }, { 2147483647,2,{ {139,1}, {2147483647,3} } } } } }; /* I_MPI_ADJUST_ALLTOALLV MPI_Alltoallv 1. Isend/Irecv + waitall algorithm 2. Plum's algorithm */ int (*intel_alltoallv_functions_table[])(void *sbuf, int *scounts, int *sdisps, MPI_Datatype sdtype, void *rbuf, int *rcounts, int *rdisps, MPI_Datatype rdtype, MPI_Comm comm ) ={ smpi_coll_tuned_alltoallv_ompi_basic_linear, smpi_coll_tuned_alltoallv_bruck }; intel_tuning_table_element intel_alltoallv_table[] = { {1,{ {2147483647,1, { {2147483647,1} } } } }, {2,{ {2147483647,1, { {2147483647,1} } } } }, {4,{ { 8,1,{ {2147483647,1}//weirdly, intel reports the use of algo 0 here } }, { 2147483647,2,{ {4,1},//0 again {2147483647,2} } } } }, {8,{ { 16,1,{ {2147483647,1} } }, { 2147483647,2,{ {0,1},//weird again, only for 0-sized messages {2147483647,2} } } } }, {16,{ { 8,1,{ {2147483647,1}//0 } }, { 2147483647,2,{ {4,1},//0 {2147483647,2} } } } } }; //These are collected from table 3.5-2 of the Intel MPI Reference Manual #define SIZECOMP_reduce_scatter\ int total_message_size = 0;\ for (i = 0; i < comm_size; i++) { \ total_message_size += rcounts[i];\ }\ size_t block_dsize = total_message_size*smpi_datatype_size(dtype);\ #define SIZECOMP_allreduce\ size_t block_dsize =rcount * smpi_datatype_size(dtype); #define SIZECOMP_alltoall\ size_t block_dsize =send_count * smpi_datatype_size(send_type); #define SIZECOMP_bcast\ size_t block_dsize =count * smpi_datatype_size(datatype); #define SIZECOMP_reduce\ size_t block_dsize =count * smpi_datatype_size(datatype); #define SIZECOMP_barrier\ size_t block_dsize = 1; #define SIZECOMP_allgather\ size_t block_dsize =recv_count * smpi_datatype_size(recv_type); #define SIZECOMP_allgatherv\ int total_message_size = 0;\ for (i = 0; i < comm_size; i++) { \ total_message_size += recv_count[i];\ }\ size_t block_dsize = total_message_size*smpi_datatype_size(recv_type); #define SIZECOMP_gather\ int rank = smpi_comm_rank(comm);\ size_t block_dsize = (send_buff == MPI_IN_PLACE || rank ==root) ?\ recv_count * smpi_datatype_size(recv_type) :\ send_count * smpi_datatype_size(send_type); #define SIZECOMP_scatter\ int rank = smpi_comm_rank(comm);\ size_t block_dsize = (sendbuf == MPI_IN_PLACE || rank !=root ) ?\ recvcount * smpi_datatype_size(recvtype) :\ sendcount * smpi_datatype_size(sendtype); #define SIZECOMP_alltoallv\ size_t block_dsize = 1; #define IMPI_COLL_SELECT(cat, ret, args, args2)\ ret smpi_coll_tuned_ ## cat ## _impi (COLL_UNPAREN args)\ {\ int comm_size = smpi_comm_size(comm);\ int i =0;\ SIZECOMP_ ## cat\ i=0;\ int j =0, k=0;\ if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){\ smpi_comm_init_smp(comm);\ }\ int local_size=1;\ if (smpi_comm_is_uniform(comm)) {\ local_size = smpi_comm_size(smpi_comm_get_intra_comm(comm));\ }\ while(i < INTEL_MAX_NB_PPN &&\ local_size!=intel_ ## cat ## _table[i].ppn)\ i++;\ if(i==INTEL_MAX_NB_PPN) i=0;\ while(comm_size>intel_ ## cat ## _table[i].elems[j].max_num_proc\ && j < INTEL_MAX_NB_THRESHOLDS)\ j++;\ while(block_dsize >=intel_ ## cat ## _table[i].elems[j].elems[k].max_size\ && k< intel_ ## cat ## _table[i].elems[j].num_elems)\ k++;\ return (intel_ ## cat ## _functions_table[intel_ ## cat ## _table[i].elems[j].elems[k].algo-1]\ args2);\ } COLL_APPLY(IMPI_COLL_SELECT, COLL_ALLGATHERV_SIG, (send_buff, send_count, send_type, recv_buff, recv_count, recv_disps, recv_type, comm)); COLL_APPLY(IMPI_COLL_SELECT, COLL_ALLREDUCE_SIG, (sbuf, rbuf, rcount, dtype, op, comm)); COLL_APPLY(IMPI_COLL_SELECT, COLL_GATHER_SIG, (send_buff, send_count, send_type, recv_buff, recv_count, recv_type, root, comm)); COLL_APPLY(IMPI_COLL_SELECT, COLL_ALLGATHER_SIG, (send_buff,send_count,send_type,recv_buff,recv_count,recv_type,comm)); COLL_APPLY(IMPI_COLL_SELECT, COLL_ALLTOALL_SIG,(send_buff, send_count, send_type, recv_buff, recv_count, recv_type,comm)); COLL_APPLY(IMPI_COLL_SELECT, COLL_ALLTOALLV_SIG, (send_buff, send_counts, send_disps, send_type, recv_buff, recv_counts, recv_disps, recv_type, comm)); COLL_APPLY(IMPI_COLL_SELECT, COLL_BCAST_SIG , (buf, count, datatype, root, comm)); COLL_APPLY(IMPI_COLL_SELECT, COLL_REDUCE_SIG,(buf,rbuf, count, datatype, op, root, comm)); COLL_APPLY(IMPI_COLL_SELECT, COLL_REDUCE_SCATTER_SIG ,(sbuf,rbuf, rcounts,dtype,op,comm)); COLL_APPLY(IMPI_COLL_SELECT, COLL_SCATTER_SIG ,(sendbuf, sendcount, sendtype,recvbuf, recvcount, recvtype,root, comm)); COLL_APPLY(IMPI_COLL_SELECT, COLL_BARRIER_SIG,(comm)); SimGrid-3.14.159/src/smpi/colls/bcast-SMP-linear.c000644 001750 001750 00000014771 13031031271 023121 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" int bcast_SMP_linear_segment_byte = 8192; int smpi_coll_tuned_bcast_SMP_linear(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int tag = COLL_TAG_BCAST; MPI_Status status; MPI_Request request; MPI_Request *request_array; MPI_Status *status_array; int rank, size; int i; MPI_Aint extent; extent = smpi_datatype_get_extent(datatype); rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } int num_core=1; if (smpi_comm_is_uniform(comm)){ num_core = smpi_comm_size(smpi_comm_get_intra_comm(comm)); }else{ //implementation buggy in this case return smpi_coll_tuned_bcast_mpich( buf , count, datatype, root, comm); } int segment = bcast_SMP_linear_segment_byte / extent; segment = segment == 0 ? 1 :segment; int pipe_length = count / segment; int remainder = count % segment; int increment = segment * extent; /* leader of each SMP do inter-communication and act as a root for intra-communication */ int to_inter = (rank + num_core) % size; int to_intra = (rank + 1) % size; int from_inter = (rank - num_core + size) % size; int from_intra = (rank + size - 1) % size; // call native when MPI communication size is too small if (size <= num_core) { XBT_WARN("MPI_bcast_SMP_linear use default MPI_bcast."); smpi_mpi_bcast(buf, count, datatype, root, comm); return MPI_SUCCESS; } // if root is not zero send to rank zero first if (root != 0) { if (rank == root) smpi_mpi_send(buf, count, datatype, 0, tag, comm); else if (rank == 0) smpi_mpi_recv(buf, count, datatype, root, tag, comm, &status); } // when a message is smaller than a block size => no pipeline if (count <= segment) { // case ROOT if (rank == 0) { smpi_mpi_send(buf, count, datatype, to_inter, tag, comm); smpi_mpi_send(buf, count, datatype, to_intra, tag, comm); } // case last ROOT of each SMP else if (rank == (((size - 1) / num_core) * num_core)) { request = smpi_mpi_irecv(buf, count, datatype, from_inter, tag, comm); smpi_mpi_wait(&request, &status); smpi_mpi_send(buf, count, datatype, to_intra, tag, comm); } // case intermediate ROOT of each SMP else if (rank % num_core == 0) { request = smpi_mpi_irecv(buf, count, datatype, from_inter, tag, comm); smpi_mpi_wait(&request, &status); smpi_mpi_send(buf, count, datatype, to_inter, tag, comm); smpi_mpi_send(buf, count, datatype, to_intra, tag, comm); } // case last non-ROOT of each SMP else if (((rank + 1) % num_core == 0) || (rank == (size - 1))) { request = smpi_mpi_irecv(buf, count, datatype, from_intra, tag, comm); smpi_mpi_wait(&request, &status); } // case intermediate non-ROOT of each SMP else { request = smpi_mpi_irecv(buf, count, datatype, from_intra, tag, comm); smpi_mpi_wait(&request, &status); smpi_mpi_send(buf, count, datatype, to_intra, tag, comm); } return MPI_SUCCESS; } // pipeline bcast else { request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); // case ROOT of each SMP if (rank % num_core == 0) { // case real root if (rank == 0) { for (i = 0; i < pipe_length; i++) { smpi_mpi_send((char *) buf + (i * increment), segment, datatype, to_inter, (tag + i), comm); smpi_mpi_send((char *) buf + (i * increment), segment, datatype, to_intra, (tag + i), comm); } } // case last ROOT of each SMP else if (rank == (((size - 1) / num_core) * num_core)) { for (i = 0; i < pipe_length; i++) { request_array[i] = smpi_mpi_irecv((char *) buf + (i * increment), segment, datatype, from_inter, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { smpi_mpi_wait(&request_array[i], &status); smpi_mpi_send((char *) buf + (i * increment), segment, datatype, to_intra, (tag + i), comm); } } // case intermediate ROOT of each SMP else { for (i = 0; i < pipe_length; i++) { request_array[i] = smpi_mpi_irecv((char *) buf + (i * increment), segment, datatype, from_inter, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { smpi_mpi_wait(&request_array[i], &status); smpi_mpi_send((char *) buf + (i * increment), segment, datatype, to_inter, (tag + i), comm); smpi_mpi_send((char *) buf + (i * increment), segment, datatype, to_intra, (tag + i), comm); } } } else { // case last non-ROOT of each SMP if (((rank + 1) % num_core == 0) || (rank == (size - 1))) { for (i = 0; i < pipe_length; i++) { request_array[i] = smpi_mpi_irecv((char *) buf + (i * increment), segment, datatype, from_intra, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { smpi_mpi_wait(&request_array[i], &status); } } // case intermediate non-ROOT of each SMP else { for (i = 0; i < pipe_length; i++) { request_array[i] = smpi_mpi_irecv((char *) buf + (i * increment), segment, datatype, from_intra, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { smpi_mpi_wait(&request_array[i], &status); smpi_mpi_send((char *) buf + (i * increment), segment, datatype, to_intra, (tag + i), comm); } } } free(request_array); free(status_array); } // when count is not divisible by block size, use default BCAST for the remainder if ((remainder != 0) && (count > segment)) { XBT_WARN("MPI_bcast_SMP_linear use default MPI_bcast."); smpi_mpi_bcast((char *) buf + (pipe_length * increment), remainder, datatype, root, comm); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allreduce-smp-binomial.c000644 001750 001750 00000011620 13031031271 024433 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /* IMPLEMENTED BY PITCH PATARASUK Non-topoloty-specific (however, number of cores/node need to be changed) all-reduce operation designed for smp clusters It uses 2-layer communication: binomial for both intra-communication inter-communication*/ /* ** NOTE ** Use -DMPICH2 if this code does not compile. MPICH1 code also work on MPICH2 on our cluster and the performance are similar. This code assume commutative and associative reduce operator (MPI_SUM, MPI_MAX, etc). */ //#include /* This fucntion performs all-reduce operation as follow. 1) binomial_tree reduce inside each SMP node 2) binomial_tree reduce intra-communication between root of each SMP node 3) binomial_tree bcast intra-communication between root of each SMP node 4) binomial_tree bcast inside each SMP node */ int smpi_coll_tuned_allreduce_smp_binomial(void *send_buf, void *recv_buf, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { int comm_size, rank; void *tmp_buf; int tag = COLL_TAG_ALLREDUCE; int mask, src, dst; if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } int num_core=1; if (smpi_comm_is_uniform(comm)){ num_core = smpi_comm_size(smpi_comm_get_intra_comm(comm)); } MPI_Status status; comm_size=smpi_comm_size(comm); rank=smpi_comm_rank(comm); MPI_Aint extent, lb; smpi_datatype_extent(dtype, &lb, &extent); tmp_buf = (void *) smpi_get_tmp_sendbuffer(count * extent); /* compute intra and inter ranking */ int intra_rank, inter_rank; intra_rank = rank % num_core; inter_rank = rank / num_core; /* size of processes participate in intra communications => should be equal to number of machines */ int inter_comm_size = (comm_size + num_core - 1) / num_core; /* copy input buffer to output buffer */ smpi_mpi_sendrecv(send_buf, count, dtype, rank, tag, recv_buf, count, dtype, rank, tag, comm, &status); /* start binomial reduce intra communication inside each SMP node */ mask = 1; while (mask < num_core) { if ((mask & intra_rank) == 0) { src = (inter_rank * num_core) + (intra_rank | mask); if (src < comm_size) { smpi_mpi_recv(tmp_buf, count, dtype, src, tag, comm, &status); smpi_op_apply(op, tmp_buf, recv_buf, &count, &dtype); } } else { dst = (inter_rank * num_core) + (intra_rank & (~mask)); smpi_mpi_send(recv_buf, count, dtype, dst, tag, comm); break; } mask <<= 1; } /* start binomial reduce inter-communication between each SMP nodes: each node only have one process that can communicate to other nodes */ if (intra_rank == 0) { mask = 1; while (mask < inter_comm_size) { if ((mask & inter_rank) == 0) { src = (inter_rank | mask) * num_core; if (src < comm_size) { smpi_mpi_recv(tmp_buf, count, dtype, src, tag, comm, &status); smpi_op_apply(op, tmp_buf, recv_buf, &count, &dtype); } } else { dst = (inter_rank & (~mask)) * num_core; smpi_mpi_send(recv_buf, count, dtype, dst, tag, comm); break; } mask <<= 1; } } /* start binomial broadcast inter-communication between each SMP nodes: each node only have one process that can communicate to other nodes */ if (intra_rank == 0) { mask = 1; while (mask < inter_comm_size) { if (inter_rank & mask) { src = (inter_rank - mask) * num_core; smpi_mpi_recv(recv_buf, count, dtype, src, tag, comm, &status); break; } mask <<= 1; } mask >>= 1; while (mask > 0) { if (inter_rank < inter_comm_size) { dst = (inter_rank + mask) * num_core; if (dst < comm_size) { smpi_mpi_send(recv_buf, count, dtype, dst, tag, comm); } } mask >>= 1; } } /* start binomial broadcast intra-communication inside each SMP nodes */ int num_core_in_current_smp = num_core; if (inter_rank == (inter_comm_size - 1)) { num_core_in_current_smp = comm_size - (inter_rank * num_core); } mask = 1; while (mask < num_core_in_current_smp) { if (intra_rank & mask) { src = (inter_rank * num_core) + (intra_rank - mask); smpi_mpi_recv(recv_buf, count, dtype, src, tag, comm, &status); break; } mask <<= 1; } mask >>= 1; while (mask > 0) { dst = (inter_rank * num_core) + (intra_rank + mask); if (dst < comm_size) { smpi_mpi_send(recv_buf, count, dtype, dst, tag, comm); } mask >>= 1; } smpi_free_tmp_buffer(tmp_buf); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/reduce-rab.c000644 001750 001750 00000110143 13031031272 022120 0ustar00mquinsonmquinson000000 000000 /* extracted from mpig_myreduce.c with :3,$s/MPL/MPI/g and :%s/\\\\$/ \\/ */ /* Copyright: Rolf Rabenseifner, 1997 * Computing Center University of Stuttgart * rabenseifner@rus.uni-stuttgart.de * * The usage of this software is free, * but this header must not be removed. */ #include "colls_private.h" #include #include #define REDUCE_NEW_ALWAYS 1 #ifdef CRAY # define SCR_LNG_OPTIM(bytelng) 128 + ((bytelng+127)/256) * 256; /* = 16 + multiple of 32 doubles*/ # define REDUCE_LIMITS /* values are lower limits for count arg. */ \ /* routine = reduce allreduce */ \ /* size = 2, 3,2**n,other 2, 3,2**n,other */ \ static int Lsh[2][4]={{ 896,1728, 576, 736},{ 448,1280, 512, 512}}; \ static int Lin[2][4]={{ 896,1728, 576, 736},{ 448,1280, 512, 512}}; \ static int Llg[2][4]={{ 896,1728, 576, 736},{ 448,1280, 512, 512}}; \ static int Lfp[2][4]={{ 896,1728, 576, 736},{ 448,1280, 512, 512}}; \ static int Ldb[2][4]={{ 896,1728, 576, 736},{ 448,1280, 512, 512}}; \ static int Lby[2][4]={{ 896,1728, 576, 736},{ 448,1280, 512, 512}}; #endif #ifdef REDUCE_NEW_ALWAYS # undef REDUCE_LIMITS # define REDUCE_LIMITS /* values are lower limits for count arg. */ \ /* routine = reduce allreduce */ \ /* size = 2, 3,2**n,other 2, 3,2**n,other */ \ static int Lsh[2][4]={{ 1, 1, 1, 1},{ 1, 1, 1, 1}}; \ static int Lin[2][4]={{ 1, 1, 1, 1},{ 1, 1, 1, 1}}; \ static int Llg[2][4]={{ 1, 1, 1, 1},{ 1, 1, 1, 1}}; \ static int Lfp[2][4]={{ 1, 1, 1, 1},{ 1, 1, 1, 1}}; \ static int Ldb[2][4]={{ 1, 1, 1, 1},{ 1, 1, 1, 1}}; \ static int Lby[2][4]={{ 1, 1, 1, 1},{ 1, 1, 1, 1}}; #endif /* Fast reduce and allreduce algorithm for longer buffers and predefined operations. This algorithm is explaned with the example of 13 nodes. The nodes are numbered 0, 1, 2, ... 12. The sendbuf content is a, b, c, ... m. The buffer array is notated with ABCDEFGH, this means that e.g. 'C' is the third 1/8 of the buffer. The algorithm computes { [(a+b)+(c+d)] + [(e+f)+(g+h)] } + { [(i+j)+k] + [l+m] } This is equivalent to the mostly used binary tree algorithm (e.g. used in mpich). size := number of nodes in the communicator. 2**n := the power of 2 that is next smaller or equal to the size. r := size - 2**n Exa.: size=13 ==> n=3, r=5 (i.e. size == 13 == 2**n+r == 2**3 + 5) The algoritm needs for the execution of one mpi_coll_reduce_fun - for r==0 exec_time = n*(L1+L2) + buf_lng * (1-1/2**n) * (T1 + T2 + O/d) - for r>0 exec_time = (n+1)*(L1+L2) + buf_lng * T1 + buf_lng*(1+1/2-1/2**n)* (T2 + O/d) with L1 = latency of a message transfer e.g. by send+recv L2 = latency of a message exchange e.g. by sendrecv T1 = additional time to transfer 1 byte (= 1/bandwidth) T2 = additional time to exchange 1 byte in both directions. O = time for one operation d = size of the datatype On a MPP system it is expected that T1==T2. In Comparison with the binary tree algorithm that needs: - for r==0 exec_time_bin_tree = n * L1 + buf_lng * n * (T1 + O/d) - for r>0 exec_time_bin_tree = (n+1)*L1 + buf_lng*(n+1)*(T1 + O/d) the new algorithm is faster if (assuming T1=T2) for n>2: for r==0: buf_lng > L2 * n / [ (n-2) * T1 + (n-1) * O/d ] for r>0: buf_lng > L2 * (n+1) / [ (n-1.5)*T1 + (n-0.5)*O/d ] for size = 5, 6, and 7: buf_lng > L2 / [0.25 * T1 + 0.58 * O/d ] for size = 4: buf_lng > L2 / [0.25 * T1 + 0.62 * O/d ] for size = 2 and 3: buf_lng > L2 / [ 0.5 * O/d ] and for O/d >> T1 we can summarize: The new algorithm is faster for about buf_lng > 2 * L2 * d / O Example L1 = L2 = 50 us, bandwidth=300MB/s, i.e. T1 = T2 = 3.3 ns/byte and 10 MFLOP/s, i.e. O = 100 ns/FLOP for double, i.e. d = 8 byte/FLOP ==> the new algorithm is faster for about buf_lng>8000 bytes i.e count > 1000 doubles ! Step 1) compute n and r Step 2) if myrank < 2*r split the buffer into ABCD and EFGH even myrank: send buffer EFGH to myrank+1 (buf_lng/2 exchange) receive buffer ABCD from myrank+1 compute op for ABCD (count/2 ops) receive result EFGH (buf_lng/2 transfer) odd myrank: send buffer ABCD to myrank+1 receive buffer EFGH from myrank+1 compute op for EFGH send result EFGH Result: node: 0 2 4 6 8 10 11 12 value: a+b c+d e+f g+h i+j k l m Step 3) The following algorithm uses only the nodes with (myrank is even && myrank < 2*r) || (myrank >= 2*r) Step 4) define NEWRANK(old) := (old < 2*r ? old/2 : old-r) define OLDRANK(new) := (new < r ? new*2 : new+r) Result: old ranks: 0 2 4 6 8 10 11 12 new ranks: 0 1 2 3 4 5 6 7 value: a+b c+d e+f g+h i+j k l m Step 5.1) Split the buffer (ABCDEFGH) in the middle, the lower half (ABCD) is computed on even (new) ranks, the opper half (EFGH) is computed on odd (new) ranks. exchange: ABCD from 1 to 0, from 3 to 2, from 5 to 4 and from 7 to 6 EFGH from 0 to 1, from 2 to 3, from 4 to 5 and from 6 to 7 (i.e. buf_lng/2 exchange) compute op in each node on its half: (i.e. count/2 ops) Result: node 0: (a+b)+(c+d) for ABCD node 1: (a+b)+(c+d) for EFGH node 2: (e+f)+(g+h) for ABCD node 3: (e+f)+(g+h) for EFGH node 4: (i+j)+ k for ABCD node 5: (i+j)+ k for EFGH node 6: l + m for ABCD node 7: l + m for EFGH Step 5.2) Same with double distance and oncemore the half of the buffer. (i.e. buf_lng/4 exchange) (i.e. count/4 ops) Result: node 0: [(a+b)+(c+d)] + [(e+f)+(g+h)] for AB node 1: [(a+b)+(c+d)] + [(e+f)+(g+h)] for EF node 2: [(a+b)+(c+d)] + [(e+f)+(g+h)] for CD node 3: [(a+b)+(c+d)] + [(e+f)+(g+h)] for GH node 4: [(i+j)+ k ] + [ l + m ] for AB node 5: [(i+j)+ k ] + [ l + m ] for EF node 6: [(i+j)+ k ] + [ l + m ] for CD node 7: [(i+j)+ k ] + [ l + m ] for GH ... Step 5.n) Same with double distance and oncemore the half of the buffer. (i.e. buf_lng/2**n exchange) (i.e. count/2**n ops) Result: 0: { [(a+b)+(c+d)] + [(e+f)+(g+h)] } + { [(i+j)+k] + [l+m] } for A 1: { [(a+b)+(c+d)] + [(e+f)+(g+h)] } + { [(i+j)+k] + [l+m] } for E 2: { [(a+b)+(c+d)] + [(e+f)+(g+h)] } + { [(i+j)+k] + [l+m] } for C 3: { [(a+b)+(c+d)] + [(e+f)+(g+h)] } + { [(i+j)+k] + [l+m] } for G 4: { [(a+b)+(c+d)] + [(e+f)+(g+h)] } + { [(i+j)+k] + [l+m] } for B 5: { [(a+b)+(c+d)] + [(e+f)+(g+h)] } + { [(i+j)+k] + [l+m] } for F 6: { [(a+b)+(c+d)] + [(e+f)+(g+h)] } + { [(i+j)+k] + [l+m] } for D 7: { [(a+b)+(c+d)] + [(e+f)+(g+h)] } + { [(i+j)+k] + [l+m] } for H For mpi_coll_allreduce_fun: ------------------ Step 6.1) Exchange on the last distance (2*n) the last result (i.e. buf_lng/2**n exchange) Step 6.2) Same with half the distance and double of the result (i.e. buf_lng/2**(n-1) exchange) ... Step 6.n) Same with distance 1 and double of the result (== half of the original buffer) (i.e. buf_lng/2 exchange) Result after 6.1 6.2 6.n on node 0: AB ABCD ABCDEFGH on node 1: EF EFGH ABCDEFGH on node 2: CD ABCD ABCDEFGH on node 3: GH EFGH ABCDEFGH on node 4: AB ABCD ABCDEFGH on node 5: EF EFGH ABCDEFGH on node 6: CD ABCD ABCDEFGH on node 7: GH EFGH ABCDEFGH Step 7) If r > 0 transfer the result from the even nodes with old rank < 2*r to the odd nodes with old rank < 2*r (i.e. buf_lng transfer) Result: { [(a+b)+(c+d)] + [(e+f)+(g+h)] } + { [(i+j)+k] + [l+m] } for ABCDEFGH on all nodes 0..12 For mpi_coll_reduce_fun: --------------- Step 6.0) If root node not in the list of the nodes with newranks (see steps 3+4) then send last result from node 0 to the root (buf_lng/2**n transfer) and replace the role of node 0 by the root node. Step 6.1) Send on the last distance (2**(n-1)) the last result from node with bit '2**(n-1)' in the 'new rank' unequal to that of root's new rank to the node with same '2**(n-1)' bit. (i.e. buf_lng/2**n transfer) Step 6.2) Same with half the distance and double of the result and bit '2**(n-2)' (i.e. buf_lng/2**(n-1) transfer) ... Step 6.n) Same with distance 1 and double of the result (== half of the original buffer) and bit '2**0' (i.e. buf_lng/2 transfer) Example: roots old rank: 10 roots new rank: 5 Results: 6.1 6.2 6.n action result action result action result on node 0: send A on node 1: send E on node 2: send C on node 3: send G on node 4: recv A => AB recv CD => ABCD send ABCD on node 5: recv E => EF recv GH => EFGH recv ABCD => ABCDEFGH on node 6: recv C => CD send CD on node 7: recv G => GH send GH Benchmark results on CRAY T3E ----------------------------- uname -a: (sn6307 hwwt3e 1.6.1.51 unicosmk CRAY T3E) MPI: /opt/ctl/mpt/1.1.0.3 datatype: MPI_DOUBLE Ldb[][] = {{ 896,1728, 576, 736},{ 448,1280, 512, 512}} env: export MPI_BUFFER_MAX=4099 compiled with: cc -c -O3 -h restrict=f old = binary tree protocol of the vendor new = the new protocol and its implementation mixed = 'new' is used if count > limit(datatype, communicator size) REDUCE: communicator size measurement count prot. unit 2 3 4 6 8 12 16 24 -------------------------------------------------------------------- latency 1 mixed us 20.7 35.1 35.6 49.1 49.2 61.8 62.4 74.8 old us 19.0 32.9 33.8 47.1 47.2 61.7 62.1 73.2 -------------------------------------------------------------------- bandwidth 128k mixed MB/s 75.4 34.9 49.1 27.3 41.1 24.6 38.0 23.7 (=buf_lng/time) old MB/s 28.8 16.2 16.3 11.6 11.6 8.8 8.8 7.2 ration = mixed/old 2.6 2.1 3.0 2.4 3.5 2.8 4.3 3.3 -------------------------------------------------------------------- limit doubles 896 1536 576 736 576 736 576 736 bandwidth limit mixed MB/s 35.9 20.5 18.6 12.3 13.5 9.2 10.0 8.6 old MB/s 35.9 20.3 17.8 13.0 12.5 9.6 9.2 7.8 ratio = mixed/old 1.00 1.01 1.04 0.95 1.08 0.96 1.09 1.10 communicator size measurement count prot. unit 32 48 64 96 128 192 256 --------------------------------------------------------------- latency 1 mixed us 77.8 88.5 90.6 102 1) old us 78.6 87.2 90.1 99.7 108 119 120 --------------------------------------------------------------- bandwidth 128k mixed MB/s 35.1 23.3 34.1 22.8 34.4 22.4 33.9 (=buf_lng/time) old MB/s 6.0 6.0 6.0 5.2 5.2 4.6 4.6 ration = mixed/old 5.8 3.9 5.7 4.4 6.6 4.8 7.4 5) --------------------------------------------------------------- limit doubles 576 736 576 736 576 736 576 2) bandwidth limit mixed MB/s 9.7 7.5 8.4 6.5 6.9 5.5 5.1 3) old MB/s 7.7 6.4 6.4 5.7 5.5 4.9 4.7 3) ratio = mixed/old 1.26 1.17 1.31 1.14 1.26 1.12 1.08 4) ALLREDUCE: communicator size measurement count prot. unit 2 3 4 6 8 12 16 24 -------------------------------------------------------------------- latency 1 mixed us 28.2 51.0 44.5 74.4 59.9 102 74.2 133 old us 26.9 48.3 42.4 69.8 57.6 96.0 75.7 126 -------------------------------------------------------------------- bandwidth 128k mixed MB/s 74.0 29.4 42.4 23.3 35.0 20.9 32.8 19.7 (=buf_lng/time) old MB/s 20.9 14.4 13.2 9.7 9.6 7.3 7.4 5.8 ration = mixed/old 3.5 2.0 3.2 2.4 3.6 2.9 4.4 3.4 -------------------------------------------------------------------- limit doubles 448 1280 512 512 512 512 512 512 bandwidth limit mixed MB/s 26.4 15.1 16.2 8.2 12.4 7.2 10.8 5.7 old MB/s 26.1 14.9 15.0 9.1 10.5 6.7 7.3 5.3 ratio = mixed/old 1.01 1.01 1.08 0.90 1.18 1.07 1.48 1.08 communicator size measurement count prot. unit 32 48 64 96 128 192 256 --------------------------------------------------------------- latency 1 mixed us 90.3 162 109 190 old us 92.7 152 104 179 122 225 135 --------------------------------------------------------------- bandwidth 128k mixed MB/s 31.1 19.7 30.1 19.2 29.8 18.7 29.0 (=buf_lng/time) old MB/s 5.9 4.8 5.0 4.1 4.4 3.4 3.8 ration = mixed/old 5.3 4.1 6.0 4.7 6.8 5.5 7.7 --------------------------------------------------------------- limit doubles 512 512 512 512 512 512 512 bandwidth limit mixed MB/s 6.6 5.6 5.7 3.5 4.4 3.2 3.8 old MB/s 6.3 4.2 5.4 3.6 4.4 3.1 ratio = mixed/old 1.05 1.33 1.06 0.97 1.00 1.03 Footnotes: 1) This line shows that the overhead to decide which protocol should be used can be ignored. 2) This line shows the limit for the count argument. If count < limit then the vendor protocol is used, otherwise the new protocol is used (see variable Ldb). 3) These lines show the bandwidth (=bufer length / execution time) for both protocols. 4) This line shows that the limit is choosen well if the ratio is between 0.95 (loosing 5% for buffer length near and >=limit) and 1.10 (not gaining 10% for buffer length near and b2[i]?b1[i]:b2[i]); break; \ case MPIM_MIN : \ for(i=0;ib2[i]?b1[i]:b2[i]); break; \ case MPIM_MIN : \ for(i=0;i 1) /*otherwise no balancing_protocol*/ { int ss; if (size==2) ss=0; else if (size==3) ss=1; else { int s = size; while (!(s & 1)) s = s >> 1; if (s==1) /* size == power of 2 */ ss = 2; else /* size != power of 2 */ ss = 3; } switch(op) { case MPIM_MAX: case MPIM_MIN: case MPIM_SUM: case MPIM_PROD: case MPIM_LAND: case MPIM_LOR: case MPIM_LXOR: case MPIM_BAND: case MPIM_BOR: case MPIM_BXOR: switch(datatype) { case MPIM_SHORT: case MPIM_UNSIGNED_SHORT: new_prot = count >= Lsh[is_all][ss]; break; case MPIM_INT: case MPIM_UNSIGNED: new_prot = count >= Lin[is_all][ss]; break; case MPIM_LONG: case MPIM_UNSIGNED_LONG: new_prot = count >= Llg[is_all][ss]; break; default: break; } default: break;} switch(op) { case MPIM_MAX: case MPIM_MIN: case MPIM_SUM: case MPIM_PROD: switch(datatype) { case MPIM_FLOAT: new_prot = count >= Lfp[is_all][ss]; break; case MPIM_DOUBLE: new_prot = count >= Ldb[is_all][ss]; break; default: break; } default: break;} switch(op) { case MPIM_BAND: case MPIM_BOR: case MPIM_BXOR: switch(datatype) { case MPIM_BYTE: new_prot = count >= Lby[is_all][ss]; break; default: break; } default: break;} # ifdef DEBUG { char *ss_str[]={"two","three","power of 2","no power of 2"}; printf("MPI_(All)Reduce: is_all=%1d, size=%1d=%s, new_prot=%1d\n", is_all, size, ss_str[ss], new_prot); fflush(stdout); } # endif } if (new_prot) { sendbuf = (char*) Sendbuf; recvbuf = (char*) Recvbuf; MPI_Comm_rank(comm, &myrank); MPI_Type_extent(mpi_datatype, &typelng); scrlng = typelng * count; #ifdef NO_CACHE_OPTIMIZATION scr1buf = malloc(scrlng); scr2buf = malloc(scrlng); scr3buf = malloc(scrlng); #else # ifdef SCR_LNG_OPTIM scrlng = SCR_LNG_OPTIM(scrlng); # endif scr2buf = malloc(3*scrlng); /* To test cache problems. */ scr1buf = scr2buf + 1*scrlng; /* scr1buf and scr3buf must not*/ scr3buf = scr2buf + 2*scrlng; /* be used for malloc because */ /* they are interchanged below.*/ #endif computed = 0; if (is_all) root = myrank; /* for correct recvbuf handling */ /*...step 1 */ # ifdef DEBUG printf("[%2d] step 1 begin\n",myrank); fflush(stdout); # endif n = 0; x_size = 1; while (2*x_size <= size) { n++; x_size = x_size * 2; } /* x_sixe == 2**n */ r = size - x_size; /*...step 2 */ # ifdef DEBUG printf("[%2d] step 2 begin n=%d, r=%d\n",myrank,n,r);fflush(stdout); # endif if (myrank < 2*r) { if ((myrank % 2) == 0 /*even*/) { MPI_I_Sendrecv(sendbuf + (count/2)*typelng, count - count/2, mpi_datatype, myrank+1, 1220, scr2buf, count/2,mpi_datatype, myrank+1, 1221, comm, &status); MPI_I_do_op(sendbuf, scr2buf, scr1buf, count/2, datatype, op); smpi_mpi_recv(scr1buf + (count/2)*typelng, count - count/2, mpi_datatype, myrank+1, 1223, comm, &status); computed = 1; # ifdef DEBUG { int i; printf("[%2d] after step 2: val=", myrank); for (i=0; i= 2*r) || ((myrank%2 == 0) && (myrank < 2*r))) mynewrank = (myrank < 2*r ? myrank/2 : myrank-r); else mynewrank = -1; if (mynewrank >= 0) { /* begin -- only for nodes with new rank */ # define OLDRANK(new) ((new) < r ? (new)*2 : (new)+r) /*...step 5 */ x_start = 0; x_count = count; for (idx=0, x_base=1; idx= 0) { /* begin -- only for nodes with new rank */ # define OLDRANK(new) ((new) < r ? (new)*2 : (new)+r) for(idx=n-1, x_base=x_size/2; idx>=0; idx--, x_base=x_base/2) { # ifdef DEBUG printf("[%2d](%2d) step 6.%d begin\n",myrank,mynewrank,n-idx); fflush(stdout); # endif if (((mynewrank/x_base) % 2) == 0 /*even*/) { MPI_I_Sendrecv(recvbuf + start_even[idx]*typelng, count_even[idx], mpi_datatype, OLDRANK(mynewrank+x_base),1241, recvbuf + start_odd[idx]*typelng, count_odd[idx], mpi_datatype, OLDRANK(mynewrank+x_base),1242, comm, &status); # ifdef DEBUG x_start = start_odd[idx]; x_count = count_odd[idx]; # endif } else /*odd*/ { MPI_I_Sendrecv(recvbuf + start_odd[idx]*typelng, count_odd[idx], mpi_datatype, OLDRANK(mynewrank-x_base),1242, recvbuf + start_even[idx]*typelng, count_even[idx], mpi_datatype, OLDRANK(mynewrank-x_base),1241, comm, &status); # ifdef DEBUG x_start = start_even[idx]; x_count = count_even[idx]; # endif } # ifdef DEBUG { int i; printf("[%2d](%2d) after step 6.%d end: start=%2d count=%2d val=", myrank,mynewrank,n-idx,x_start,x_count); for (i=0; i= 0) { /* begin -- only for nodes with new rank */ # define OLDRANK(new) ((new)==newroot ? root \ : ((new)=0; idx--, x_base=x_base/2) { # ifdef DEBUG printf("[%2d](%2d) step 6.%d begin\n",myrank,mynewrank,n-idx); fflush(stdout); # endif if ((mynewrank & x_base) != (newroot & x_base)) { if (((mynewrank/x_base) % 2) == 0 /*even*/) { x_start = start_even[idx]; x_count = count_even[idx]; partner = mynewrank+x_base; } else { x_start = start_odd[idx]; x_count = count_odd[idx]; partner = mynewrank-x_base; } smpi_mpi_send(scr1buf + x_start*typelng, x_count, mpi_datatype, OLDRANK(partner), 1244, comm); } else /*odd*/ { if (((mynewrank/x_base) % 2) == 0 /*even*/) { x_start = start_odd[idx]; x_count = count_odd[idx]; partner = mynewrank+x_base; } else { x_start = start_even[idx]; x_count = count_even[idx]; partner = mynewrank-x_base; } smpi_mpi_recv((myrank==root ? recvbuf : scr1buf) + x_start*typelng, x_count, mpi_datatype, OLDRANK(partner), 1244, comm, &status); # ifdef DEBUG { int i; printf("[%2d](%2d) after step 6.%d end: start=%2d count=%2d val=", myrank,mynewrank,n-idx,x_start,x_count); for (i=0; i int smpi_coll_tuned_allreduce_rdb(void *sbuff, void *rbuff, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { int nprocs, rank, tag = COLL_TAG_ALLREDUCE; int mask, dst, pof2, newrank, rem, newdst; MPI_Aint extent, lb; MPI_Status status; void *tmp_buf = NULL; /* #ifdef MPICH2_REDUCTION MPI_User_function * uop = MPIR_Op_table[op % 16 - 1]; #else MPI_User_function *uop; struct MPIR_OP *op_ptr; op_ptr = MPIR_ToPointer(op); uop = op_ptr->op; #endif */ nprocs=smpi_comm_size(comm); rank=smpi_comm_rank(comm); smpi_datatype_extent(dtype, &lb, &extent); tmp_buf = (void *) smpi_get_tmp_sendbuffer(count * extent); smpi_mpi_sendrecv(sbuff, count, dtype, rank, 500, rbuff, count, dtype, rank, 500, comm, &status); // find nearest power-of-two less than or equal to comm_size pof2 = 1; while (pof2 <= nprocs) pof2 <<= 1; pof2 >>= 1; rem = nprocs - pof2; // In the non-power-of-two case, all even-numbered // processes of rank < 2*rem send their data to // (rank+1). These even-numbered processes no longer // participate in the algorithm until the very end. The // remaining processes form a nice power-of-two. if (rank < 2 * rem) { // even if (rank % 2 == 0) { smpi_mpi_send(rbuff, count, dtype, rank + 1, tag, comm); // temporarily set the rank to -1 so that this // process does not pariticipate in recursive // doubling newrank = -1; } else // odd { smpi_mpi_recv(tmp_buf, count, dtype, rank - 1, tag, comm, &status); // do the reduction on received data. since the // ordering is right, it doesn't matter whether // the operation is commutative or not. smpi_op_apply(op, tmp_buf, rbuff, &count, &dtype); // change the rank newrank = rank / 2; } } else // rank >= 2 * rem newrank = rank - rem; // If op is user-defined or count is less than pof2, use // recursive doubling algorithm. Otherwise do a reduce-scatter // followed by allgather. (If op is user-defined, // derived datatypes are allowed and the user could pass basic // datatypes on one process and derived on another as long as // the type maps are the same. Breaking up derived // datatypes to do the reduce-scatter is tricky, therefore // using recursive doubling in that case.) if (newrank != -1) { mask = 0x1; while (mask < pof2) { newdst = newrank ^ mask; // find real rank of dest dst = (newdst < rem) ? newdst * 2 + 1 : newdst + rem; // Send the most current data, which is in recvbuf. Recv // into tmp_buf smpi_mpi_sendrecv(rbuff, count, dtype, dst, tag, tmp_buf, count, dtype, dst, tag, comm, &status); // tmp_buf contains data received in this step. // recvbuf contains data accumulated so far // op is commutative OR the order is already right // we assume it is commuttive op // if (op -> op_commute || (dst < rank)) if ((dst < rank)) { smpi_op_apply(op, tmp_buf, rbuff, &count, &dtype); } else // op is noncommutative and the order is not right { smpi_op_apply(op, rbuff, tmp_buf, &count, &dtype); // copy result back into recvbuf smpi_mpi_sendrecv(tmp_buf, count, dtype, rank, tag, rbuff, count, dtype, rank, tag, comm, &status); } mask <<= 1; } } // In the non-power-of-two case, all odd-numbered processes of // rank < 2 * rem send the result to (rank-1), the ranks who didn't // participate above. if (rank < 2 * rem) { if (rank % 2) // odd smpi_mpi_send(rbuff, count, dtype, rank - 1, tag, comm); else // even smpi_mpi_recv(rbuff, count, dtype, rank + 1, tag, comm, &status); } smpi_free_tmp_buffer(tmp_buf); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allreduce-lr.c000644 001750 001750 00000007556 13031031271 022476 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /* IMPLEMENTED BY PITCH PATARASUK Non-topoloty-specific all-reduce operation designed bandwidth optimally Bug fixing by Xin Yuan, 04/04/2008 */ /* ** NOTE ** Use -DMPICH2_REDUCTION if this code does not compile. MPICH1 code also work on MPICH2 on our cluster and the performance are similar. This code assume commutative and associative reduce operator (MPI_SUM, MPI_MAX, etc). */ //#include int smpi_coll_tuned_allreduce_lr(void *sbuf, void *rbuf, int rcount, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { int tag = COLL_TAG_ALLREDUCE; MPI_Status status; int rank, i, size, count; int send_offset, recv_offset; int remainder, remainder_flag, remainder_offset; rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); /* make it compatible with all data type */ MPI_Aint extent; extent = smpi_datatype_get_extent(dtype); /* when communication size is smaller than number of process (not support) */ if (rcount < size) { XBT_WARN("MPI_allreduce_lr use default MPI_allreduce."); smpi_mpi_allreduce(sbuf, rbuf, rcount, dtype, op, comm); return MPI_SUCCESS; } /* when communication size is not divisible by number of process: call the native implementation for the remain chunk at the end of the operation */ if (rcount % size != 0) { remainder = rcount % size; remainder_flag = 1; remainder_offset = (rcount / size) * size * extent; } else { remainder = remainder_flag = remainder_offset = 0; } /* size of each point-to-point communication is equal to the size of the whole message divided by number of processes */ count = rcount / size; /* our ALL-REDUCE implementation 1. copy (partial of)send_buf to recv_buf 2. use logical ring reduce-scatter 3. use logical ring all-gather */ // copy partial data send_offset = ((rank - 1 + size) % size) * count * extent; recv_offset = ((rank - 1 + size) % size) * count * extent; smpi_mpi_sendrecv((char *) sbuf + send_offset, count, dtype, rank, tag - 1, (char *) rbuf + recv_offset, count, dtype, rank, tag - 1, comm, &status); // reduce-scatter for (i = 0; i < (size - 1); i++) { send_offset = ((rank - 1 - i + 2 * size) % size) * count * extent; recv_offset = ((rank - 2 - i + 2 * size) % size) * count * extent; // recv_offset = ((rank-i+2*size)%size)*count*extent; smpi_mpi_sendrecv((char *) rbuf + send_offset, count, dtype, ((rank + 1) % size), tag + i, (char *) rbuf + recv_offset, count, dtype, ((rank + size - 1) % size), tag + i, comm, &status); // compute result to rbuf+recv_offset smpi_op_apply(op, (char *) sbuf + recv_offset, (char *) rbuf + recv_offset, &count, &dtype); } // all-gather for (i = 0; i < (size - 1); i++) { send_offset = ((rank - i + 2 * size) % size) * count * extent; recv_offset = ((rank - 1 - i + 2 * size) % size) * count * extent; smpi_mpi_sendrecv((char *) rbuf + send_offset, count, dtype, ((rank + 1) % size), tag + i, (char *) rbuf + recv_offset, count, dtype, ((rank + size - 1) % size), tag + i, comm, &status); } /* when communication size is not divisible by number of process: call the native implementation for the remain chunk at the end of the operation */ if (remainder_flag) { return mpi_coll_allreduce_fun((char *) sbuf + remainder_offset, (char *) rbuf + remainder_offset, remainder, dtype, op, comm); } return 0; } SimGrid-3.14.159/src/smpi/colls/allgatherv-ompi-bruck.c000644 001750 001750 00000016405 13031031271 024313 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2009 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * Copyright (c) 2009 University of Houston. All rights reserved. * * Additional copyrights may follow */ #include "colls_private.h" /* * ompi_coll_tuned_allgatherv_intra_bruck * * Function: allgather using O(log(N)) steps. * Accepts: Same arguments as MPI_Allgather * Returns: MPI_SUCCESS or error code * * Description: Variation to All-to-all algorithm described by Bruck et al.in * "Efficient Algorithms for All-to-all Communications * in Multiport Message-Passing Systems" * Note: Unlike in case of allgather implementation, we relay on * indexed datatype to select buffers appropriately. * The only additional memory requirement is for creation of * temporary datatypes. * Example on 7 nodes (memory lay out need not be in-order) * Initial set up: * # 0 1 2 3 4 5 6 * [0] [ ] [ ] [ ] [ ] [ ] [ ] * [ ] [1] [ ] [ ] [ ] [ ] [ ] * [ ] [ ] [2] [ ] [ ] [ ] [ ] * [ ] [ ] [ ] [3] [ ] [ ] [ ] * [ ] [ ] [ ] [ ] [4] [ ] [ ] * [ ] [ ] [ ] [ ] [ ] [5] [ ] * [ ] [ ] [ ] [ ] [ ] [ ] [6] * Step 0: send message to (rank - 2^0), receive message from (rank + 2^0) * # 0 1 2 3 4 5 6 * [0] [ ] [ ] [ ] [ ] [ ] [0] * [1] [1] [ ] [ ] [ ] [ ] [ ] * [ ] [2] [2] [ ] [ ] [ ] [ ] * [ ] [ ] [3] [3] [ ] [ ] [ ] * [ ] [ ] [ ] [4] [4] [ ] [ ] * [ ] [ ] [ ] [ ] [5] [5] [ ] * [ ] [ ] [ ] [ ] [ ] [6] [6] * Step 1: send message to (rank - 2^1), receive message from (rank + 2^1). * message contains all blocks from (rank) .. (rank + 2^2) with * wrap around. * # 0 1 2 3 4 5 6 * [0] [ ] [ ] [ ] [0] [0] [0] * [1] [1] [ ] [ ] [ ] [1] [1] * [2] [2] [2] [ ] [ ] [ ] [2] * [3] [3] [3] [3] [ ] [ ] [ ] * [ ] [4] [4] [4] [4] [ ] [ ] * [ ] [ ] [5] [5] [5] [5] [ ] * [ ] [ ] [ ] [6] [6] [6] [6] * Step 2: send message to (rank - 2^2), receive message from (rank + 2^2). * message size is "all remaining blocks" * # 0 1 2 3 4 5 6 * [0] [0] [0] [0] [0] [0] [0] * [1] [1] [1] [1] [1] [1] [1] * [2] [2] [2] [2] [2] [2] [2] * [3] [3] [3] [3] [3] [3] [3] * [4] [4] [4] [4] [4] [4] [4] * [5] [5] [5] [5] [5] [5] [5] * [6] [6] [6] [6] [6] [6] [6] */ int smpi_coll_tuned_allgatherv_ompi_bruck(void *sbuf, int scount, MPI_Datatype sdtype, void *rbuf, int *rcounts, int *rdispls, MPI_Datatype rdtype, MPI_Comm comm) { int sendto, recvfrom, blockcount, i; unsigned int distance; int *new_rcounts = NULL, *new_rdispls = NULL; int *new_scounts = NULL, *new_sdispls = NULL; ptrdiff_t slb, rlb, sext, rext; char *tmpsend = NULL, *tmprecv = NULL; MPI_Datatype new_rdtype = MPI_DATATYPE_NULL, new_sdtype = MPI_DATATYPE_NULL; unsigned int size = smpi_comm_size(comm); unsigned int rank = smpi_comm_rank(comm); XBT_DEBUG( "coll:tuned:allgather_ompi_bruck rank %d", rank); smpi_datatype_extent (sdtype, &slb, &sext); smpi_datatype_extent (rdtype, &rlb, &rext); /* Initialization step: - if send buffer is not MPI_IN_PLACE, copy send buffer to block rank of the receive buffer. */ tmprecv = (char*) rbuf + rdispls[rank] * rext; if (MPI_IN_PLACE != sbuf) { tmpsend = (char*) sbuf; smpi_datatype_copy(tmpsend, scount, sdtype, tmprecv, rcounts[rank], rdtype); } /* Communication step: At every step i, rank r: - doubles the distance - sends message with blockcount blocks, (rbuf[rank] .. rbuf[rank + 2^i]) to rank (r - distance) - receives message of blockcount blocks, (rbuf[r + distance] ... rbuf[(r+distance) + 2^i]) from rank (r + distance) - blockcount doubles until the last step when only the remaining data is exchanged. */ new_rcounts = (int*) calloc(4*size, sizeof(int)); new_rdispls = new_rcounts + size; new_scounts = new_rdispls + size; new_sdispls = new_scounts + size; for (distance = 1; distance < size; distance<<=1) { recvfrom = (rank + distance) % size; sendto = (rank - distance + size) % size; if (distance <= (size >> 1)) { blockcount = distance; } else { blockcount = size - distance; } /* create send and receive datatypes */ for (i = 0; i < blockcount; i++) { const int tmp_srank = (rank + i) % size; const int tmp_rrank = (recvfrom + i) % size; new_scounts[i] = rcounts[tmp_srank]; new_sdispls[i] = rdispls[tmp_srank]; new_rcounts[i] = rcounts[tmp_rrank]; new_rdispls[i] = rdispls[tmp_rrank]; } smpi_datatype_indexed(blockcount, new_scounts, new_sdispls, rdtype, &new_sdtype); smpi_datatype_indexed(blockcount, new_rcounts, new_rdispls, rdtype, &new_rdtype); smpi_datatype_commit(&new_sdtype); smpi_datatype_commit(&new_rdtype); /* Sendreceive */ smpi_mpi_sendrecv(rbuf, 1, new_sdtype, sendto, COLL_TAG_ALLGATHERV, rbuf, 1, new_rdtype, recvfrom, COLL_TAG_ALLGATHERV, comm, MPI_STATUS_IGNORE); smpi_datatype_unuse(new_sdtype); smpi_datatype_unuse(new_rdtype); } free(new_rcounts); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/alltoall-ring.c000644 001750 001750 00000003610 13031031271 022647 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** * Function: alltoall_ring * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works in P - 1 steps. In step i, node j - i -> j -> j + i. * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_alltoall_ring(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Status s; MPI_Aint send_chunk, recv_chunk; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALL; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); send_chunk = smpi_datatype_get_extent(send_type); recv_chunk = smpi_datatype_get_extent(recv_type); send_chunk *= send_count; recv_chunk *= recv_count; for (i = 0; i < num_procs; i++) { src = (rank - i + num_procs) % num_procs; dst = (rank + i) % num_procs; smpi_mpi_sendrecv(send_ptr + dst * send_chunk, send_count, send_type, dst, tag, recv_ptr + src * recv_chunk, recv_count, recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/colls_private.h000644 001750 001750 00000000637 13031031273 022771 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_COLLS_PRIVATE_H #define SMPI_COLLS_PRIVATE_H #include "colls.h" XBT_LOG_EXTERNAL_DEFAULT_CATEGORY(smpi_colls); #endif SimGrid-3.14.159/src/smpi/colls/allgatherv-pair.c000644 001750 001750 00000011056 13031031271 023173 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** Copyright (c) 2006, Ahmad Faraj & Xin Yuan, All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Florida State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ************************************************************************* * Any results obtained from executing this software require the * * acknowledgment and citation of the software and its owners. * * The full citation is given below: * * * * A. Faraj and X. Yuan. "Automatic Generation and Tuning of MPI * * Collective Communication Routines." The 19th ACM International * * Conference on Supercomputing (ICS), Cambridge, Massachusetts, * * June 20-22, 2005. * ************************************************************************* *****************************************************************************/ /***************************************************************************** * Function: allgather_pair * return: int * inputs: * send_buff: send input buffer * send_count: number of elements to send * send_type: data type of elements being sent * recv_buff: receive output buffer * recv_count: number of elements to received * recv_type: data type of elements being received * comm: communication * Descrp: Function works when P is power of two. In each phase of P - 1 * phases, nodes in pair communicate their data. * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_allgatherv_pair(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint extent; unsigned int i, src, dst; int tag = COLL_TAG_ALLGATHERV; MPI_Status status; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; unsigned int rank = smpi_comm_rank(comm); unsigned int num_procs = smpi_comm_size(comm); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "allgatherv pair algorithm can't be used with non power of two number of processes ! "); extent = smpi_datatype_get_extent(send_type); // local send/recv smpi_mpi_sendrecv(send_ptr, send_count, send_type, rank, tag, recv_ptr + recv_disps[rank] * extent, recv_counts[rank], recv_type, rank, tag, comm, &status); for (i = 1; i < num_procs; i++) { src = dst = rank ^ i; smpi_mpi_sendrecv(send_ptr, send_count, send_type, dst, tag, recv_ptr + recv_disps[src] * extent, recv_counts[src], recv_type, src, tag, comm, &status); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/bcast-arrival-scatter.c000644 001750 001750 00000015537 13031031271 024316 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" #ifndef BCAST_ARRIVAL_PATTERN_AWARE_HEADER_SIZE #define BCAST_ARRIVAL_PATTERN_AWARE_HEADER_SIZE 128 #endif #ifndef BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE #define BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE 128 #endif /* Non-topology-specific pipelined linear-bcast function */ int smpi_coll_tuned_bcast_arrival_scatter(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int tag = -COLL_TAG_BCAST;//in order to use ANY_TAG, make this one positive int header_tag = 10; MPI_Status status; int curr_remainder; int curr_size; int curr_increment; int send_offset; int recv_offset; int send_count; int recv_count; MPI_Status temp_status_array[BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE]; int rank, size; int i, k; int sent_count; int header_index; int flag_array[BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE]; int already_sent[BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE]; int header_buf[BCAST_ARRIVAL_PATTERN_AWARE_HEADER_SIZE]; char temp_buf[BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE]; int will_send[BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE]; int max_node = BCAST_ARRIVAL_PATTERN_AWARE_MAX_NODE; int header_size = BCAST_ARRIVAL_PATTERN_AWARE_HEADER_SIZE; MPI_Aint extent; extent = smpi_datatype_get_extent(datatype); /* source and destination */ int to, from; rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); /* message too small */ if (count < size) { XBT_WARN("MPI_bcast_arrival_scatter use default MPI_bcast."); smpi_mpi_bcast(buf, count, datatype, root, comm); return MPI_SUCCESS; } /* if root is not zero send to rank zero first this can be modified to make it faster by using logical src, dst. */ if (root != 0) { if (rank == root) { smpi_mpi_send(buf, count, datatype, 0, tag - 1, comm); } else if (rank == 0) { smpi_mpi_recv(buf, count, datatype, root, tag - 1, comm, &status); } } /* value == 0 means root has not send data (or header) to the node yet */ for (i = 0; i < max_node; i++) { already_sent[i] = 0; } /* start bcast */ /* root */ if (rank == 0) { for (i = 0; i < max_node; i++) will_send[i] = 0; sent_count = 0; while (sent_count < (size - 1)) { for (k = 0; k < 3; k++) { for (i = 1; i < size; i++) { if ((already_sent[i] == 0) && (will_send[i] == 0)) { smpi_mpi_iprobe(i, MPI_ANY_TAG, comm, &flag_array[i], &temp_status_array[i]); if (flag_array[i] == 1) { will_send[i] = 1; smpi_mpi_recv(&temp_buf[i], 1, MPI_CHAR, i, tag, comm, &status); i = 0; } } } } header_index = 0; /* recv 1-byte message in this round */ for (i = 1; i < size; i++) { /* message arrive */ if ((will_send[i] == 1) && (already_sent[i] == 0)) { header_buf[header_index] = i; header_index++; sent_count++; /* will send in the next step */ already_sent[i] = 1; } } /* if (header_index != 0) { printf("header index = %d node = ",header_index); for (i=0;i j denote the communication from node i to node j. The * order of communications for node i is i -> i + 1, i -> i + 2, ..., * i -> (i + p -1) % P. * * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_allgather_spreading_simple(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Request *reqs, *req_ptr; MPI_Aint extent; int i, src, dst, rank, num_procs, num_reqs; int tag = COLL_TAG_ALLGATHER; MPI_Status status; char *recv_ptr = (char *) recv_buff; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); extent = smpi_datatype_get_extent(send_type); num_reqs = (2 * num_procs) - 2; reqs = (MPI_Request *) xbt_malloc(num_reqs * sizeof(MPI_Request)); if (!reqs) { printf("allgather-spreading-simple.c:40: cannot allocate memory\n"); MPI_Finalize(); exit(0); } req_ptr = reqs; smpi_mpi_sendrecv(send_buff, send_count, send_type, rank, tag, (char *) recv_buff + rank * recv_count * extent, recv_count, recv_type, rank, tag, comm, &status); for (i = 0; i < num_procs; i++) { src = (rank + i) % num_procs; if (src == rank) continue; *(req_ptr++) = smpi_mpi_irecv(recv_ptr + src * recv_count * extent, recv_count, recv_type, src, tag, comm); } for (i = 0; i < num_procs; i++) { dst = (rank + i) % num_procs; if (dst == rank) continue; *(req_ptr++) = smpi_mpi_isend(send_buff, send_count, send_type, dst, tag, comm); } smpi_mpi_waitall(num_reqs, reqs, MPI_STATUSES_IGNORE); free(reqs); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/bcast-SMP-binary.c000644 001750 001750 00000022431 13031031271 023123 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" int bcast_SMP_binary_segment_byte = 8192; int smpi_coll_tuned_bcast_SMP_binary(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int tag = COLL_TAG_BCAST; MPI_Status status; MPI_Request request; MPI_Request *request_array; MPI_Status *status_array; int rank, size; int i; MPI_Aint extent; extent = smpi_datatype_get_extent(datatype); rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } int host_num_core=1; if (smpi_comm_is_uniform(comm)){ host_num_core = smpi_comm_size(smpi_comm_get_intra_comm(comm)); }else{ //implementation buggy in this case return smpi_coll_tuned_bcast_mpich( buf , count, datatype, root, comm); } int segment = bcast_SMP_binary_segment_byte / extent; int pipe_length = count / segment; int remainder = count % segment; int to_intra_left = (rank / host_num_core) * host_num_core + (rank % host_num_core) * 2 + 1; int to_intra_right = (rank / host_num_core) * host_num_core + (rank % host_num_core) * 2 + 2; int to_inter_left = ((rank / host_num_core) * 2 + 1) * host_num_core; int to_inter_right = ((rank / host_num_core) * 2 + 2) * host_num_core; int from_inter = (((rank / host_num_core) - 1) / 2) * host_num_core; int from_intra = (rank / host_num_core) * host_num_core + ((rank % host_num_core) - 1) / 2; int increment = segment * extent; int base = (rank / host_num_core) * host_num_core; int num_core = host_num_core; if (((rank / host_num_core) * host_num_core) == ((size / host_num_core) * host_num_core)) num_core = size - (rank / host_num_core) * host_num_core; // if root is not zero send to rank zero first if (root != 0) { if (rank == root) smpi_mpi_send(buf, count, datatype, 0, tag, comm); else if (rank == 0) smpi_mpi_recv(buf, count, datatype, root, tag, comm, &status); } // when a message is smaller than a block size => no pipeline if (count <= segment) { // case ROOT-of-each-SMP if (rank % host_num_core == 0) { // case ROOT if (rank == 0) { //printf("node %d left %d right %d\n",rank,to_inter_left,to_inter_right); if (to_inter_left < size) smpi_mpi_send(buf, count, datatype, to_inter_left, tag, comm); if (to_inter_right < size) smpi_mpi_send(buf, count, datatype, to_inter_right, tag, comm); if ((to_intra_left - base) < num_core) smpi_mpi_send(buf, count, datatype, to_intra_left, tag, comm); if ((to_intra_right - base) < num_core) smpi_mpi_send(buf, count, datatype, to_intra_right, tag, comm); } // case LEAVES ROOT-of-eash-SMP else if (to_inter_left >= size) { //printf("node %d from %d\n",rank,from_inter); request = smpi_mpi_irecv(buf, count, datatype, from_inter, tag, comm); smpi_mpi_wait(&request, &status); if ((to_intra_left - base) < num_core) smpi_mpi_send(buf, count, datatype, to_intra_left, tag, comm); if ((to_intra_right - base) < num_core) smpi_mpi_send(buf, count, datatype, to_intra_right, tag, comm); } // case INTERMEDIAT ROOT-of-each-SMP else { //printf("node %d left %d right %d from %d\n",rank,to_inter_left,to_inter_right,from_inter); request = smpi_mpi_irecv(buf, count, datatype, from_inter, tag, comm); smpi_mpi_wait(&request, &status); smpi_mpi_send(buf, count, datatype, to_inter_left, tag, comm); if (to_inter_right < size) smpi_mpi_send(buf, count, datatype, to_inter_right, tag, comm); if ((to_intra_left - base) < num_core) smpi_mpi_send(buf, count, datatype, to_intra_left, tag, comm); if ((to_intra_right - base) < num_core) smpi_mpi_send(buf, count, datatype, to_intra_right, tag, comm); } } // case non ROOT-of-each-SMP else { // case leaves if ((to_intra_left - base) >= num_core) { request = smpi_mpi_irecv(buf, count, datatype, from_intra, tag, comm); smpi_mpi_wait(&request, &status); } // case intermediate else { request = smpi_mpi_irecv(buf, count, datatype, from_intra, tag, comm); smpi_mpi_wait(&request, &status); smpi_mpi_send(buf, count, datatype, to_intra_left, tag, comm); if ((to_intra_right - base) < num_core) smpi_mpi_send(buf, count, datatype, to_intra_right, tag, comm); } } return MPI_SUCCESS; } // pipeline bcast else { request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); // case ROOT-of-each-SMP if (rank % host_num_core == 0) { // case ROOT if (rank == 0) { for (i = 0; i < pipe_length; i++) { //printf("node %d left %d right %d\n",rank,to_inter_left,to_inter_right); if (to_inter_left < size) smpi_mpi_send((char *) buf + (i * increment), segment, datatype, to_inter_left, (tag + i), comm); if (to_inter_right < size) smpi_mpi_send((char *) buf + (i * increment), segment, datatype, to_inter_right, (tag + i), comm); if ((to_intra_left - base) < num_core) smpi_mpi_send((char *) buf + (i * increment), segment, datatype, to_intra_left, (tag + i), comm); if ((to_intra_right - base) < num_core) smpi_mpi_send((char *) buf + (i * increment), segment, datatype, to_intra_right, (tag + i), comm); } } // case LEAVES ROOT-of-eash-SMP else if (to_inter_left >= size) { //printf("node %d from %d\n",rank,from_inter); for (i = 0; i < pipe_length; i++) { request_array[i] = smpi_mpi_irecv((char *) buf + (i * increment), segment, datatype, from_inter, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { smpi_mpi_wait(&request_array[i], &status); if ((to_intra_left - base) < num_core) smpi_mpi_send((char *) buf + (i * increment), segment, datatype, to_intra_left, (tag + i), comm); if ((to_intra_right - base) < num_core) smpi_mpi_send((char *) buf + (i * increment), segment, datatype, to_intra_right, (tag + i), comm); } } // case INTERMEDIAT ROOT-of-each-SMP else { //printf("node %d left %d right %d from %d\n",rank,to_inter_left,to_inter_right,from_inter); for (i = 0; i < pipe_length; i++) { request_array[i] = smpi_mpi_irecv((char *) buf + (i * increment), segment, datatype, from_inter, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { smpi_mpi_wait(&request_array[i], &status); smpi_mpi_send((char *) buf + (i * increment), segment, datatype, to_inter_left, (tag + i), comm); if (to_inter_right < size) smpi_mpi_send((char *) buf + (i * increment), segment, datatype, to_inter_right, (tag + i), comm); if ((to_intra_left - base) < num_core) smpi_mpi_send((char *) buf + (i * increment), segment, datatype, to_intra_left, (tag + i), comm); if ((to_intra_right - base) < num_core) smpi_mpi_send((char *) buf + (i * increment), segment, datatype, to_intra_right, (tag + i), comm); } } } // case non-ROOT-of-each-SMP else { // case leaves if ((to_intra_left - base) >= num_core) { for (i = 0; i < pipe_length; i++) { request_array[i] = smpi_mpi_irecv((char *) buf + (i * increment), segment, datatype, from_intra, (tag + i), comm); } smpi_mpi_waitall((pipe_length), request_array, status_array); } // case intermediate else { for (i = 0; i < pipe_length; i++) { request_array[i] = smpi_mpi_irecv((char *) buf + (i * increment), segment, datatype, from_intra, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { smpi_mpi_wait(&request_array[i], &status); smpi_mpi_send((char *) buf + (i * increment), segment, datatype, to_intra_left, (tag + i), comm); if ((to_intra_right - base) < num_core) smpi_mpi_send((char *) buf + (i * increment), segment, datatype, to_intra_right, (tag + i), comm); } } } free(request_array); free(status_array); } // when count is not divisible by block size, use default BCAST for the remainder if ((remainder != 0) && (count > segment)) { XBT_WARN("MPI_bcast_SMP_binary use default MPI_bcast."); smpi_mpi_bcast((char *) buf + (pipe_length * increment), remainder, datatype, root, comm); } return 1; } SimGrid-3.14.159/src/smpi/colls/smpi_mpich_selector.c000644 001750 001750 00000070541 13031031272 024146 0ustar00mquinsonmquinson000000 000000 /* selector for collective algorithms based on mpich decision logic */ /* Copyright (c) 2009-2010, 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /* This is the default implementation of allreduce. The algorithm is: Algorithm: MPI_Allreduce For the heterogeneous case, we call MPI_Reduce followed by MPI_Bcast in order to meet the requirement that all processes must have the same result. For the homogeneous case, we use the following algorithms. For long messages and for builtin ops and if count >= pof2 (where pof2 is the nearest power-of-two less than or equal to the number of processes), we use Rabenseifner's algorithm (see http://www.hlrs.de/mpi/myreduce.html). This algorithm implements the allreduce in two steps: first a reduce-scatter, followed by an allgather. A recursive-halving algorithm (beginning with processes that are distance 1 apart) is used for the reduce-scatter, and a recursive doubling algorithm is used for the allgather. The non-power-of-two case is handled by dropping to the nearest lower power-of-two: the first few even-numbered processes send their data to their right neighbors (rank+1), and the reduce-scatter and allgather happen among the remaining power-of-two processes. At the end, the first few even-numbered processes get the result from their right neighbors. For the power-of-two case, the cost for the reduce-scatter is lgp.alpha + n.((p-1)/p).beta + n.((p-1)/p).gamma. The cost for the allgather lgp.alpha + n.((p-1)/p).beta. Therefore, the total cost is: Cost = 2.lgp.alpha + 2.n.((p-1)/p).beta + n.((p-1)/p).gamma For the non-power-of-two case, Cost = (2.floor(lgp)+2).alpha + (2.((p-1)/p) + 2).n.beta + n.(1+(p-1)/p).gamma For short messages, for user-defined ops, and for count < pof2 we use a recursive doubling algorithm (similar to the one in MPI_Allgather). We use this algorithm in the case of user-defined ops because in this case derived datatypes are allowed, and the user could pass basic datatypes on one process and derived on another as long as the type maps are the same. Breaking up derived datatypes to do the reduce-scatter is tricky. Cost = lgp.alpha + n.lgp.beta + n.lgp.gamma Possible improvements: End Algorithm: MPI_Allreduce */ int smpi_coll_tuned_allreduce_mpich(void *sbuf, void *rbuf, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { size_t dsize, block_dsize; int comm_size = smpi_comm_size(comm); const size_t large_message = 2048; //MPIR_PARAM_ALLREDUCE_SHORT_MSG_SIZE dsize = smpi_datatype_size(dtype); block_dsize = dsize * count; /* find nearest power-of-two less than or equal to comm_size */ int pof2 = 1; while (pof2 <= comm_size) pof2 <<= 1; pof2 >>=1; if (block_dsize > large_message && count >= pof2 && smpi_op_is_commute(op)) { //for long messages return (smpi_coll_tuned_allreduce_rab_rdb (sbuf, rbuf, count, dtype, op, comm)); }else { //for short ones and count < pof2 return (smpi_coll_tuned_allreduce_rdb (sbuf, rbuf, count, dtype, op, comm)); } } /* This is the default implementation of alltoall. The algorithm is: Algorithm: MPI_Alltoall We use four algorithms for alltoall. For short messages and (comm_size >= 8), we use the algorithm by Jehoshua Bruck et al, IEEE TPDS, Nov. 1997. It is a store-and-forward algorithm that takes lgp steps. Because of the extra communication, the bandwidth requirement is (n/2).lgp.beta. Cost = lgp.alpha + (n/2).lgp.beta where n is the total amount of data a process needs to send to all other processes. For medium size messages and (short messages for comm_size < 8), we use an algorithm that posts all irecvs and isends and then does a waitall. We scatter the order of sources and destinations among the processes, so that all processes don't try to send/recv to/from the same process at the same time. *** Modification: We post only a small number of isends and irecvs at a time and wait on them as suggested by Tony Ladd. *** *** See comments below about an additional modification that we may want to consider *** For long messages and power-of-two number of processes, we use a pairwise exchange algorithm, which takes p-1 steps. We calculate the pairs by using an exclusive-or algorithm: for (i=1; i=8 -> bruck // medium size messages and (short messages for comm_size < 8), we // use an algorithm that posts all irecvs and isends and then does a // waitall. // For long messages and power-of-two number of processes, we use a // pairwise exchange algorithm // For a non-power-of-two number of processes, we use an // algorithm in which, in step i, each process receives from (rank-i) // and sends to (rank+i). dsize = smpi_datatype_size(sdtype); block_dsize = dsize * scount; if ((block_dsize < short_size) && (communicator_size >= 8)) { return smpi_coll_tuned_alltoall_bruck(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } else if (block_dsize < medium_size) { return smpi_coll_tuned_alltoall_basic_linear(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); }else if (communicator_size%2){ return smpi_coll_tuned_alltoall_ring(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } return smpi_coll_tuned_alltoall_ring (sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } int smpi_coll_tuned_alltoallv_mpich(void *sbuf, int *scounts, int *sdisps, MPI_Datatype sdtype, void *rbuf, int *rcounts, int *rdisps, MPI_Datatype rdtype, MPI_Comm comm ) { /* For starters, just keep the original algorithm. */ return smpi_coll_tuned_alltoallv_bruck(sbuf, scounts, sdisps, sdtype, rbuf, rcounts, rdisps,rdtype, comm); } int smpi_coll_tuned_barrier_mpich(MPI_Comm comm) { return smpi_coll_tuned_barrier_ompi_bruck(comm); } /* This is the default implementation of broadcast. The algorithm is: Algorithm: MPI_Bcast For short messages, we use a binomial tree algorithm. Cost = lgp.alpha + n.lgp.beta For long messages, we do a scatter followed by an allgather. We first scatter the buffer using a binomial tree algorithm. This costs lgp.alpha + n.((p-1)/p).beta If the datatype is contiguous and the communicator is homogeneous, we treat the data as bytes and divide (scatter) it among processes by using ceiling division. For the noncontiguous or heterogeneous cases, we first pack the data into a temporary buffer by using MPI_Pack, scatter it as bytes, and unpack it after the allgather. For the allgather, we use a recursive doubling algorithm for medium-size messages and power-of-two number of processes. This takes lgp steps. In each step pairs of processes exchange all the data they have (we take care of non-power-of-two situations). This costs approximately lgp.alpha + n.((p-1)/p).beta. (Approximately because it may be slightly more in the non-power-of-two case, but it's still a logarithmic algorithm.) Therefore, for long messages Total Cost = 2.lgp.alpha + 2.n.((p-1)/p).beta Note that this algorithm has twice the latency as the tree algorithm we use for short messages, but requires lower bandwidth: 2.n.beta versus n.lgp.beta. Therefore, for long messages and when lgp > 2, this algorithm will perform better. For long messages and for medium-size messages and non-power-of-two processes, we use a ring algorithm for the allgather, which takes p-1 steps, because it performs better than recursive doubling. Total Cost = (lgp+p-1).alpha + 2.n.((p-1)/p).beta Possible improvements: For clusters of SMPs, we may want to do something differently to take advantage of shared memory on each node. End Algorithm: MPI_Bcast */ int smpi_coll_tuned_bcast_mpich(void *buff, int count, MPI_Datatype datatype, int root, MPI_Comm comm ) { /* Decision function based on MX results for messages up to 36MB and communicator sizes up to 64 nodes */ const size_t small_message_size = 12288; const size_t intermediate_message_size = 524288; int communicator_size; //int segsize = 0; size_t message_size, dsize; communicator_size = smpi_comm_size(comm); /* else we need data size for decision function */ dsize = smpi_datatype_size(datatype); message_size = dsize * (unsigned long)count; /* needed for decision */ /* Handle messages of small and intermediate size, and single-element broadcasts */ if ((message_size < small_message_size) || (communicator_size <= 8)) { /* Binomial without segmentation */ return smpi_coll_tuned_bcast_binomial_tree (buff, count, datatype, root, comm); } else if (message_size < intermediate_message_size && !(communicator_size%2)) { // SplittedBinary with 1KB segments return smpi_coll_tuned_bcast_scatter_rdb_allgather(buff, count, datatype, root, comm); } //Handle large message sizes return smpi_coll_tuned_bcast_scatter_LR_allgather (buff, count, datatype, root, comm); } /* This is the default implementation of reduce. The algorithm is: Algorithm: MPI_Reduce For long messages and for builtin ops and if count >= pof2 (where pof2 is the nearest power-of-two less than or equal to the number of processes), we use Rabenseifner's algorithm (see http://www.hlrs.de/organization/par/services/models/mpi/myreduce.html ). This algorithm implements the reduce in two steps: first a reduce-scatter, followed by a gather to the root. A recursive-halving algorithm (beginning with processes that are distance 1 apart) is used for the reduce-scatter, and a binomial tree algorithm is used for the gather. The non-power-of-two case is handled by dropping to the nearest lower power-of-two: the first few odd-numbered processes send their data to their left neighbors (rank-1), and the reduce-scatter happens among the remaining power-of-two processes. If the root is one of the excluded processes, then after the reduce-scatter, rank 0 sends its result to the root and exits; the root now acts as rank 0 in the binomial tree algorithm for gather. For the power-of-two case, the cost for the reduce-scatter is lgp.alpha + n.((p-1)/p).beta + n.((p-1)/p).gamma. The cost for the gather to root is lgp.alpha + n.((p-1)/p).beta. Therefore, the total cost is: Cost = 2.lgp.alpha + 2.n.((p-1)/p).beta + n.((p-1)/p).gamma For the non-power-of-two case, assuming the root is not one of the odd-numbered processes that get excluded in the reduce-scatter, Cost = (2.floor(lgp)+1).alpha + (2.((p-1)/p) + 1).n.beta + n.(1+(p-1)/p).gamma For short messages, user-defined ops, and count < pof2, we use a binomial tree algorithm for both short and long messages. Cost = lgp.alpha + n.lgp.beta + n.lgp.gamma We use the binomial tree algorithm in the case of user-defined ops because in this case derived datatypes are allowed, and the user could pass basic datatypes on one process and derived on another as long as the type maps are the same. Breaking up derived datatypes to do the reduce-scatter is tricky. FIXME: Per the MPI-2.1 standard this case is not possible. We should be able to use the reduce-scatter/gather approach as long as count >= pof2. [goodell@ 2009-01-21] Possible improvements: End Algorithm: MPI_Reduce */ int smpi_coll_tuned_reduce_mpich( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm ) { int communicator_size=0; //int segsize = 0; size_t message_size, dsize; communicator_size = smpi_comm_size(comm); /* need data size for decision function */ dsize=smpi_datatype_size(datatype); message_size = dsize * count; /* needed for decision */ int pof2 = 1; while (pof2 <= communicator_size) pof2 <<= 1; pof2 >>= 1; if ((count < pof2) || (message_size < 2048) || !smpi_op_is_commute(op)) { return smpi_coll_tuned_reduce_binomial (sendbuf, recvbuf, count, datatype, op, root, comm); } return smpi_coll_tuned_reduce_scatter_gather(sendbuf, recvbuf, count, datatype, op, root, comm/*, module, segsize, max_requests*/); } /* This is the default implementation of reduce_scatter. The algorithm is: Algorithm: MPI_Reduce_scatter If the operation is commutative, for short and medium-size messages, we use a recursive-halving algorithm in which the first p/2 processes send the second n/2 data to their counterparts in the other half and receive the first n/2 data from them. This procedure continues recursively, halving the data communicated at each step, for a total of lgp steps. If the number of processes is not a power-of-two, we convert it to the nearest lower power-of-two by having the first few even-numbered processes send their data to the neighboring odd-numbered process at (rank+1). Those odd-numbered processes compute the result for their left neighbor as well in the recursive halving algorithm, and then at the end send the result back to the processes that didn't participate. Therefore, if p is a power-of-two, Cost = lgp.alpha + n.((p-1)/p).beta + n.((p-1)/p).gamma If p is not a power-of-two, Cost = (floor(lgp)+2).alpha + n.(1+(p-1+n)/p).beta + n.(1+(p-1)/p).gamma The above cost in the non power-of-two case is approximate because there is some imbalance in the amount of work each process does because some processes do the work of their neighbors as well. For commutative operations and very long messages we use we use a pairwise exchange algorithm similar to the one used in MPI_Alltoall. At step i, each process sends n/p amount of data to (rank+i) and receives n/p amount of data from (rank-i). Cost = (p-1).alpha + n.((p-1)/p).beta + n.((p-1)/p).gamma If the operation is not commutative, we do the following: We use a recursive doubling algorithm, which takes lgp steps. At step 1, processes exchange (n-n/p) amount of data; at step 2, (n-2n/p) amount of data; at step 3, (n-4n/p) amount of data, and so forth. Cost = lgp.alpha + n.(lgp-(p-1)/p).beta + n.(lgp-(p-1)/p).gamma Possible improvements: End Algorithm: MPI_Reduce_scatter */ int smpi_coll_tuned_reduce_scatter_mpich( void *sbuf, void *rbuf, int *rcounts, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm ) { int comm_size, i; size_t total_message_size; if(sbuf==rbuf)sbuf=MPI_IN_PLACE; //restore MPI_IN_PLACE as these algorithms handle it XBT_DEBUG("smpi_coll_tuned_reduce_scatter_mpich"); comm_size = smpi_comm_size(comm); // We need data size for decision function total_message_size = 0; for (i = 0; i < comm_size; i++) { total_message_size += rcounts[i]; } if( smpi_op_is_commute(op) && total_message_size > 524288) { return smpi_coll_tuned_reduce_scatter_mpich_pair (sbuf, rbuf, rcounts, dtype, op, comm); }else if (!smpi_op_is_commute(op)) { int is_block_regular = 1; for (i = 0; i < (comm_size - 1); ++i) { if (rcounts[i] != rcounts[i+1]) { is_block_regular = 0; break; } } /* slightly retask pof2 to mean pof2 equal or greater, not always greater as it is above */ int pof2 = 1; while (pof2 < comm_size) pof2 <<= 1; if (pof2 == comm_size && is_block_regular) { /* noncommutative, pof2 size, and block regular */ return smpi_coll_tuned_reduce_scatter_mpich_noncomm(sbuf, rbuf, rcounts, dtype, op, comm); } return smpi_coll_tuned_reduce_scatter_mpich_rdb(sbuf, rbuf, rcounts, dtype, op, comm); }else{ return smpi_coll_tuned_reduce_scatter_mpich_rdb(sbuf, rbuf, rcounts, dtype, op, comm); } } /* This is the default implementation of allgather. The algorithm is: Algorithm: MPI_Allgather For short messages and non-power-of-two no. of processes, we use the algorithm from the Jehoshua Bruck et al IEEE TPDS Nov 97 paper. It is a variant of the disemmination algorithm for barrier. It takes ceiling(lg p) steps. Cost = lgp.alpha + n.((p-1)/p).beta where n is total size of data gathered on each process. For short or medium-size messages and power-of-two no. of processes, we use the recursive doubling algorithm. Cost = lgp.alpha + n.((p-1)/p).beta TODO: On TCP, we may want to use recursive doubling instead of the Bruck algorithm in all cases because of the pairwise-exchange property of recursive doubling (see Benson et al paper in Euro PVM/MPI 2003). It is interesting to note that either of the above algorithms for MPI_Allgather has the same cost as the tree algorithm for MPI_Gather! For long messages or medium-size messages and non-power-of-two no. of processes, we use a ring algorithm. In the first step, each process i sends its contribution to process i+1 and receives the contribution from process i-1 (with wrap-around). From the second step onwards, each process i forwards to process i+1 the data it received from process i-1 in the previous step. This takes a total of p-1 steps. Cost = (p-1).alpha + n.((p-1)/p).beta We use this algorithm instead of recursive doubling for long messages because we find that this communication pattern (nearest neighbor) performs twice as fast as recursive doubling for long messages (on Myrinet and IBM SP). Possible improvements: End Algorithm: MPI_Allgather */ int smpi_coll_tuned_allgather_mpich(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, MPI_Comm comm ) { int communicator_size, pow2_size; size_t dsize, total_dsize; communicator_size = smpi_comm_size(comm); /* Determine complete data size */ dsize=smpi_datatype_size(sdtype); total_dsize = dsize * scount * communicator_size; for (pow2_size = 1; pow2_size < communicator_size; pow2_size <<=1); /* Decision as in MPICH-2 presented in Thakur et.al. "Optimization of Collective Communication Operations in MPICH", International Journal of High Performance Computing Applications, Vol. 19, No. 1, 49-66 (2005) - for power-of-two processes and small and medium size messages (up to 512KB) use recursive doubling - for non-power-of-two processes and small messages (80KB) use bruck, - for everything else use ring. */ if ((pow2_size == communicator_size) && (total_dsize < 524288)) { return smpi_coll_tuned_allgather_rdb(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } else if (total_dsize <= 81920) { return smpi_coll_tuned_allgather_bruck(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } return smpi_coll_tuned_allgather_ring(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } /* This is the default implementation of allgatherv. The algorithm is: Algorithm: MPI_Allgatherv For short messages and non-power-of-two no. of processes, we use the algorithm from the Jehoshua Bruck et al IEEE TPDS Nov 97 paper. It is a variant of the disemmination algorithm for barrier. It takes ceiling(lg p) steps. Cost = lgp.alpha + n.((p-1)/p).beta where n is total size of data gathered on each process. For short or medium-size messages and power-of-two no. of processes, we use the recursive doubling algorithm. Cost = lgp.alpha + n.((p-1)/p).beta TODO: On TCP, we may want to use recursive doubling instead of the Bruck algorithm in all cases because of the pairwise-exchange property of recursive doubling (see Benson et al paper in Euro PVM/MPI 2003). For long messages or medium-size messages and non-power-of-two no. of processes, we use a ring algorithm. In the first step, each process i sends its contribution to process i+1 and receives the contribution from process i-1 (with wrap-around). From the second step onwards, each process i forwards to process i+1 the data it received from process i-1 in the previous step. This takes a total of p-1 steps. Cost = (p-1).alpha + n.((p-1)/p).beta Possible improvements: End Algorithm: MPI_Allgatherv */ int smpi_coll_tuned_allgatherv_mpich(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int *rcounts, int *rdispls, MPI_Datatype rdtype, MPI_Comm comm ) { int communicator_size, pow2_size,i; size_t total_dsize; communicator_size = smpi_comm_size(comm); /* Determine complete data size */ total_dsize = 0; for (i=0; i>= 1; remain = size - tmp_size; /* If comm size is not a power of two, have the first "remain" procs with an even rank send to rank + 1, leaving a power of two procs to do the rest of the algorithm */ if (rank < 2 * remain) { if ((rank & 1) == 0) { smpi_mpi_send(result_buf, count, dtype, rank + 1, COLL_TAG_REDUCE_SCATTER, comm); /* we don't participate from here on out */ tmp_rank = -1; } else { smpi_mpi_recv(recv_buf, count, dtype, rank - 1, COLL_TAG_REDUCE_SCATTER, comm, MPI_STATUS_IGNORE); /* integrate their results into our temp results */ smpi_op_apply(op, recv_buf, result_buf, &count, &dtype); /* adjust rank to be the bottom "remain" ranks */ tmp_rank = rank / 2; } } else { /* just need to adjust rank to show that the bottom "even remain" ranks dropped out */ tmp_rank = rank - remain; } /* For ranks not kicked out by the above code, perform the recursive halving */ if (tmp_rank >= 0) { int *tmp_disps = NULL, *tmp_rcounts = NULL; int mask, send_index, recv_index, last_index; /* recalculate disps and rcounts to account for the special "remainder" processes that are no longer doing anything */ tmp_rcounts = (int*) xbt_malloc(tmp_size * sizeof(int)); if (NULL == tmp_rcounts) { err = MPI_ERR_OTHER; goto cleanup; } tmp_disps = (int*) xbt_malloc(tmp_size * sizeof(int)); if (NULL == tmp_disps) { xbt_free(tmp_rcounts); err = MPI_ERR_OTHER; goto cleanup; } for (i = 0 ; i < tmp_size ; ++i) { if (i < remain) { /* need to include old neighbor as well */ tmp_rcounts[i] = rcounts[i * 2 + 1] + rcounts[i * 2]; } else { tmp_rcounts[i] = rcounts[i + remain]; } } tmp_disps[0] = 0; for (i = 0; i < tmp_size - 1; ++i) { tmp_disps[i + 1] = tmp_disps[i] + tmp_rcounts[i]; } /* do the recursive halving communication. Don't use the dimension information on the communicator because I think the information is invalidated by our "shrinking" of the communicator */ mask = tmp_size >> 1; send_index = recv_index = 0; last_index = tmp_size; while (mask > 0) { int tmp_peer, peer, send_count, recv_count; MPI_Request request; tmp_peer = tmp_rank ^ mask; peer = (tmp_peer < remain) ? tmp_peer * 2 + 1 : tmp_peer + remain; /* figure out if we're sending, receiving, or both */ send_count = recv_count = 0; if (tmp_rank < tmp_peer) { send_index = recv_index + mask; for (i = send_index ; i < last_index ; ++i) { send_count += tmp_rcounts[i]; } for (i = recv_index ; i < send_index ; ++i) { recv_count += tmp_rcounts[i]; } } else { recv_index = send_index + mask; for (i = send_index ; i < recv_index ; ++i) { send_count += tmp_rcounts[i]; } for (i = recv_index ; i < last_index ; ++i) { recv_count += tmp_rcounts[i]; } } /* actual data transfer. Send from result_buf, receive into recv_buf */ if (send_count > 0 && recv_count != 0) { request=smpi_mpi_irecv(recv_buf + (ptrdiff_t)tmp_disps[recv_index] * extent, recv_count, dtype, peer, COLL_TAG_REDUCE_SCATTER, comm); if (MPI_SUCCESS != err) { xbt_free(tmp_rcounts); xbt_free(tmp_disps); goto cleanup; } } if (recv_count > 0 && send_count != 0) { smpi_mpi_send(result_buf + (ptrdiff_t)tmp_disps[send_index] * extent, send_count, dtype, peer, COLL_TAG_REDUCE_SCATTER, comm); if (MPI_SUCCESS != err) { xbt_free(tmp_rcounts); xbt_free(tmp_disps); goto cleanup; } } if (send_count > 0 && recv_count != 0) { smpi_mpi_wait(&request, MPI_STATUS_IGNORE); } /* if we received something on this step, push it into the results buffer */ if (recv_count > 0) { smpi_op_apply(op, recv_buf + (ptrdiff_t)tmp_disps[recv_index] * extent, result_buf + (ptrdiff_t)tmp_disps[recv_index] * extent, &recv_count, &dtype); } /* update for next iteration */ send_index = recv_index; last_index = recv_index + mask; mask >>= 1; } /* copy local results from results buffer into real receive buffer */ if (0 != rcounts[rank]) { err = smpi_datatype_copy(result_buf + disps[rank] * extent, rcounts[rank], dtype, rbuf, rcounts[rank], dtype); if (MPI_SUCCESS != err) { xbt_free(tmp_rcounts); xbt_free(tmp_disps); goto cleanup; } } xbt_free(tmp_rcounts); xbt_free(tmp_disps); } /* Now fix up the non-power of two case, by having the odd procs send the even procs the proper results */ if (rank < (2 * remain)) { if ((rank & 1) == 0) { if (rcounts[rank]) { smpi_mpi_recv(rbuf, rcounts[rank], dtype, rank + 1, COLL_TAG_REDUCE_SCATTER, comm, MPI_STATUS_IGNORE); } } else { if (rcounts[rank - 1]) { smpi_mpi_send(result_buf + disps[rank - 1] * extent, rcounts[rank - 1], dtype, rank - 1, COLL_TAG_REDUCE_SCATTER, comm); } } } cleanup: if (NULL != disps) xbt_free(disps); if (NULL != recv_buf_free) smpi_free_tmp_buffer(recv_buf_free); if (NULL != result_buf_free) smpi_free_tmp_buffer(result_buf_free); return err; } /* copied function (with appropriate renaming) ends here */ /* * smpi_coll_tuned_reduce_scatter_ompi_ring * * Function: Ring algorithm for reduce_scatter operation * Accepts: Same as MPI_Reduce_scatter() * Returns: MPI_SUCCESS or error code * * Description: Implements ring algorithm for reduce_scatter: * the block sizes defined in rcounts are exchanged and 8 updated until they reach proper destination. * Algorithm requires 2 * max(rcounts) extra buffering * * Limitations: The algorithm DOES NOT preserve order of operations so it * can be used only for commutative operations. * Example on 5 nodes: * Initial state * # 0 1 2 3 4 * [00] [10] -> [20] [30] [40] * [01] [11] [21] -> [31] [41] * [02] [12] [22] [32] -> [42] * -> [03] [13] [23] [33] [43] --> .. * [04] -> [14] [24] [34] [44] * * COMPUTATION PHASE * Step 0: rank r sends block (r-1) to rank (r+1) and * receives block (r+1) from rank (r-1) [with wraparound]. * # 0 1 2 3 4 * [00] [10] [10+20] -> [30] [40] * [01] [11] [21] [21+31] -> [41] * -> [02] [12] [22] [32] [32+42] -->.. * [43+03] -> [13] [23] [33] [43] * [04] [04+14] -> [24] [34] [44] * * Step 1: * # 0 1 2 3 4 * [00] [10] [10+20] [10+20+30] -> [40] * -> [01] [11] [21] [21+31] [21+31+41] -> * [32+42+02] -> [12] [22] [32] [32+42] * [03] [43+03+13] -> [23] [33] [43] * [04] [04+14] [04+14+24] -> [34] [44] * * Step 2: * # 0 1 2 3 4 * -> [00] [10] [10+20] [10+20+30] [10+20+30+40] -> * [21+31+41+01]-> [11] [21] [21+31] [21+31+41] * [32+42+02] [32+42+02+12]-> [22] [32] [32+42] * [03] [43+03+13] [43+03+13+23]-> [33] [43] * [04] [04+14] [04+14+24] [04+14+24+34] -> [44] * * Step 3: * # 0 1 2 3 4 * [10+20+30+40+00] [10] [10+20] [10+20+30] [10+20+30+40] * [21+31+41+01] [21+31+41+01+11] [21] [21+31] [21+31+41] * [32+42+02] [32+42+02+12] [32+42+02+12+22] [32] [32+42] * [03] [43+03+13] [43+03+13+23] [43+03+13+23+33] [43] * [04] [04+14] [04+14+24] [04+14+24+34] [04+14+24+34+44] * DONE :) * */ int smpi_coll_tuned_reduce_scatter_ompi_ring(void *sbuf, void *rbuf, int *rcounts, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm ) { int ret, line, rank, size, i, k, recv_from, send_to, total_count, max_block_count; int inbi, *displs = NULL; char *tmpsend = NULL, *tmprecv = NULL, *accumbuf = NULL, *accumbuf_free = NULL; char *inbuf_free[2] = {NULL, NULL}, *inbuf[2] = {NULL, NULL}; ptrdiff_t true_lb, true_extent, lb, extent, max_real_segsize; MPI_Request reqs[2] = {NULL, NULL}; size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); XBT_DEBUG( "coll:tuned:reduce_scatter_ompi_ring rank %d, size %d", rank, size); /* Determine the maximum number of elements per node, corresponding block size, and displacements array. */ displs = (int*) xbt_malloc(size * sizeof(int)); if (NULL == displs) { ret = -1; line = __LINE__; goto error_hndl; } displs[0] = 0; total_count = rcounts[0]; max_block_count = rcounts[0]; for (i = 1; i < size; i++) { displs[i] = total_count; total_count += rcounts[i]; if (max_block_count < rcounts[i]) max_block_count = rcounts[i]; } /* Special case for size == 1 */ if (1 == size) { if (MPI_IN_PLACE != sbuf) { ret = smpi_datatype_copy((char*)sbuf, total_count, dtype, (char*)rbuf, total_count, dtype); if (ret < 0) { line = __LINE__; goto error_hndl; } } xbt_free(displs); return MPI_SUCCESS; } /* Allocate and initialize temporary buffers, we need: - a temporary buffer to perform reduction (size total_count) since rbuf can be of rcounts[rank] size. - up to two temporary buffers used for communication/computation overlap. */ smpi_datatype_extent(dtype, &lb, &extent); smpi_datatype_extent(dtype, &true_lb, &true_extent); max_real_segsize = true_extent + (ptrdiff_t)(max_block_count - 1) * extent; accumbuf_free = (char*)smpi_get_tmp_recvbuffer(true_extent + (ptrdiff_t)(total_count - 1) * extent); if (NULL == accumbuf_free) { ret = -1; line = __LINE__; goto error_hndl; } accumbuf = accumbuf_free - lb; inbuf_free[0] = (char*)smpi_get_tmp_sendbuffer(max_real_segsize); if (NULL == inbuf_free[0]) { ret = -1; line = __LINE__; goto error_hndl; } inbuf[0] = inbuf_free[0] - lb; if (size > 2) { inbuf_free[1] = (char*)smpi_get_tmp_sendbuffer(max_real_segsize); if (NULL == inbuf_free[1]) { ret = -1; line = __LINE__; goto error_hndl; } inbuf[1] = inbuf_free[1] - lb; } /* Handle MPI_IN_PLACE for size > 1 */ if (MPI_IN_PLACE == sbuf) { sbuf = rbuf; } ret = smpi_datatype_copy((char*)sbuf, total_count, dtype, accumbuf, total_count, dtype); if (ret < 0) { line = __LINE__; goto error_hndl; } /* Computation loop */ /* For each of the remote nodes: - post irecv for block (r-2) from (r-1) with wrap around - send block (r-1) to (r+1) - in loop for every step k = 2 .. n - post irecv for block (r - 1 + n - k) % n - wait on block (r + n - k) % n to arrive - compute on block (r + n - k ) % n - send block (r + n - k) % n - wait on block (r) - compute on block (r) - copy block (r) to rbuf Note that we must be careful when computing the begining of buffers and for send operations and computation we must compute the exact block size. */ send_to = (rank + 1) % size; recv_from = (rank + size - 1) % size; inbi = 0; /* Initialize first receive from the neighbor on the left */ reqs[inbi]=smpi_mpi_irecv(inbuf[inbi], max_block_count, dtype, recv_from, COLL_TAG_REDUCE_SCATTER, comm ); tmpsend = accumbuf + (ptrdiff_t)displs[recv_from] * extent; smpi_mpi_send(tmpsend, rcounts[recv_from], dtype, send_to, COLL_TAG_REDUCE_SCATTER, comm); for (k = 2; k < size; k++) { const int prevblock = (rank + size - k) % size; inbi = inbi ^ 0x1; /* Post irecv for the current block */ reqs[inbi]=smpi_mpi_irecv(inbuf[inbi], max_block_count, dtype, recv_from, COLL_TAG_REDUCE_SCATTER, comm ); /* Wait on previous block to arrive */ smpi_mpi_wait(&reqs[inbi ^ 0x1], MPI_STATUS_IGNORE); /* Apply operation on previous block: result goes to rbuf rbuf[prevblock] = inbuf[inbi ^ 0x1] (op) rbuf[prevblock] */ tmprecv = accumbuf + (ptrdiff_t)displs[prevblock] * extent; smpi_op_apply(op, inbuf[inbi ^ 0x1], tmprecv, &(rcounts[prevblock]), &dtype); /* send previous block to send_to */ smpi_mpi_send(tmprecv, rcounts[prevblock], dtype, send_to, COLL_TAG_REDUCE_SCATTER, comm); } /* Wait on the last block to arrive */ smpi_mpi_wait(&reqs[inbi], MPI_STATUS_IGNORE); /* Apply operation on the last block (my block) rbuf[rank] = inbuf[inbi] (op) rbuf[rank] */ tmprecv = accumbuf + (ptrdiff_t)displs[rank] * extent; smpi_op_apply(op, inbuf[inbi], tmprecv, &(rcounts[rank]), &dtype); /* Copy result from tmprecv to rbuf */ ret = smpi_datatype_copy(tmprecv, rcounts[rank], dtype, (char*)rbuf, rcounts[rank], dtype); if (ret < 0) { line = __LINE__; goto error_hndl; } if (NULL != displs) xbt_free(displs); if (NULL != accumbuf_free) smpi_free_tmp_buffer(accumbuf_free); if (NULL != inbuf_free[0]) smpi_free_tmp_buffer(inbuf_free[0]); if (NULL != inbuf_free[1]) smpi_free_tmp_buffer(inbuf_free[1]); return MPI_SUCCESS; error_hndl: XBT_DEBUG( "%s:%4d\tRank %d Error occurred %d\n", __FILE__, line, rank, ret); if (NULL != displs) xbt_free(displs); if (NULL != accumbuf_free) smpi_free_tmp_buffer(accumbuf_free); if (NULL != inbuf_free[0]) smpi_free_tmp_buffer(inbuf_free[0]); if (NULL != inbuf_free[1]) smpi_free_tmp_buffer(inbuf_free[1]); return ret; } SimGrid-3.14.159/src/smpi/colls/allreduce-smp-rsag-rab.c000644 001750 001750 00000014167 13031031271 024350 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * implemented by Pitch Patarasuk, 07/01/2007 */ #include "colls_private.h" //#include /* This fucntion performs all-reduce operation as follow. 1) binomial_tree reduce inside each SMP node 2) reduce-scatter -inter between root of each SMP node 3) allgather - inter between root of each SMP node 4) binomial_tree bcast inside each SMP node */ int smpi_coll_tuned_allreduce_smp_rsag_rab(void *sbuf, void *rbuf, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { int comm_size, rank; void *tmp_buf; int tag = COLL_TAG_ALLREDUCE; int mask, src, dst; MPI_Status status; if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } int num_core=1; if (smpi_comm_is_uniform(comm)){ num_core = smpi_comm_size(smpi_comm_get_intra_comm(comm)); } comm_size = smpi_comm_size(comm); if((comm_size&(comm_size-1))) THROWF(arg_error,0, "allreduce smp rsag rab algorithm can't be used with non power of two number of processes ! "); rank = smpi_comm_rank(comm); MPI_Aint extent; extent = smpi_datatype_get_extent(dtype); tmp_buf = (void *) smpi_get_tmp_sendbuffer(count * extent); int intra_rank, inter_rank; intra_rank = rank % num_core; inter_rank = rank / num_core; //printf("node %d intra_rank = %d, inter_rank = %d\n", rank, intra_rank, inter_rank); int inter_comm_size = (comm_size + num_core - 1) / num_core; smpi_mpi_sendrecv(sbuf, count, dtype, rank, tag, rbuf, count, dtype, rank, tag, comm, &status); // SMP_binomial_reduce mask = 1; while (mask < num_core) { if ((mask & intra_rank) == 0) { src = (inter_rank * num_core) + (intra_rank | mask); // if (src < ((inter_rank + 1) * num_core)) { if (src < comm_size) { smpi_mpi_recv(tmp_buf, count, dtype, src, tag, comm, &status); smpi_op_apply(op, tmp_buf, rbuf, &count, &dtype); //printf("Node %d recv from node %d when mask is %d\n", rank, src, mask); } } else { dst = (inter_rank * num_core) + (intra_rank & (~mask)); smpi_mpi_send(rbuf, count, dtype, dst, tag, comm); //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); break; } mask <<= 1; } // INTER: reduce-scatter if (intra_rank == 0) { int dst, base_offset, send_base_offset, recv_base_offset, recv_chunk; int curr_count, i, recv_offset, send_offset; // reduce-scatter recv_chunk = extent * count / (comm_size / num_core); mask = 1; curr_count = count / 2; int phase = 0; base_offset = 0; while (mask < (comm_size / num_core)) { dst = inter_rank ^ mask; // compute offsets // right-handside if (inter_rank & mask) { recv_base_offset = base_offset + curr_count; send_base_offset = base_offset; base_offset = recv_base_offset; } // left-handside else { recv_base_offset = base_offset; send_base_offset = base_offset + curr_count; } send_offset = send_base_offset * extent; recv_offset = recv_base_offset * extent; // if (rank==7) // printf("node %d send to %d in phase %d s_offset = %d r_offset = %d count = %d\n",rank,dst,phase, send_offset, recv_offset, curr_count); smpi_mpi_sendrecv((char *)rbuf + send_offset, curr_count, dtype, (dst * num_core), tag, tmp_buf, curr_count, dtype, (dst * num_core), tag, comm, &status); smpi_op_apply(op, tmp_buf, (char *)rbuf + recv_offset, &curr_count, &dtype); mask *= 2; curr_count /= 2; phase++; } // INTER: allgather int size = (comm_size / num_core) / 2; base_offset = 0; mask = 1; while (mask < (comm_size / num_core)) { if (inter_rank & mask) { base_offset += size; } mask <<= 1; size /= 2; } curr_count *= 2; mask >>= 1; i = 1; phase = 0; while (mask >= 1) { // destination pair for both send and recv dst = inter_rank ^ mask; // compute offsets send_base_offset = base_offset; if (inter_rank & mask) { recv_base_offset = base_offset - i; base_offset -= i; } else { recv_base_offset = base_offset + i; } send_offset = send_base_offset * recv_chunk; recv_offset = recv_base_offset * recv_chunk; // if (rank==7) //printf("node %d send to %d in phase %d s_offset = %d r_offset = %d count = %d\n",rank,dst,phase, send_offset, recv_offset, curr_count); smpi_mpi_sendrecv((char *)rbuf + send_offset, curr_count, dtype, (dst * num_core), tag, (char *)rbuf + recv_offset, curr_count, dtype, (dst * num_core), tag, comm, &status); curr_count *= 2; i *= 2; mask >>= 1; phase++; } } // INTER // intra SMP binomial bcast int num_core_in_current_smp = num_core; if (inter_rank == (inter_comm_size - 1)) { num_core_in_current_smp = comm_size - (inter_rank * num_core); } // printf("Node %d num_core = %d\n",rank, num_core_in_current_smp); mask = 1; while (mask < num_core_in_current_smp) { if (intra_rank & mask) { src = (inter_rank * num_core) + (intra_rank - mask); //printf("Node %d recv from node %d when mask is %d\n", rank, src, mask); smpi_mpi_recv(rbuf, count, dtype, src, tag, comm, &status); break; } mask <<= 1; } mask >>= 1; //printf("My rank = %d my mask = %d\n", rank,mask); while (mask > 0) { dst = (inter_rank * num_core) + (intra_rank + mask); if (dst < comm_size) { //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); smpi_mpi_send(rbuf, count, dtype, dst, tag, comm); } mask >>= 1; } smpi_free_tmp_buffer(tmp_buf); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/bcast-NTSL.c000644 001750 001750 00000011207 13031031271 021761 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" static int bcast_NTSL_segment_size_in_byte = 8192; /* Non-topology-specific pipelined linear-bcast function 0->1, 1->2 ,2->3, ....., ->last node : in a pipeline fashion */ int smpi_coll_tuned_bcast_NTSL(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int tag = COLL_TAG_BCAST; MPI_Status status; MPI_Request request; MPI_Request *send_request_array; MPI_Request *recv_request_array; MPI_Status *send_status_array; MPI_Status *recv_status_array; int rank, size; int i; MPI_Aint extent; extent = smpi_datatype_get_extent(datatype); rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); /* source node and destination nodes (same through out the functions) */ int to = (rank + 1) % size; int from = (rank + size - 1) % size; /* segment is segment size in number of elements (not bytes) */ int segment = extent == 0 ? 1 : (bcast_NTSL_segment_size_in_byte / extent); segment = segment == 0 ? 1 :segment; /* pipeline length */ int pipe_length = count / segment; /* use for buffer offset for sending and receiving data = segment size in byte */ int increment = segment * extent; /* if the input size is not divisible by segment size => the small remainder will be done with native implementation */ int remainder = count % segment; /* if root is not zero send to rank zero first this can be modified to make it faster by using logical src, dst. */ if (root != 0) { if (rank == root) { smpi_mpi_send(buf, count, datatype, 0, tag, comm); } else if (rank == 0) { smpi_mpi_recv(buf, count, datatype, root, tag, comm, &status); } } /* when a message is smaller than a block size => no pipeline */ if (count <= segment) { if (rank == 0) { smpi_mpi_send(buf, count, datatype, to, tag, comm); } else if (rank == (size - 1)) { request = smpi_mpi_irecv(buf, count, datatype, from, tag, comm); smpi_mpi_wait(&request, &status); } else { request = smpi_mpi_irecv(buf, count, datatype, from, tag, comm); smpi_mpi_wait(&request, &status); smpi_mpi_send(buf, count, datatype, to, tag, comm); } return MPI_SUCCESS; } /* pipeline bcast */ else { send_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); recv_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); send_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); recv_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); /* root send data */ if (rank == 0) { for (i = 0; i < pipe_length; i++) { send_request_array[i] = smpi_mpi_isend((char *) buf + (i * increment), segment, datatype, to, (tag + i), comm); } smpi_mpi_waitall((pipe_length), send_request_array, send_status_array); } /* last node only receive data */ else if (rank == (size - 1)) { for (i = 0; i < pipe_length; i++) { recv_request_array[i] = smpi_mpi_irecv((char *) buf + (i * increment), segment, datatype, from, (tag + i), comm); } smpi_mpi_waitall((pipe_length), recv_request_array, recv_status_array); } /* intermediate nodes relay (receive, then send) data */ else { for (i = 0; i < pipe_length; i++) { recv_request_array[i] = smpi_mpi_irecv((char *) buf + (i * increment), segment, datatype, from, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { smpi_mpi_wait(&recv_request_array[i], &status); send_request_array[i] = smpi_mpi_isend((char *) buf + (i * increment), segment, datatype, to, (tag + i), comm); } smpi_mpi_waitall((pipe_length), send_request_array, send_status_array); } free(send_request_array); free(recv_request_array); free(send_status_array); free(recv_status_array); } /* end pipeline */ /* when count is not divisible by block size, use default BCAST for the remainder */ if ((remainder != 0) && (count > segment)) { XBT_WARN("MPI_bcast_arrival_NTSL use default MPI_bcast."); smpi_mpi_bcast((char *) buf + (pipe_length * increment), remainder, datatype, root, comm); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/colls_global.c000644 001750 001750 00000000646 13031031272 022551 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "xbt.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_colls, smpi, "Logging specific to SMPI collectives"); SimGrid-3.14.159/src/smpi/colls/smpi_mvapich2_selector.c000644 001750 001750 00000105216 13031031272 024555 0ustar00mquinsonmquinson000000 000000 /* selector for collective algorithms based on mvapich decision logic */ /* Copyright (c) 2009-2010, 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" #include "smpi_mvapich2_selector_stampede.h" int smpi_coll_tuned_alltoall_mvapich2( void *sendbuf, int sendcount, MPI_Datatype sendtype, void* recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm) { if(mv2_alltoall_table_ppn_conf==NULL) init_mv2_alltoall_tables_stampede(); int sendtype_size, recvtype_size, nbytes, comm_size; char * tmp_buf = NULL; int mpi_errno=MPI_SUCCESS; int range = 0; int range_threshold = 0; int conf_index = 0; comm_size = smpi_comm_size(comm); sendtype_size=smpi_datatype_size(sendtype); recvtype_size=smpi_datatype_size(recvtype); nbytes = sendtype_size * sendcount; /* check if safe to use partial subscription mode */ /* Search for the corresponding system size inside the tuning table */ while ((range < (mv2_size_alltoall_tuning_table[conf_index] - 1)) && (comm_size > mv2_alltoall_thresholds_table[conf_index][range].numproc)) { range++; } /* Search for corresponding inter-leader function */ while ((range_threshold < (mv2_alltoall_thresholds_table[conf_index][range].size_table - 1)) && (nbytes > mv2_alltoall_thresholds_table[conf_index][range].algo_table[range_threshold].max) && (mv2_alltoall_thresholds_table[conf_index][range].algo_table[range_threshold].max != -1)) { range_threshold++; } MV2_Alltoall_function = mv2_alltoall_thresholds_table[conf_index][range].algo_table[range_threshold] .MV2_pt_Alltoall_function; if(sendbuf != MPI_IN_PLACE) { mpi_errno = MV2_Alltoall_function(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm); } else { range_threshold = 0; if(nbytes < mv2_alltoall_thresholds_table[conf_index][range].in_place_algo_table[range_threshold].min ||nbytes > mv2_alltoall_thresholds_table[conf_index][range].in_place_algo_table[range_threshold].max ) { tmp_buf = (char *)smpi_get_tmp_sendbuffer( comm_size * recvcount * recvtype_size ); mpi_errno = smpi_datatype_copy((char *)recvbuf, comm_size*recvcount, recvtype, (char *)tmp_buf, comm_size*recvcount, recvtype); mpi_errno = MV2_Alltoall_function(tmp_buf, recvcount, recvtype, recvbuf, recvcount, recvtype, comm ); smpi_free_tmp_buffer(tmp_buf); } else { mpi_errno = MPIR_Alltoall_inplace_MV2(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm ); } } return (mpi_errno); } int smpi_coll_tuned_allgather_mvapich2(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm) { int mpi_errno = MPI_SUCCESS; int nbytes = 0, comm_size, recvtype_size; int range = 0; int partial_sub_ok = 0; int conf_index = 0; int range_threshold = 0; int is_two_level = 0; int local_size = -1; MPI_Comm shmem_comm; //MPI_Comm *shmem_commptr=NULL; /* Get the size of the communicator */ comm_size = smpi_comm_size(comm); recvtype_size=smpi_datatype_size(recvtype); nbytes = recvtype_size * recvcount; if(mv2_allgather_table_ppn_conf==NULL) init_mv2_allgather_tables_stampede(); if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } int i; if (smpi_comm_is_uniform(comm)){ shmem_comm = smpi_comm_get_intra_comm(comm); local_size = smpi_comm_size(shmem_comm); i = 0; if (mv2_allgather_table_ppn_conf[0] == -1) { // Indicating user defined tuning conf_index = 0; goto conf_check_end; } do { if (local_size == mv2_allgather_table_ppn_conf[i]) { conf_index = i; partial_sub_ok = 1; break; } i++; } while(i < mv2_allgather_num_ppn_conf); } conf_check_end: if (partial_sub_ok != 1) { conf_index = 0; } /* Search for the corresponding system size inside the tuning table */ while ((range < (mv2_size_allgather_tuning_table[conf_index] - 1)) && (comm_size > mv2_allgather_thresholds_table[conf_index][range].numproc)) { range++; } /* Search for corresponding inter-leader function */ while ((range_threshold < (mv2_allgather_thresholds_table[conf_index][range].size_inter_table - 1)) && (nbytes > mv2_allgather_thresholds_table[conf_index][range].inter_leader[range_threshold].max) && (mv2_allgather_thresholds_table[conf_index][range].inter_leader[range_threshold].max != -1)) { range_threshold++; } /* Set inter-leader pt */ MV2_Allgather_function = mv2_allgather_thresholds_table[conf_index][range].inter_leader[range_threshold]. MV2_pt_Allgather_function; is_two_level = mv2_allgather_thresholds_table[conf_index][range].two_level[range_threshold]; /* intracommunicator */ if(is_two_level ==1){ if(partial_sub_ok ==1){ if (smpi_comm_is_blocked(comm)){ mpi_errno = MPIR_2lvl_Allgather_MV2(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm); }else{ mpi_errno = smpi_coll_tuned_allgather_mpich(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm); } } else { mpi_errno = MPIR_Allgather_RD_MV2(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm); } } else if(MV2_Allgather_function == &MPIR_Allgather_Bruck_MV2 || MV2_Allgather_function == &MPIR_Allgather_RD_MV2 || MV2_Allgather_function == &MPIR_Allgather_Ring_MV2) { mpi_errno = MV2_Allgather_function(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm); }else{ return MPI_ERR_OTHER; } return mpi_errno; } int smpi_coll_tuned_gather_mvapich2(void *sendbuf, int sendcnt, MPI_Datatype sendtype, void *recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm) { if(mv2_gather_thresholds_table==NULL) init_mv2_gather_tables_stampede(); int mpi_errno = MPI_SUCCESS; int range = 0; int range_threshold = 0; int range_intra_threshold = 0; int nbytes = 0; int comm_size = 0; int recvtype_size, sendtype_size; int rank = -1; comm_size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); if (rank == root) { recvtype_size=smpi_datatype_size(recvtype); nbytes = recvcnt * recvtype_size; } else { sendtype_size=smpi_datatype_size(sendtype); nbytes = sendcnt * sendtype_size; } /* Search for the corresponding system size inside the tuning table */ while ((range < (mv2_size_gather_tuning_table - 1)) && (comm_size > mv2_gather_thresholds_table[range].numproc)) { range++; } /* Search for corresponding inter-leader function */ while ((range_threshold < (mv2_gather_thresholds_table[range].size_inter_table - 1)) && (nbytes > mv2_gather_thresholds_table[range].inter_leader[range_threshold].max) && (mv2_gather_thresholds_table[range].inter_leader[range_threshold].max != -1)) { range_threshold++; } /* Search for corresponding intra node function */ while ((range_intra_threshold < (mv2_gather_thresholds_table[range].size_intra_table - 1)) && (nbytes > mv2_gather_thresholds_table[range].intra_node[range_intra_threshold].max) && (mv2_gather_thresholds_table[range].intra_node[range_intra_threshold].max != -1)) { range_intra_threshold++; } if (smpi_comm_is_blocked(comm) ) { // Set intra-node function pt for gather_two_level MV2_Gather_intra_node_function = mv2_gather_thresholds_table[range].intra_node[range_intra_threshold]. MV2_pt_Gather_function; //Set inter-leader pt MV2_Gather_inter_leader_function = mv2_gather_thresholds_table[range].inter_leader[range_threshold]. MV2_pt_Gather_function; // We call Gather function mpi_errno = MV2_Gather_inter_leader_function(sendbuf, sendcnt, sendtype, recvbuf, recvcnt, recvtype, root, comm); } else { // Indeed, direct (non SMP-aware)gather is MPICH one mpi_errno = smpi_coll_tuned_gather_mpich(sendbuf, sendcnt, sendtype, recvbuf, recvcnt, recvtype, root, comm); } return mpi_errno; } int smpi_coll_tuned_allgatherv_mvapich2(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvtype, MPI_Comm comm ) { int mpi_errno = MPI_SUCCESS; int range = 0, comm_size, total_count, recvtype_size, i; int range_threshold = 0; int nbytes = 0; if(mv2_allgatherv_thresholds_table==NULL) init_mv2_allgatherv_tables_stampede(); comm_size = smpi_comm_size(comm); total_count = 0; for (i = 0; i < comm_size; i++) total_count += recvcounts[i]; recvtype_size=smpi_datatype_size(recvtype); nbytes = total_count * recvtype_size; /* Search for the corresponding system size inside the tuning table */ while ((range < (mv2_size_allgatherv_tuning_table - 1)) && (comm_size > mv2_allgatherv_thresholds_table[range].numproc)) { range++; } /* Search for corresponding inter-leader function */ while ((range_threshold < (mv2_allgatherv_thresholds_table[range].size_inter_table - 1)) && (nbytes > comm_size * mv2_allgatherv_thresholds_table[range].inter_leader[range_threshold].max) && (mv2_allgatherv_thresholds_table[range].inter_leader[range_threshold].max != -1)) { range_threshold++; } /* Set inter-leader pt */ MV2_Allgatherv_function = mv2_allgatherv_thresholds_table[range].inter_leader[range_threshold]. MV2_pt_Allgatherv_function; if (MV2_Allgatherv_function == &MPIR_Allgatherv_Rec_Doubling_MV2) { if(!(comm_size & (comm_size - 1))) { mpi_errno = MPIR_Allgatherv_Rec_Doubling_MV2(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm); } else { mpi_errno = MPIR_Allgatherv_Bruck_MV2(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm); } } else { mpi_errno = MV2_Allgatherv_function(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm); } return mpi_errno; } int smpi_coll_tuned_allreduce_mvapich2(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int mpi_errno = MPI_SUCCESS; //int rank = 0, int comm_size = 0; comm_size = smpi_comm_size(comm); //rank = smpi_comm_rank(comm); if (count == 0) { return MPI_SUCCESS; } if (mv2_allreduce_thresholds_table == NULL) init_mv2_allreduce_tables_stampede(); /* check if multiple threads are calling this collective function */ MPI_Aint sendtype_size = 0; int nbytes = 0; int range = 0, range_threshold = 0, range_threshold_intra = 0; int is_two_level = 0; int is_commutative = 0; MPI_Aint true_lb, true_extent; sendtype_size=smpi_datatype_size(datatype); nbytes = count * sendtype_size; smpi_datatype_extent(datatype, &true_lb, &true_extent); //MPI_Op *op_ptr; //is_commutative = smpi_op_is_commute(op); { /* Search for the corresponding system size inside the tuning table */ while ((range < (mv2_size_allreduce_tuning_table - 1)) && (comm_size > mv2_allreduce_thresholds_table[range].numproc)) { range++; } /* Search for corresponding inter-leader function */ /* skip mcast poiters if mcast is not available */ if(mv2_allreduce_thresholds_table[range].mcast_enabled != 1){ while ((range_threshold < (mv2_allreduce_thresholds_table[range].size_inter_table - 1)) && ((mv2_allreduce_thresholds_table[range]. inter_leader[range_threshold].MV2_pt_Allreduce_function == &MPIR_Allreduce_mcst_reduce_redscat_gather_MV2) || (mv2_allreduce_thresholds_table[range]. inter_leader[range_threshold].MV2_pt_Allreduce_function == &MPIR_Allreduce_mcst_reduce_two_level_helper_MV2) )) { range_threshold++; } } while ((range_threshold < (mv2_allreduce_thresholds_table[range].size_inter_table - 1)) && (nbytes > mv2_allreduce_thresholds_table[range].inter_leader[range_threshold].max) && (mv2_allreduce_thresholds_table[range].inter_leader[range_threshold].max != -1)) { range_threshold++; } if(mv2_allreduce_thresholds_table[range].is_two_level_allreduce[range_threshold] == 1){ is_two_level = 1; } /* Search for corresponding intra-node function */ while ((range_threshold_intra < (mv2_allreduce_thresholds_table[range].size_intra_table - 1)) && (nbytes > mv2_allreduce_thresholds_table[range].intra_node[range_threshold_intra].max) && (mv2_allreduce_thresholds_table[range].intra_node[range_threshold_intra].max != -1)) { range_threshold_intra++; } MV2_Allreduce_function = mv2_allreduce_thresholds_table[range].inter_leader[range_threshold] .MV2_pt_Allreduce_function; MV2_Allreduce_intra_function = mv2_allreduce_thresholds_table[range].intra_node[range_threshold_intra] .MV2_pt_Allreduce_function; /* check if mcast is ready, otherwise replace mcast with other algorithm */ if((MV2_Allreduce_function == &MPIR_Allreduce_mcst_reduce_redscat_gather_MV2)|| (MV2_Allreduce_function == &MPIR_Allreduce_mcst_reduce_two_level_helper_MV2)){ { MV2_Allreduce_function = &MPIR_Allreduce_pt2pt_rd_MV2; } if(is_two_level != 1) { MV2_Allreduce_function = &MPIR_Allreduce_pt2pt_rd_MV2; } } if(is_two_level == 1){ // check if shm is ready, if not use other algorithm first if (is_commutative) { if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } mpi_errno = MPIR_Allreduce_two_level_MV2(sendbuf, recvbuf, count, datatype, op, comm); } else { mpi_errno = MPIR_Allreduce_pt2pt_rd_MV2(sendbuf, recvbuf, count, datatype, op, comm); } } else { mpi_errno = MV2_Allreduce_function(sendbuf, recvbuf, count, datatype, op, comm); } } //comm->ch.intra_node_done=0; return (mpi_errno); } int smpi_coll_tuned_alltoallv_mvapich2(void *sbuf, int *scounts, int *sdisps, MPI_Datatype sdtype, void *rbuf, int *rcounts, int *rdisps, MPI_Datatype rdtype, MPI_Comm comm ) { if (sbuf == MPI_IN_PLACE) { return smpi_coll_tuned_alltoallv_ompi_basic_linear(sbuf, scounts, sdisps, sdtype, rbuf, rcounts, rdisps,rdtype, comm); } else /* For starters, just keep the original algorithm. */ return smpi_coll_tuned_alltoallv_ring(sbuf, scounts, sdisps, sdtype, rbuf, rcounts, rdisps,rdtype, comm); } int smpi_coll_tuned_barrier_mvapich2(MPI_Comm comm) { return smpi_coll_tuned_barrier_mvapich2_pair(comm); } int smpi_coll_tuned_bcast_mvapich2(void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int mpi_errno = MPI_SUCCESS; int comm_size/*, rank*/; int two_level_bcast = 1; size_t nbytes = 0; int range = 0; int range_threshold = 0; int range_threshold_intra = 0; int is_homogeneous, is_contig; MPI_Aint type_size; //, position; void *tmp_buf = NULL; MPI_Comm shmem_comm; //MPID_Datatype *dtp; if (count == 0) return MPI_SUCCESS; if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } if(!mv2_bcast_thresholds_table) init_mv2_bcast_tables_stampede(); comm_size = smpi_comm_size(comm); //rank = smpi_comm_rank(comm); is_contig=1; /* if (HANDLE_GET_KIND(datatype) == HANDLE_KIND_BUILTIN)*/ /* is_contig = 1;*/ /* else {*/ /* MPID_Datatype_get_ptr(datatype, dtp);*/ /* is_contig = dtp->is_contig;*/ /* }*/ is_homogeneous = 1; /* MPI_Type_size() might not give the accurate size of the packed * datatype for heterogeneous systems (because of padding, encoding, * etc). On the other hand, MPI_Pack_size() can become very * expensive, depending on the implementation, especially for * heterogeneous systems. We want to use MPI_Type_size() wherever * possible, and MPI_Pack_size() in other places. */ //if (is_homogeneous) { type_size=smpi_datatype_size(datatype); /* } else { MPIR_Pack_size_impl(1, datatype, &type_size); }*/ nbytes = (size_t) (count) * (type_size); /* Search for the corresponding system size inside the tuning table */ while ((range < (mv2_size_bcast_tuning_table - 1)) && (comm_size > mv2_bcast_thresholds_table[range].numproc)) { range++; } /* Search for corresponding inter-leader function */ while ((range_threshold < (mv2_bcast_thresholds_table[range].size_inter_table - 1)) && (nbytes > mv2_bcast_thresholds_table[range].inter_leader[range_threshold].max) && (mv2_bcast_thresholds_table[range].inter_leader[range_threshold].max != -1)) { range_threshold++; } /* Search for corresponding intra-node function */ while ((range_threshold_intra < (mv2_bcast_thresholds_table[range].size_intra_table - 1)) && (nbytes > mv2_bcast_thresholds_table[range].intra_node[range_threshold_intra].max) && (mv2_bcast_thresholds_table[range].intra_node[range_threshold_intra].max != -1)) { range_threshold_intra++; } MV2_Bcast_function = mv2_bcast_thresholds_table[range].inter_leader[range_threshold]. MV2_pt_Bcast_function; MV2_Bcast_intra_node_function = mv2_bcast_thresholds_table[range]. intra_node[range_threshold_intra].MV2_pt_Bcast_function; /* if (mv2_user_bcast_intra == NULL && */ /* MV2_Bcast_intra_node_function == &MPIR_Knomial_Bcast_intra_node_MV2) {*/ /* MV2_Bcast_intra_node_function = &MPIR_Shmem_Bcast_MV2;*/ /* }*/ if (mv2_bcast_thresholds_table[range].inter_leader[range_threshold]. zcpy_pipelined_knomial_factor != -1) { zcpy_knomial_factor = mv2_bcast_thresholds_table[range].inter_leader[range_threshold]. zcpy_pipelined_knomial_factor; } if (mv2_pipelined_zcpy_knomial_factor != -1) { zcpy_knomial_factor = mv2_pipelined_zcpy_knomial_factor; } if(MV2_Bcast_intra_node_function == NULL) { /* if tuning table do not have any intra selection, set func pointer to ** default one for mcast intra node */ MV2_Bcast_intra_node_function = &MPIR_Shmem_Bcast_MV2; } /* Set value of pipeline segment size */ bcast_segment_size = mv2_bcast_thresholds_table[range].bcast_segment_size; /* Set value of inter node knomial factor */ mv2_inter_node_knomial_factor = mv2_bcast_thresholds_table[range].inter_node_knomial_factor; /* Set value of intra node knomial factor */ mv2_intra_node_knomial_factor = mv2_bcast_thresholds_table[range].intra_node_knomial_factor; /* Check if we will use a two level algorithm or not */ two_level_bcast = #if defined(_MCST_SUPPORT_) mv2_bcast_thresholds_table[range].is_two_level_bcast[range_threshold] || comm->ch.is_mcast_ok; #else mv2_bcast_thresholds_table[range].is_two_level_bcast[range_threshold]; #endif if (two_level_bcast == 1) { if (!is_contig || !is_homogeneous) { tmp_buf=(void *)smpi_get_tmp_sendbuffer(nbytes); /* position = 0;*/ /* if (rank == root) {*/ /* mpi_errno =*/ /* MPIR_Pack_impl(buffer, count, datatype, tmp_buf, nbytes, &position);*/ /* if (mpi_errno)*/ /* MPIU_ERR_POP(mpi_errno);*/ /* }*/ } #ifdef CHANNEL_MRAIL_GEN2 if ((mv2_enable_zcpy_bcast == 1) && (&MPIR_Pipelined_Bcast_Zcpy_MV2 == MV2_Bcast_function)) { if (!is_contig || !is_homogeneous) { mpi_errno = MPIR_Pipelined_Bcast_Zcpy_MV2(tmp_buf, nbytes, MPI_BYTE, root, comm); } else { mpi_errno = MPIR_Pipelined_Bcast_Zcpy_MV2(buffer, count, datatype, root, comm); } } else #endif /* defined(CHANNEL_MRAIL_GEN2) */ { shmem_comm = smpi_comm_get_intra_comm(comm); if (!is_contig || !is_homogeneous) { mpi_errno = MPIR_Bcast_tune_inter_node_helper_MV2(tmp_buf, nbytes, MPI_BYTE, root, comm); } else { mpi_errno = MPIR_Bcast_tune_inter_node_helper_MV2(buffer, count, datatype, root, comm); } /* We are now done with the inter-node phase */ root = INTRA_NODE_ROOT; if (!is_contig || !is_homogeneous) { mpi_errno = MV2_Bcast_intra_node_function(tmp_buf, nbytes, MPI_BYTE, root, shmem_comm); } else { mpi_errno = MV2_Bcast_intra_node_function(buffer, count, datatype, root, shmem_comm); } } /* if (!is_contig || !is_homogeneous) {*/ /* if (rank != root) {*/ /* position = 0;*/ /* mpi_errno = MPIR_Unpack_impl(tmp_buf, nbytes, &position, buffer,*/ /* count, datatype);*/ /* }*/ /* }*/ } else { /* We use Knomial for intra node */ MV2_Bcast_intra_node_function = &MPIR_Knomial_Bcast_intra_node_MV2; /* if (mv2_enable_shmem_bcast == 0) {*/ /* Fall back to non-tuned version */ /* MPIR_Bcast_intra_MV2(buffer, count, datatype, root, comm);*/ /* } else {*/ mpi_errno = MV2_Bcast_function(buffer, count, datatype, root, comm); /* }*/ } return mpi_errno; } int smpi_coll_tuned_reduce_mvapich2( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { if(mv2_reduce_thresholds_table == NULL) init_mv2_reduce_tables_stampede(); int mpi_errno = MPI_SUCCESS; int range = 0; int range_threshold = 0; int range_intra_threshold = 0; int is_commutative, pof2; int comm_size = 0; int nbytes = 0; int sendtype_size; int is_two_level = 0; comm_size = smpi_comm_size(comm); sendtype_size=smpi_datatype_size(datatype); nbytes = count * sendtype_size; if (count == 0) return MPI_SUCCESS; is_commutative = smpi_op_is_commute(op); /* find nearest power-of-two less than or equal to comm_size */ for( pof2 = 1; pof2 <= comm_size; pof2 <<= 1 ); pof2 >>=1; /* Search for the corresponding system size inside the tuning table */ while ((range < (mv2_size_reduce_tuning_table - 1)) && (comm_size > mv2_reduce_thresholds_table[range].numproc)) { range++; } /* Search for corresponding inter-leader function */ while ((range_threshold < (mv2_reduce_thresholds_table[range].size_inter_table - 1)) && (nbytes > mv2_reduce_thresholds_table[range].inter_leader[range_threshold].max) && (mv2_reduce_thresholds_table[range].inter_leader[range_threshold].max != -1)) { range_threshold++; } /* Search for corresponding intra node function */ while ((range_intra_threshold < (mv2_reduce_thresholds_table[range].size_intra_table - 1)) && (nbytes > mv2_reduce_thresholds_table[range].intra_node[range_intra_threshold].max) && (mv2_reduce_thresholds_table[range].intra_node[range_intra_threshold].max != -1)) { range_intra_threshold++; } /* Set intra-node function pt for reduce_two_level */ MV2_Reduce_intra_function = mv2_reduce_thresholds_table[range].intra_node[range_intra_threshold]. MV2_pt_Reduce_function; /* Set inter-leader pt */ MV2_Reduce_function = mv2_reduce_thresholds_table[range].inter_leader[range_threshold]. MV2_pt_Reduce_function; if(mv2_reduce_intra_knomial_factor<0) { mv2_reduce_intra_knomial_factor = mv2_reduce_thresholds_table[range].intra_k_degree; } if(mv2_reduce_inter_knomial_factor<0) { mv2_reduce_inter_knomial_factor = mv2_reduce_thresholds_table[range].inter_k_degree; } if(mv2_reduce_thresholds_table[range].is_two_level_reduce[range_threshold] == 1){ is_two_level = 1; } /* We call Reduce function */ if(is_two_level == 1) { if (is_commutative == 1) { if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } mpi_errno = MPIR_Reduce_two_level_helper_MV2(sendbuf, recvbuf, count, datatype, op, root, comm); } else { mpi_errno = MPIR_Reduce_binomial_MV2(sendbuf, recvbuf, count, datatype, op, root, comm); } } else if(MV2_Reduce_function == &MPIR_Reduce_inter_knomial_wrapper_MV2 ){ if(is_commutative ==1) { mpi_errno = MV2_Reduce_function(sendbuf, recvbuf, count, datatype, op, root, comm); } else { mpi_errno = MPIR_Reduce_binomial_MV2(sendbuf, recvbuf, count, datatype, op, root, comm); } } else if(MV2_Reduce_function == &MPIR_Reduce_redscat_gather_MV2){ if (/*(HANDLE_GET_KIND(op) == HANDLE_KIND_BUILTIN) &&*/ (count >= pof2)) { mpi_errno = MV2_Reduce_function(sendbuf, recvbuf, count, datatype, op, root, comm); } else { mpi_errno = MPIR_Reduce_binomial_MV2(sendbuf, recvbuf, count, datatype, op, root, comm); } } else { mpi_errno = MV2_Reduce_function(sendbuf, recvbuf, count, datatype, op, root, comm); } return mpi_errno; } int smpi_coll_tuned_reduce_scatter_mvapich2(void *sendbuf, void *recvbuf, int *recvcnts, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int mpi_errno = MPI_SUCCESS; int i = 0, comm_size = smpi_comm_size(comm), total_count = 0, type_size = 0, nbytes = 0; int range = 0; int range_threshold = 0; int is_commutative = 0; int *disps = xbt_malloc(comm_size * sizeof (int)); if(mv2_red_scat_thresholds_table==NULL) init_mv2_reduce_scatter_tables_stampede(); is_commutative=smpi_op_is_commute(op); for (i = 0; i < comm_size; i++) { disps[i] = total_count; total_count += recvcnts[i]; } type_size=smpi_datatype_size(datatype); nbytes = total_count * type_size; if (is_commutative) { /* Search for the corresponding system size inside the tuning table */ while ((range < (mv2_size_red_scat_tuning_table - 1)) && (comm_size > mv2_red_scat_thresholds_table[range].numproc)) { range++; } /* Search for corresponding inter-leader function */ while ((range_threshold < (mv2_red_scat_thresholds_table[range].size_inter_table - 1)) && (nbytes > mv2_red_scat_thresholds_table[range].inter_leader[range_threshold].max) && (mv2_red_scat_thresholds_table[range].inter_leader[range_threshold].max != -1)) { range_threshold++; } /* Set inter-leader pt */ MV2_Red_scat_function = mv2_red_scat_thresholds_table[range].inter_leader[range_threshold]. MV2_pt_Red_scat_function; mpi_errno = MV2_Red_scat_function(sendbuf, recvbuf, recvcnts, datatype, op, comm); } else { int is_block_regular = 1; for (i = 0; i < (comm_size - 1); ++i) { if (recvcnts[i] != recvcnts[i+1]) { is_block_regular = 0; break; } } int pof2 = 1; while (pof2 < comm_size) pof2 <<= 1; if (pof2 == comm_size && is_block_regular) { /* noncommutative, pof2 size, and block regular */ mpi_errno = MPIR_Reduce_scatter_non_comm_MV2(sendbuf, recvbuf, recvcnts, datatype, op, comm); } mpi_errno = smpi_coll_tuned_reduce_scatter_mpich_rdb(sendbuf, recvbuf, recvcnts, datatype, op, comm); } xbt_free(disps); return mpi_errno; } int smpi_coll_tuned_scatter_mvapich2(void *sendbuf, int sendcnt, MPI_Datatype sendtype, void *recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm) { int range = 0, range_threshold = 0, range_threshold_intra = 0; int mpi_errno = MPI_SUCCESS; // int mpi_errno_ret = MPI_SUCCESS; int rank, nbytes, comm_size; int recvtype_size, sendtype_size; int partial_sub_ok = 0; int conf_index = 0; int local_size = -1; int i; MPI_Comm shmem_comm; // MPID_Comm *shmem_commptr=NULL; if(mv2_scatter_thresholds_table==NULL) init_mv2_scatter_tables_stampede(); if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } comm_size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); if (rank == root) { sendtype_size=smpi_datatype_size(sendtype); nbytes = sendcnt * sendtype_size; } else { recvtype_size=smpi_datatype_size(recvtype); nbytes = recvcnt * recvtype_size; } // check if safe to use partial subscription mode if (smpi_comm_is_uniform(comm)) { shmem_comm = smpi_comm_get_intra_comm(comm); local_size = smpi_comm_size(shmem_comm); i = 0; if (mv2_scatter_table_ppn_conf[0] == -1) { // Indicating user defined tuning conf_index = 0; }else{ do { if (local_size == mv2_scatter_table_ppn_conf[i]) { conf_index = i; partial_sub_ok = 1; break; } i++; } while(i < mv2_scatter_num_ppn_conf); } } if (partial_sub_ok != 1) { conf_index = 0; } /* Search for the corresponding system size inside the tuning table */ while ((range < (mv2_size_scatter_tuning_table[conf_index] - 1)) && (comm_size > mv2_scatter_thresholds_table[conf_index][range].numproc)) { range++; } /* Search for corresponding inter-leader function */ while ((range_threshold < (mv2_scatter_thresholds_table[conf_index][range].size_inter_table - 1)) && (nbytes > mv2_scatter_thresholds_table[conf_index][range].inter_leader[range_threshold].max) && (mv2_scatter_thresholds_table[conf_index][range].inter_leader[range_threshold].max != -1)) { range_threshold++; } /* Search for corresponding intra-node function */ while ((range_threshold_intra < (mv2_scatter_thresholds_table[conf_index][range].size_intra_table - 1)) && (nbytes > mv2_scatter_thresholds_table[conf_index][range].intra_node[range_threshold_intra].max) && (mv2_scatter_thresholds_table[conf_index][range].intra_node[range_threshold_intra].max != -1)) { range_threshold_intra++; } MV2_Scatter_function = mv2_scatter_thresholds_table[conf_index][range].inter_leader[range_threshold] .MV2_pt_Scatter_function; if(MV2_Scatter_function == &MPIR_Scatter_mcst_wrap_MV2) { #if defined(_MCST_SUPPORT_) if(comm->ch.is_mcast_ok == 1 && mv2_use_mcast_scatter == 1 && comm->ch.shmem_coll_ok == 1) { MV2_Scatter_function = &MPIR_Scatter_mcst_MV2; } else #endif /*#if defined(_MCST_SUPPORT_) */ { if(mv2_scatter_thresholds_table[conf_index][range].inter_leader[range_threshold + 1]. MV2_pt_Scatter_function != NULL) { MV2_Scatter_function = mv2_scatter_thresholds_table[conf_index][range].inter_leader[range_threshold + 1] .MV2_pt_Scatter_function; } else { /* Fallback! */ MV2_Scatter_function = &MPIR_Scatter_MV2_Binomial; } } } if( (MV2_Scatter_function == &MPIR_Scatter_MV2_two_level_Direct) || (MV2_Scatter_function == &MPIR_Scatter_MV2_two_level_Binomial)) { if( smpi_comm_is_blocked(comm)) { MV2_Scatter_intra_function = mv2_scatter_thresholds_table[conf_index][range].intra_node[range_threshold_intra] .MV2_pt_Scatter_function; mpi_errno = MV2_Scatter_function(sendbuf, sendcnt, sendtype, recvbuf, recvcnt, recvtype, root, comm); } else { mpi_errno = MPIR_Scatter_MV2_Binomial(sendbuf, sendcnt, sendtype, recvbuf, recvcnt, recvtype, root, comm); } } else { mpi_errno = MV2_Scatter_function(sendbuf, sendcnt, sendtype, recvbuf, recvcnt, recvtype, root, comm); } return (mpi_errno); } void smpi_coll_cleanup_mvapich2(void){ int i=0; if(mv2_alltoall_thresholds_table) xbt_free(mv2_alltoall_thresholds_table[i]); xbt_free(mv2_alltoall_thresholds_table); xbt_free(mv2_size_alltoall_tuning_table); xbt_free(mv2_alltoall_table_ppn_conf); xbt_free(mv2_gather_thresholds_table); if(mv2_allgather_thresholds_table) xbt_free(mv2_allgather_thresholds_table[0]); xbt_free(mv2_size_allgather_tuning_table); xbt_free(mv2_allgather_table_ppn_conf); xbt_free(mv2_allgather_thresholds_table); xbt_free(mv2_allgatherv_thresholds_table); xbt_free(mv2_reduce_thresholds_table); xbt_free(mv2_red_scat_thresholds_table); xbt_free(mv2_allreduce_thresholds_table); xbt_free(mv2_bcast_thresholds_table); if(mv2_scatter_thresholds_table) xbt_free(mv2_scatter_thresholds_table[0]); xbt_free(mv2_scatter_thresholds_table); xbt_free(mv2_size_scatter_tuning_table); xbt_free(mv2_scatter_table_ppn_conf); } SimGrid-3.14.159/src/smpi/colls/allreduce-mvapich-two-level.c000644 001750 001750 00000014010 13031031271 025403 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2009 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * * Additional copyrights may follow */ /* -*- Mode: C; c-basic-offset:4 ; -*- */ /* Copyright (c) 2001-2014, The Ohio State University. All rights * reserved. * * This file is part of the MVAPICH2 software package developed by the * team members of The Ohio State University's Network-Based Computing * Laboratory (NBCL), headed by Professor Dhabaleswar K. (DK) Panda. * * For detailed copyright and licensing information, please refer to the * copyright file COPYRIGHT in the top level MVAPICH2 directory. */ /* * * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ #include "colls_private.h" #define MPIR_Allreduce_pt2pt_rd_MV2 smpi_coll_tuned_allreduce_rdb #define MPIR_Allreduce_pt2pt_rs_MV2 smpi_coll_tuned_allreduce_mvapich2_rs extern int (*MV2_Allreduce_function)(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm); extern int (*MV2_Allreduce_intra_function)( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm); static int MPIR_Allreduce_reduce_p2p_MV2( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { mpi_coll_reduce_fun(sendbuf,recvbuf,count,datatype,op,0,comm); return MPI_SUCCESS; } static int MPIR_Allreduce_reduce_shmem_MV2( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { mpi_coll_reduce_fun(sendbuf,recvbuf,count,datatype,op,0,comm); return MPI_SUCCESS; } /* general two level allreduce helper function */ int smpi_coll_tuned_allreduce_mvapich2_two_level(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int mpi_errno = MPI_SUCCESS; int total_size = 0; MPI_Aint true_lb, true_extent; MPI_Comm shmem_comm = MPI_COMM_NULL, leader_comm = MPI_COMM_NULL; int local_rank = -1, local_size = 0; //if not set (use of the algo directly, without mvapich2 selector) if(MV2_Allreduce_intra_function==NULL) MV2_Allreduce_intra_function = smpi_coll_tuned_allreduce_mpich; if(MV2_Allreduce_function==NULL) MV2_Allreduce_function = smpi_coll_tuned_allreduce_rdb; if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } if (count == 0) { return MPI_SUCCESS; } smpi_datatype_extent(datatype, &true_lb, &true_extent); total_size = smpi_comm_size(comm); shmem_comm = smpi_comm_get_intra_comm(comm); local_rank = smpi_comm_rank(shmem_comm); local_size = smpi_comm_size(shmem_comm); leader_comm = smpi_comm_get_leaders_comm(comm); if (local_rank == 0) { if (sendbuf != MPI_IN_PLACE) { smpi_datatype_copy(sendbuf, count, datatype, recvbuf, count, datatype); } } /* Doing the shared memory gather and reduction by the leader */ if (local_rank == 0) { if ((MV2_Allreduce_intra_function == &MPIR_Allreduce_reduce_shmem_MV2) || (MV2_Allreduce_intra_function == &MPIR_Allreduce_reduce_p2p_MV2) ) { mpi_errno = MV2_Allreduce_intra_function(sendbuf, recvbuf, count, datatype, op, comm); } else { mpi_errno = MV2_Allreduce_intra_function(sendbuf, recvbuf, count, datatype, op, shmem_comm); } if (local_size != total_size) { void* sendtmpbuf = (char *)smpi_get_tmp_sendbuffer(count*smpi_datatype_get_extent(datatype)); smpi_datatype_copy(recvbuf, count, datatype,sendtmpbuf, count, datatype); /* inter-node allreduce */ if(MV2_Allreduce_function == &MPIR_Allreduce_pt2pt_rd_MV2){ mpi_errno = MPIR_Allreduce_pt2pt_rd_MV2(sendtmpbuf, recvbuf, count, datatype, op, leader_comm); } else { mpi_errno = MPIR_Allreduce_pt2pt_rs_MV2(sendtmpbuf, recvbuf, count, datatype, op, leader_comm); } smpi_free_tmp_buffer(sendtmpbuf); } } else { /* insert the first reduce here */ if ((MV2_Allreduce_intra_function == &MPIR_Allreduce_reduce_shmem_MV2) || (MV2_Allreduce_intra_function == &MPIR_Allreduce_reduce_p2p_MV2) ) { mpi_errno = MV2_Allreduce_intra_function(sendbuf, recvbuf, count, datatype, op, comm); } else { mpi_errno = MV2_Allreduce_intra_function(sendbuf, recvbuf, count, datatype, op, shmem_comm); } } /* Broadcasting the mesage from leader to the rest */ /* Note: shared memory broadcast could improve the performance */ mpi_errno = mpi_coll_bcast_fun(recvbuf, count, datatype, 0, shmem_comm); return (mpi_errno); } SimGrid-3.14.159/src/smpi/colls/alltoallv-pair-one-barrier.c000644 001750 001750 00000004235 13031031271 025240 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** * Function: alltoall_pair * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works when P is power of two. In each phase of P - 1 phases, nodes in pair communicate their data. * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_alltoallv_pair_one_barrier(void *send_buff, int *send_counts, int *send_disps, MPI_Datatype send_type, void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint send_chunk, recv_chunk; MPI_Status s; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALLV; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "alltoallv pair algorithm can't be used with non power of two number of processes ! "); send_chunk = smpi_datatype_get_extent(send_type); recv_chunk = smpi_datatype_get_extent(recv_type); smpi_mpi_barrier(comm); for (i = 0; i < num_procs; i++) { src = dst = rank ^ i; smpi_mpi_sendrecv(send_ptr + send_disps[dst] * send_chunk, send_counts[dst], send_type, dst, tag, recv_ptr + recv_disps[src] * recv_chunk, recv_counts[src], recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allreduce-mvapich-rs.c000644 001750 001750 00000026457 13031031271 024133 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ /* Copyright (c) 2001-2014, The Ohio State University. All rights * reserved. * * This file is part of the MVAPICH2 software package developed by the * team members of The Ohio State University's Network-Based Computing * Laboratory (NBCL), headed by Professor Dhabaleswar K. (DK) Panda. * * For detailed copyright and licensing information, please refer to the * copyright file COPYRIGHT in the top level MVAPICH2 directory. * */ #include "colls_private.h" int smpi_coll_tuned_allreduce_mvapich2_rs(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int mpi_errno = MPI_SUCCESS; int newrank = 0; unsigned int mask, pof2; int dst, is_commutative, rem, newdst, i, send_idx, recv_idx, last_idx, send_cnt, recv_cnt, *cnts, *disps; MPI_Aint true_lb, true_extent, extent; void *tmp_buf, *tmp_buf_free; if (count == 0) { return MPI_SUCCESS; } /* homogeneous */ unsigned int comm_size = smpi_comm_size(comm); unsigned int rank = smpi_comm_rank(comm); is_commutative = smpi_op_is_commute(op); /* need to allocate temporary buffer to store incoming data */ smpi_datatype_extent(datatype, &true_lb, &true_extent); extent = smpi_datatype_get_extent(datatype); tmp_buf_free= smpi_get_tmp_recvbuffer(count * (MAX(extent, true_extent))); /* adjust for potential negative lower bound in datatype */ tmp_buf = (void *) ((char *) tmp_buf_free - true_lb); /* copy local data into recvbuf */ if (sendbuf != MPI_IN_PLACE) { mpi_errno = smpi_datatype_copy(sendbuf, count, datatype, recvbuf, count, datatype); } /* find nearest power-of-two less than or equal to comm_size */ for( pof2 = 1; pof2 <= comm_size; pof2 <<= 1 ); pof2 >>=1; rem = comm_size - pof2; /* In the non-power-of-two case, all even-numbered processes of rank < 2*rem send their data to (rank+1). These even-numbered processes no longer participate in the algorithm until the very end. The remaining processes form a nice power-of-two. */ if (rank < 2 * rem) { if (rank % 2 == 0) { /* even */ smpi_mpi_send(recvbuf, count, datatype, rank + 1, COLL_TAG_ALLREDUCE, comm); /* temporarily set the rank to -1 so that this process does not pariticipate in recursive doubling */ newrank = -1; } else { /* odd */ smpi_mpi_recv(tmp_buf, count, datatype, rank - 1, COLL_TAG_ALLREDUCE, comm, MPI_STATUS_IGNORE); /* do the reduction on received data. since the ordering is right, it doesn't matter whether the operation is commutative or not. */ smpi_op_apply(op, tmp_buf, recvbuf, &count, &datatype); /* change the rank */ newrank = rank / 2; } } else { /* rank >= 2*rem */ newrank = rank - rem; } /* If op is user-defined or count is less than pof2, use recursive doubling algorithm. Otherwise do a reduce-scatter followed by allgather. (If op is user-defined, derived datatypes are allowed and the user could pass basic datatypes on one process and derived on another as long as the type maps are the same. Breaking up derived datatypes to do the reduce-scatter is tricky, therefore using recursive doubling in that case.) */ if (newrank != -1) { if (/*(HANDLE_GET_KIND(op) != HANDLE_KIND_BUILTIN) ||*/ (count < pof2)) { /* use recursive doubling */ mask = 0x1; while (mask < pof2) { newdst = newrank ^ mask; /* find real rank of dest */ dst = (newdst < rem) ? newdst * 2 + 1 : newdst + rem; /* Send the most current data, which is in recvbuf. Recv into tmp_buf */ smpi_mpi_sendrecv(recvbuf, count, datatype, dst, COLL_TAG_ALLREDUCE, tmp_buf, count, datatype, dst, COLL_TAG_ALLREDUCE, comm, MPI_STATUS_IGNORE); /* tmp_buf contains data received in this step. recvbuf contains data accumulated so far */ if (is_commutative || (dst < rank)) { /* op is commutative OR the order is already right */ smpi_op_apply(op, tmp_buf, recvbuf, &count, &datatype); } else { /* op is noncommutative and the order is not right */ smpi_op_apply(op, recvbuf, tmp_buf, &count, &datatype); /* copy result back into recvbuf */ mpi_errno = smpi_datatype_copy(tmp_buf, count, datatype, recvbuf, count, datatype); } mask <<= 1; } } else { /* do a reduce-scatter followed by allgather */ /* for the reduce-scatter, calculate the count that each process receives and the displacement within the buffer */ cnts = (int *)xbt_malloc(pof2 * sizeof (int)); disps = (int *)xbt_malloc(pof2 * sizeof (int)); for (i = 0; i < (pof2 - 1); i++) { cnts[i] = count / pof2; } cnts[pof2 - 1] = count - (count / pof2) * (pof2 - 1); disps[0] = 0; for (i = 1; i < pof2; i++) { disps[i] = disps[i - 1] + cnts[i - 1]; } mask = 0x1; send_idx = recv_idx = 0; last_idx = pof2; while (mask < pof2) { newdst = newrank ^ mask; /* find real rank of dest */ dst = (newdst < rem) ? newdst * 2 + 1 : newdst + rem; send_cnt = recv_cnt = 0; if (newrank < newdst) { send_idx = recv_idx + pof2 / (mask * 2); for (i = send_idx; i < last_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < send_idx; i++) recv_cnt += cnts[i]; } else { recv_idx = send_idx + pof2 / (mask * 2); for (i = send_idx; i < recv_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < last_idx; i++) recv_cnt += cnts[i]; } /* Send data from recvbuf. Recv into tmp_buf */ smpi_mpi_sendrecv((char *) recvbuf + disps[send_idx] * extent, send_cnt, datatype, dst, COLL_TAG_ALLREDUCE, (char *) tmp_buf + disps[recv_idx] * extent, recv_cnt, datatype, dst, COLL_TAG_ALLREDUCE, comm, MPI_STATUS_IGNORE); /* tmp_buf contains data received in this step. recvbuf contains data accumulated so far */ /* This algorithm is used only for predefined ops and predefined ops are always commutative. */ smpi_op_apply(op, (char *) tmp_buf + disps[recv_idx] * extent, (char *) recvbuf + disps[recv_idx] * extent, &recv_cnt, &datatype); /* update send_idx for next iteration */ send_idx = recv_idx; mask <<= 1; /* update last_idx, but not in last iteration because the value is needed in the allgather step below. */ if (mask < pof2) last_idx = recv_idx + pof2 / mask; } /* now do the allgather */ mask >>= 1; while (mask > 0) { newdst = newrank ^ mask; /* find real rank of dest */ dst = (newdst < rem) ? newdst * 2 + 1 : newdst + rem; send_cnt = recv_cnt = 0; if (newrank < newdst) { /* update last_idx except on first iteration */ if (mask != pof2 / 2) { last_idx = last_idx + pof2 / (mask * 2); } recv_idx = send_idx + pof2 / (mask * 2); for (i = send_idx; i < recv_idx; i++) { send_cnt += cnts[i]; } for (i = recv_idx; i < last_idx; i++) { recv_cnt += cnts[i]; } } else { recv_idx = send_idx - pof2 / (mask * 2); for (i = send_idx; i < last_idx; i++) { send_cnt += cnts[i]; } for (i = recv_idx; i < send_idx; i++) { recv_cnt += cnts[i]; } } smpi_mpi_sendrecv((char *) recvbuf + disps[send_idx] * extent, send_cnt, datatype, dst, COLL_TAG_ALLREDUCE, (char *) recvbuf + disps[recv_idx] * extent, recv_cnt, datatype, dst, COLL_TAG_ALLREDUCE, comm, MPI_STATUS_IGNORE); if (newrank > newdst) { send_idx = recv_idx; } mask >>= 1; } xbt_free(disps); xbt_free(cnts); } } /* In the non-power-of-two case, all odd-numbered processes of rank < 2*rem send the result to (rank-1), the ranks who didn't participate above. */ if (rank < 2 * rem) { if (rank % 2) { /* odd */ smpi_mpi_send(recvbuf, count, datatype, rank - 1, COLL_TAG_ALLREDUCE, comm); } else { /* even */ smpi_mpi_recv(recvbuf, count, datatype, rank + 1, COLL_TAG_ALLREDUCE, comm, MPI_STATUS_IGNORE); } } smpi_free_tmp_buffer(tmp_buf_free); return (mpi_errno); } SimGrid-3.14.159/src/smpi/colls/bcast-arrival-pattern-aware.c000644 001750 001750 00000026260 13031031271 025416 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" static int bcast_NTSL_segment_size_in_byte = 8192; #define HEADER_SIZE 1024 #define MAX_NODE 1024 /* Non-topology-specific pipelined linear-bcast function */ int smpi_coll_tuned_bcast_arrival_pattern_aware(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int tag = -COLL_TAG_BCAST; MPI_Status status; MPI_Request request; MPI_Request *send_request_array; MPI_Request *recv_request_array; MPI_Status *send_status_array; MPI_Status *recv_status_array; MPI_Status temp_status_array[MAX_NODE]; int rank, size; int i, j; int sent_count; int header_index; int flag_array[MAX_NODE]; int already_sent[MAX_NODE]; int to_clean[MAX_NODE]; int header_buf[HEADER_SIZE]; char temp_buf[MAX_NODE]; MPI_Aint extent; extent = smpi_datatype_get_extent(datatype); /* destination */ int to; rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); /* segment is segment size in number of elements (not bytes) */ int segment = bcast_NTSL_segment_size_in_byte / extent; segment = segment == 0 ? 1 :segment; /* pipeline length */ int pipe_length = count / segment; /* use for buffer offset for sending and receiving data = segment size in byte */ int increment = segment * extent; /* if the input size is not divisible by segment size => the small remainder will be done with native implementation */ int remainder = count % segment; /* if root is not zero send to rank zero first this can be modified to make it faster by using logical src, dst. */ if (root != 0) { if (rank == root) { smpi_mpi_send(buf, count, datatype, 0, tag, comm); } else if (rank == 0) { smpi_mpi_recv(buf, count, datatype, root, tag, comm, &status); } } /* value == 0 means root has not send data (or header) to the node yet */ for (i = 0; i < MAX_NODE; i++) { already_sent[i] = 0; to_clean[i] = 0; } /* when a message is smaller than a block size => no pipeline */ if (count <= segment) { if (rank == 0) { sent_count = 0; while (sent_count < (size - 1)) { for (i = 1; i < size; i++) { smpi_mpi_iprobe(i, MPI_ANY_TAG, comm, &flag_array[i], MPI_STATUSES_IGNORE); } header_index = 0; /* recv 1-byte message */ for (i = 1; i < size; i++) { /* message arrive */ if ((flag_array[i] == 1) && (already_sent[i] == 0)) { smpi_mpi_recv(temp_buf, 1, MPI_CHAR, i, tag, comm, &status); header_buf[header_index] = i; header_index++; sent_count++; /* will send in the next step */ already_sent[i] = 1; } } /* send header followed by data */ if (header_index != 0) { header_buf[header_index] = -1; to = header_buf[0]; smpi_mpi_send(header_buf, HEADER_SIZE, MPI_INT, to, tag, comm); smpi_mpi_send(buf, count, datatype, to, tag, comm); } /* randomly MPI_Send to one */ else { /* search for the first node that never received data before */ for (i = 1; i < size; i++) { if (already_sent[i] == 0) { header_buf[0] = i; header_buf[1] = -1; smpi_mpi_send(header_buf, HEADER_SIZE, MPI_INT, i, tag, comm); smpi_mpi_send(buf, count, datatype, i, tag, comm); already_sent[i] = 1; sent_count++; break; } } } } /* while loop */ } /* non-root */ else { /* send 1-byte message to root */ smpi_mpi_send(temp_buf, 1, MPI_CHAR, 0, tag, comm); /* wait for header and data, forward when required */ smpi_mpi_recv(header_buf, HEADER_SIZE, MPI_INT, MPI_ANY_SOURCE, tag, comm, &status); smpi_mpi_recv(buf, count, datatype, MPI_ANY_SOURCE, tag, comm, &status); /* search for where it is */ int myordering = 0; while (rank != header_buf[myordering]) { myordering++; } /* send header followed by data */ if (header_buf[myordering + 1] != -1) { smpi_mpi_send(header_buf, HEADER_SIZE, MPI_INT, header_buf[myordering + 1], tag, comm); smpi_mpi_send(buf, count, datatype, header_buf[myordering + 1], tag, comm); } } } /* pipeline bcast */ else { send_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); recv_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); send_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); recv_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); if (rank == 0) { //double start2 = MPI_Wtime(); sent_count = 0; //int iteration = 0; while (sent_count < (size - 1)) { //iteration++; //start = MPI_Wtime(); for (i = 1; i < size; i++) { smpi_mpi_iprobe(i, MPI_ANY_TAG, comm, &flag_array[i], &temp_status_array[i]); } //total = MPI_Wtime() - start; //total *= 1000; //printf("Iprobe time = %.2f\n",total); header_index = 0; MPI_Wtime(); /* recv 1-byte message */ for (i = 1; i < size; i++) { /* message arrive */ if ((flag_array[i] == 1) && (already_sent[i] == 0)) { smpi_mpi_recv(&temp_buf[i], 1, MPI_CHAR, i, tag, comm, &status); header_buf[header_index] = i; header_index++; sent_count++; /* will send in the next step */ already_sent[i] = 1; } } //total = MPI_Wtime() - start; //total *= 1000; //printf("Recv 1-byte time = %.2f\n",total); /* if (header_index != 0) { printf("header index = %d node = ",header_index); for (i=0;i segment)) { XBT_WARN("MPI_bcast_arrival_pattern_aware use default MPI_bcast."); smpi_mpi_bcast((char *)buf + (pipe_length * increment), remainder, datatype, root, comm); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allgather-rdb.c000644 001750 001750 00000010341 13031031271 022615 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" int smpi_coll_tuned_allgather_rdb(void *sbuf, int send_count, MPI_Datatype send_type, void *rbuf, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { // MPI variables MPI_Status status; MPI_Aint send_chunk, recv_chunk; // local int variables unsigned int i, j, k, dst, send_offset, recv_offset, tree_root; int dst_tree_root, rank_tree_root, last_recv_count = 0, num_procs_completed; int offset, tmp_mask; int tag = COLL_TAG_ALLGATHER; int mask = 1; int success = 0; int curr_count = recv_count; // local string variables char *send_ptr = (char *) sbuf; char *recv_ptr = (char *) rbuf; // get size of the communicator, followed by rank unsigned int num_procs = smpi_comm_size(comm); unsigned int rank = smpi_comm_rank(comm); // get size of single element's type for send buffer and recv buffer send_chunk = smpi_datatype_get_extent(send_type); recv_chunk = smpi_datatype_get_extent(recv_type); // multiply size of each element by number of elements to send or recv send_chunk *= send_count; recv_chunk *= recv_count; // perform a local copy smpi_mpi_sendrecv(send_ptr, send_count, send_type, rank, tag, recv_ptr + rank * recv_chunk, recv_count, recv_type, rank, tag, comm, &status); i = 0; while (mask < num_procs) { dst = rank ^ mask; dst_tree_root = dst >> i; dst_tree_root <<= i; rank_tree_root = rank >> i; rank_tree_root <<= i; send_offset = rank_tree_root * send_chunk; recv_offset = dst_tree_root * recv_chunk; if (dst < num_procs) { smpi_mpi_sendrecv(recv_ptr + send_offset, curr_count, send_type, dst, tag, recv_ptr + recv_offset, mask * recv_count, recv_type, dst, tag, comm, &status); last_recv_count = smpi_mpi_get_count(&status, recv_type); curr_count += last_recv_count; } if (dst_tree_root + mask > num_procs) { num_procs_completed = num_procs - rank_tree_root - mask; /* num_procs_completed is the number of processes in this subtree that have all the data. Send data to others in a tree fashion. First find root of current tree that is being divided into two. k is the number of least-significant bits in this process's rank that must be zeroed out to find the rank of the root */ j = mask; k = 0; while (j) { j >>= 1; k++; } k--; offset = recv_chunk * (rank_tree_root + mask); tmp_mask = mask >> 1; while (tmp_mask) { dst = rank ^ tmp_mask; tree_root = rank >> k; tree_root <<= k; /* send only if this proc has data and destination doesn't have data. at any step, multiple processes can send if they have the data */ if ((dst > rank) && (rank < tree_root + num_procs_completed) && (dst >= tree_root + num_procs_completed)) { smpi_mpi_send(recv_ptr + offset, last_recv_count, recv_type, dst, tag, comm); /* last_recv_cnt was set in the previous receive. that's the amount of data to be sent now. */ } /* recv only if this proc. doesn't have data and sender has data */ else if ((dst < rank) && (dst < tree_root + num_procs_completed) && (rank >= tree_root + num_procs_completed)) { smpi_mpi_recv(recv_ptr + offset, recv_count * num_procs_completed, recv_type, dst, tag, comm, &status); // num_procs_completed is also equal to the no. of processes // whose data we don't have last_recv_count = smpi_mpi_get_count(&status, recv_type); curr_count += last_recv_count; } tmp_mask >>= 1; k--; } } mask <<= 1; i++; } return success; } SimGrid-3.14.159/src/smpi/colls/allgather-SMP-NTS.c000644 001750 001750 00000014015 13031031271 023151 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" int smpi_coll_tuned_allgather_SMP_NTS(void *sbuf, int scount, MPI_Datatype stype, void *rbuf, int rcount, MPI_Datatype rtype, MPI_Comm comm) { int src, dst, comm_size, rank; comm_size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); MPI_Aint rextent, sextent; rextent = smpi_datatype_get_extent(rtype); sextent = smpi_datatype_get_extent(stype); int tag = COLL_TAG_ALLGATHER; int i, send_offset, recv_offset; int intra_rank, inter_rank; if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } int num_core=1; if (smpi_comm_is_uniform(comm)){ num_core = smpi_comm_size(smpi_comm_get_intra_comm(comm)); } intra_rank = rank % num_core; inter_rank = rank / num_core; int inter_comm_size = (comm_size + num_core - 1) / num_core; int num_core_in_current_smp = num_core; if(comm_size%num_core) THROWF(arg_error,0, "allgather SMP NTS algorithm can't be used with non multiple of NUM_CORE=%d number of processes ! ", num_core); /* for too small number of processes, use default implementation */ if (comm_size <= num_core) { XBT_WARN("MPI_allgather_SMP_NTS use default MPI_allgather."); smpi_mpi_allgather(sbuf, scount, stype, rbuf, rcount, rtype, comm); return MPI_SUCCESS; } // the last SMP node may have fewer number of running processes than all others if (inter_rank == (inter_comm_size - 1)) { num_core_in_current_smp = comm_size - (inter_rank * num_core); } //copy corresponding message from sbuf to rbuf recv_offset = rank * rextent * rcount; smpi_mpi_sendrecv(sbuf, scount, stype, rank, tag, ((char *) rbuf + recv_offset), rcount, rtype, rank, tag, comm, MPI_STATUS_IGNORE); //gather to root of each SMP for (i = 1; i < num_core_in_current_smp; i++) { dst = (inter_rank * num_core) + (intra_rank + i) % (num_core_in_current_smp); src = (inter_rank * num_core) + (intra_rank - i + num_core_in_current_smp) % (num_core_in_current_smp); recv_offset = src * rextent * rcount; smpi_mpi_sendrecv(sbuf, scount, stype, dst, tag, ((char *) rbuf + recv_offset), rcount, rtype, src, tag, comm, MPI_STATUS_IGNORE); } // INTER-SMP-ALLGATHER // Every root of each SMP node post INTER-Sendrecv, then do INTRA-Bcast for each receiving message // Use logical ring algorithm // root of each SMP if (intra_rank == 0) { MPI_Request *rrequest_array = xbt_new(MPI_Request, inter_comm_size - 1); MPI_Request *srequest_array = xbt_new(MPI_Request, inter_comm_size - 1); src = ((inter_rank - 1 + inter_comm_size) % inter_comm_size) * num_core; dst = ((inter_rank + 1) % inter_comm_size) * num_core; // post all inter Irecv for (i = 0; i < inter_comm_size - 1; i++) { recv_offset = ((inter_rank - i - 1 + inter_comm_size) % inter_comm_size) * num_core * sextent * scount; rrequest_array[i] = smpi_mpi_irecv((char *)rbuf + recv_offset, rcount * num_core, rtype, src, tag + i, comm); } // send first message send_offset = ((inter_rank + inter_comm_size) % inter_comm_size) * num_core * sextent * scount; srequest_array[0] = smpi_mpi_isend((char *)rbuf + send_offset, scount * num_core, stype, dst, tag, comm); // loop : recv-inter , send-inter, send-intra (linear-bcast) for (i = 0; i < inter_comm_size - 2; i++) { recv_offset = ((inter_rank - i - 1 + inter_comm_size) % inter_comm_size) * num_core * sextent * scount; smpi_mpi_wait(&rrequest_array[i], MPI_STATUS_IGNORE); srequest_array[i + 1] = smpi_mpi_isend((char *)rbuf + recv_offset, scount * num_core, stype, dst, tag + i + 1, comm); if (num_core_in_current_smp > 1) { smpi_mpi_send((char *)rbuf + recv_offset, scount * num_core, stype, (rank + 1), tag + i + 1, comm); } } // recv last message and send_intra recv_offset = ((inter_rank - i - 1 + inter_comm_size) % inter_comm_size) * num_core * sextent * scount; //recv_offset = ((inter_rank + 1) % inter_comm_size) * num_core * sextent * scount; //i=inter_comm_size-2; smpi_mpi_wait(&rrequest_array[i], MPI_STATUS_IGNORE); if (num_core_in_current_smp > 1) { smpi_mpi_send((char *)rbuf + recv_offset, scount * num_core, stype, (rank + 1), tag + i + 1, comm); } smpi_mpi_waitall(inter_comm_size - 1, srequest_array, MPI_STATUSES_IGNORE); xbt_free(rrequest_array); xbt_free(srequest_array); } // last rank of each SMP else if (intra_rank == (num_core_in_current_smp - 1)) { for (i = 0; i < inter_comm_size - 1; i++) { recv_offset = ((inter_rank - i - 1 + inter_comm_size) % inter_comm_size) * num_core * sextent * scount; smpi_mpi_recv((char *) rbuf + recv_offset, (rcount * num_core), rtype, rank - 1, tag + i + 1, comm, MPI_STATUS_IGNORE); } } // intermediate rank of each SMP else { for (i = 0; i < inter_comm_size - 1; i++) { recv_offset = ((inter_rank - i - 1 + inter_comm_size) % inter_comm_size) * num_core * sextent * scount; smpi_mpi_recv((char *) rbuf + recv_offset, (rcount * num_core), rtype, rank - 1, tag + i + 1, comm, MPI_STATUS_IGNORE); smpi_mpi_send((char *) rbuf + recv_offset, (scount * num_core), stype, (rank + 1), tag + i + 1, comm); } } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allgatherv-mpich-rdb.c000644 001750 001750 00000016131 13031031271 024104 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* Short or medium size message and power-of-two no. of processes. Use * recursive doubling algorithm */ #include "colls_private.h" int smpi_coll_tuned_allgatherv_mpich_rdb ( void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvtype, MPI_Comm comm) { int j, i; MPI_Status status; MPI_Aint recvtype_extent, recvtype_true_extent, recvtype_true_lb; int curr_cnt, dst, total_count; void *tmp_buf, *tmp_buf_rl; unsigned int mask, dst_tree_root, my_tree_root, position, send_offset, recv_offset, last_recv_cnt=0, nprocs_completed, k, offset, tmp_mask, tree_root; unsigned int comm_size = smpi_comm_size(comm); unsigned int rank = smpi_comm_rank(comm); total_count = 0; for (i=0; i> i; dst_tree_root <<= i; my_tree_root = rank >> i; my_tree_root <<= i; if (dst < comm_size) { send_offset = 0; for (j=0; j comm_size) { nprocs_completed = comm_size - my_tree_root - mask; /* nprocs_completed is the number of processes in this subtree that have all the data. Send data to others in a tree fashion. First find root of current tree that is being divided into two. k is the number of least-significant bits in this process's rank that must be zeroed out to find the rank of the root */ j = mask; k = 0; while (j) { j >>= 1; k++; } k--; tmp_mask = mask >> 1; while (tmp_mask) { dst = rank ^ tmp_mask; tree_root = rank >> k; tree_root <<= k; /* send only if this proc has data and destination doesn't have data. at any step, multiple processes can send if they have the data */ if ((dst > rank) && (rank < tree_root + nprocs_completed) && (dst >= tree_root + nprocs_completed)) { offset = 0; for (j=0; j<(my_tree_root+mask); j++) offset += recvcounts[j]; offset *= recvtype_extent; smpi_mpi_send(((char *)tmp_buf + offset), last_recv_cnt, recvtype, dst, COLL_TAG_ALLGATHERV, comm); /* last_recv_cnt was set in the previous receive. that's the amount of data to be sent now. */ } /* recv only if this proc. doesn't have data and sender has data */ else if ((dst < rank) && (dst < tree_root + nprocs_completed) && (rank >= tree_root + nprocs_completed)) { offset = 0; for (j=0; j<(my_tree_root+mask); j++) offset += recvcounts[j]; smpi_mpi_recv(((char *)tmp_buf + offset * recvtype_extent), total_count - offset, recvtype, dst, COLL_TAG_ALLGATHERV, comm, &status); /* for convenience, recv is posted for a bigger amount than will be sent */ last_recv_cnt=smpi_mpi_get_count(&status, recvtype); curr_cnt += last_recv_cnt; } tmp_mask >>= 1; k--; } } /* --END EXPERIMENTAL-- */ mask <<= 1; i++; } /* copy data from tmp_buf to recvbuf */ position = 0; for (j=0; jch.use_intra_sock_comm == 1) && (use_limic_gather) &&((num_scheme == USE_GATHER_PT_PT_BINOMIAL) || (num_scheme == USE_GATHER_PT_PT_DIRECT) ||(num_scheme == USE_GATHER_PT_LINEAR_BINOMIAL) || (num_scheme == USE_GATHER_PT_LINEAR_DIRECT) || (num_scheme == USE_GATHER_LINEAR_PT_BINOMIAL) || (num_scheme == USE_GATHER_LINEAR_PT_DIRECT) || (num_scheme == USE_GATHER_LINEAR_LINEAR) || (num_scheme == USE_GATHER_SINGLE_LEADER))) { mpi_errno = MV2_Gather_intra_node_function(sendbuf, sendcnt, sendtype, recvbuf, recvcnt,recvtype, root, comm); } else #endif/*#if defined(_SMP_LIMIC_)*/ { if (local_rank == 0) { /* Node leader, allocate tmp_buffer */ if (rank == root) { tmp_buf = smpi_get_tmp_recvbuffer(recvcnt * MAX(recvtype_extent, recvtype_true_extent) * local_size); } else { tmp_buf = smpi_get_tmp_sendbuffer(sendcnt * MAX(sendtype_extent, sendtype_true_extent) * local_size); } if (tmp_buf == NULL) { mpi_errno = MPI_ERR_OTHER; return mpi_errno; } } /*while testing mpich2 gather test, we see that * which basically splits the comm, and we come to * a point, where use_intra_sock_comm == 0, but if the * intra node function is MPIR_Intra_node_LIMIC_Gather_MV2, * it would use the intra sock comm. In such cases, we * fallback to binomial as a default case.*/ #if defined(_SMP_LIMIC_) if(*MV2_Gather_intra_node_function == MPIR_Intra_node_LIMIC_Gather_MV2) { mpi_errno = MPIR_pt_pt_intra_gather(sendbuf,sendcnt, sendtype, recvbuf, recvcnt, recvtype, root, rank, tmp_buf, nbytes, TEMP_BUF_HAS_NO_DATA, shmem_commptr, MPIR_Gather_intra); } else #endif { /*We are gathering the data into tmp_buf and the output * will be of MPI_BYTE datatype. Since the tmp_buf has no * local data, we pass is_data_avail = TEMP_BUF_HAS_NO_DATA*/ mpi_errno = MPIR_pt_pt_intra_gather(sendbuf,sendcnt, sendtype, recvbuf, recvcnt, recvtype, root, rank, tmp_buf, nbytes, TEMP_BUF_HAS_NO_DATA, shmem_comm, MV2_Gather_intra_node_function ); } } leader_comm = smpi_comm_get_leaders_comm(comm); int* leaders_map = smpi_comm_get_leaders_map(comm); leader_of_root = smpi_group_rank(smpi_comm_group(comm),leaders_map[root]); leader_root = smpi_group_rank(smpi_comm_group(leader_comm),leaders_map[root]); /* leader_root is the rank of the leader of the root in leader_comm. * leader_root is to be used as the root of the inter-leader gather ops */ if (!smpi_comm_is_uniform(comm)) { if (local_rank == 0) { int *displs = NULL; int *recvcnts = NULL; int *node_sizes; int i = 0; /* Node leaders have all the data. But, different nodes can have * different number of processes. Do a Gather first to get the * buffer lengths at each leader, followed by a Gatherv to move * the actual data */ if (leader_comm_rank == leader_root && root != leader_of_root) { /* The root of the Gather operation is not a node-level * leader and this process's rank in the leader_comm * is the same as leader_root */ if(rank == root) { leader_gather_buf = smpi_get_tmp_recvbuffer(recvcnt * MAX(recvtype_extent, recvtype_true_extent) * comm_size); } else { leader_gather_buf = smpi_get_tmp_sendbuffer(sendcnt * MAX(sendtype_extent, sendtype_true_extent) * comm_size); } if (leader_gather_buf == NULL) { mpi_errno = MPI_ERR_OTHER; return mpi_errno; } } node_sizes = smpi_comm_get_non_uniform_map(comm); if (leader_comm_rank == leader_root) { displs = xbt_malloc(sizeof (int) * leader_comm_size); recvcnts = xbt_malloc(sizeof (int) * leader_comm_size); if (!displs || !recvcnts) { mpi_errno = MPI_ERR_OTHER; return mpi_errno; } } if (root == leader_of_root) { /* The root of the gather operation is also the node * leader. Receive into recvbuf and we are done */ if (leader_comm_rank == leader_root) { recvcnts[0] = node_sizes[0] * recvcnt; displs[0] = 0; for (i = 1; i < leader_comm_size; i++) { displs[i] = displs[i - 1] + node_sizes[i - 1] * recvcnt; recvcnts[i] = node_sizes[i] * recvcnt; } } smpi_mpi_gatherv(tmp_buf, local_size * nbytes, MPI_BYTE, recvbuf, recvcnts, displs, recvtype, leader_root, leader_comm); } else { /* The root of the gather operation is not the node leader. * Receive into leader_gather_buf and then send * to the root */ if (leader_comm_rank == leader_root) { recvcnts[0] = node_sizes[0] * nbytes; displs[0] = 0; for (i = 1; i < leader_comm_size; i++) { displs[i] = displs[i - 1] + node_sizes[i - 1] * nbytes; recvcnts[i] = node_sizes[i] * nbytes; } } smpi_mpi_gatherv(tmp_buf, local_size * nbytes, MPI_BYTE, leader_gather_buf, recvcnts, displs, MPI_BYTE, leader_root, leader_comm); } if (leader_comm_rank == leader_root) { xbt_free(displs); xbt_free(recvcnts); } } } else { /* All nodes have the same number of processes. * Just do one Gather to get all * the data at the leader of the root process */ if (local_rank == 0) { if (leader_comm_rank == leader_root && root != leader_of_root) { /* The root of the Gather operation is not a node-level leader */ leader_gather_buf = smpi_get_tmp_sendbuffer(nbytes * comm_size); if (leader_gather_buf == NULL) { mpi_errno = MPI_ERR_OTHER; return mpi_errno; } } if (root == leader_of_root) { mpi_errno = MPIR_Gather_MV2_Direct(tmp_buf, nbytes * local_size, MPI_BYTE, recvbuf, recvcnt * local_size, recvtype, leader_root, leader_comm); } else { mpi_errno = MPIR_Gather_MV2_Direct(tmp_buf, nbytes * local_size, MPI_BYTE, leader_gather_buf, nbytes * local_size, MPI_BYTE, leader_root, leader_comm); } } } if ((local_rank == 0) && (root != rank) && (leader_of_root == rank)) { smpi_mpi_send(leader_gather_buf, nbytes * comm_size, MPI_BYTE, root, COLL_TAG_GATHER, comm); } if (rank == root && local_rank != 0) { /* The root of the gather operation is not the node leader. Receive y* data from the node leader */ smpi_mpi_recv(recvbuf, recvcnt * comm_size, recvtype, leader_of_root, COLL_TAG_GATHER, comm, &status); } /* check if multiple threads are calling this collective function */ if (local_rank == 0 ) { if (tmp_buf != NULL) { smpi_free_tmp_buffer(tmp_buf); } if (leader_gather_buf != NULL) { smpi_free_tmp_buffer(leader_gather_buf); } } return (mpi_errno); } SimGrid-3.14.159/src/smpi/colls/allgather-loosely-lr.c000644 001750 001750 00000010514 13031031271 024151 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" int smpi_coll_tuned_allgather_loosely_lr(void *sbuf, int scount, MPI_Datatype stype, void *rbuf, int rcount, MPI_Datatype rtype, MPI_Comm comm) { int comm_size, rank; int tag = COLL_TAG_ALLGATHER; int i, j, send_offset, recv_offset; int intra_rank, inter_rank, inter_comm_size, intra_comm_size; int inter_dst, inter_src; comm_size = smpi_comm_size(comm); if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } int num_core=1; if (smpi_comm_is_uniform(comm)){ num_core = smpi_comm_size(smpi_comm_get_intra_comm(comm)); } if(comm_size%num_core) THROWF(arg_error,0, "allgather loosely lr algorithm can't be used with non multiple of NUM_CORE=%d number of processes ! ",num_core); rank = smpi_comm_rank(comm); MPI_Aint rextent, sextent; rextent = smpi_datatype_get_extent(rtype); sextent = smpi_datatype_get_extent(stype); MPI_Request inter_rrequest; MPI_Request rrequest_array[128]; MPI_Request srequest_array[128]; MPI_Request inter_srequest_array[128]; int rrequest_count = 0; int srequest_count = 0; int inter_srequest_count = 0; MPI_Status status; intra_rank = rank % num_core; inter_rank = rank / num_core; inter_comm_size = (comm_size + num_core - 1) / num_core; intra_comm_size = num_core; int src_seg, dst_seg; //copy corresponding message from sbuf to rbuf recv_offset = rank * rextent * rcount; smpi_mpi_sendrecv(sbuf, scount, stype, rank, tag, (char *)rbuf + recv_offset, rcount, rtype, rank, tag, comm, &status); int dst, src; int inter_send_offset, inter_recv_offset; rrequest_count = 0; srequest_count = 0; inter_srequest_count = 0; for (i = 0; i < inter_comm_size; i++) { // inter_communication inter_dst = (rank + intra_comm_size) % comm_size; inter_src = (rank - intra_comm_size + comm_size) % comm_size; src_seg = ((inter_rank - 1 - i + inter_comm_size) % inter_comm_size) * intra_comm_size + intra_rank; dst_seg = ((inter_rank - i + inter_comm_size) % inter_comm_size) * intra_comm_size + intra_rank; inter_send_offset = dst_seg * sextent * scount; inter_recv_offset = src_seg * rextent * rcount; for (j = 0; j < intra_comm_size; j++) { // inter communication if (intra_rank == j) { if (i != inter_comm_size - 1) { inter_rrequest = smpi_mpi_irecv((char *)rbuf + inter_recv_offset, rcount, rtype, inter_src, tag, comm); inter_srequest_array[inter_srequest_count++] = smpi_mpi_isend((char *)rbuf + inter_send_offset, scount, stype, inter_dst, tag, comm); } } //intra_communication src = inter_rank * intra_comm_size + j; dst = inter_rank * intra_comm_size + j; src_seg = ((inter_rank - i + inter_comm_size) % inter_comm_size) * intra_comm_size + j; dst_seg = ((inter_rank - i + inter_comm_size) % inter_comm_size) * intra_comm_size + intra_rank; send_offset = dst_seg * sextent * scount; recv_offset = src_seg * rextent * rcount; if (j != intra_rank) { rrequest_array[rrequest_count++] = smpi_mpi_irecv((char *)rbuf + recv_offset, rcount, rtype, src, tag, comm); srequest_array[srequest_count++] = smpi_mpi_isend((char *)rbuf + send_offset, scount, stype, dst, tag, comm); } } // intra loop // wait for inter communication to finish for these rounds (# of round equals num_core) if (i != inter_comm_size - 1) { smpi_mpi_wait(&inter_rrequest, &status); } } //inter loop smpi_mpi_waitall(rrequest_count, rrequest_array, MPI_STATUSES_IGNORE); smpi_mpi_waitall(srequest_count, srequest_array, MPI_STATUSES_IGNORE); smpi_mpi_waitall(inter_srequest_count, inter_srequest_array, MPI_STATUSES_IGNORE); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/reduce_scatter-mpich.c000644 001750 001750 00000050605 13031031272 024207 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" static inline int MPIU_Mirror_permutation(unsigned int x, int bits) { /* a mask for the high order bits that should be copied as-is */ int high_mask = ~((0x1 << bits) - 1); int retval = x & high_mask; int i; for (i = 0; i < bits; ++i) { unsigned int bitval = (x & (0x1 << i)) >> i; /* 0x1 or 0x0 */ retval |= bitval << ((bits - i) - 1); } return retval; } int smpi_coll_tuned_reduce_scatter_mpich_pair(void *sendbuf, void *recvbuf, int recvcounts[], MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int rank, comm_size, i; MPI_Aint extent, true_extent, true_lb; int *disps; void *tmp_recvbuf; int mpi_errno = MPI_SUCCESS; int total_count, dst, src; int is_commutative; comm_size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); extent =smpi_datatype_get_extent(datatype); smpi_datatype_extent(datatype, &true_lb, &true_extent); if (smpi_op_is_commute(op)) { is_commutative = 1; } disps = (int*)xbt_malloc( comm_size * sizeof(int)); total_count = 0; for (i=0; i peer) { /* we have the higher rank: send top half, recv bottom half */ recv_offset += size; } else { /* we have the lower rank: recv top half, send bottom half */ send_offset += size; } smpi_mpi_sendrecv(outgoing_data + send_offset*true_extent, size, datatype, peer, COLL_TAG_SCATTER, incoming_data + recv_offset*true_extent, size, datatype, peer, COLL_TAG_SCATTER, comm, MPI_STATUS_IGNORE); /* always perform the reduction at recv_offset, the data at send_offset is now our peer's responsibility */ if (rank > peer) { /* higher ranked value so need to call op(received_data, my_data) */ smpi_op_apply(op, incoming_data + recv_offset*true_extent, outgoing_data + recv_offset*true_extent, &size, &datatype ); /* buf0_was_inout = buf0_was_inout; */ } else { /* lower ranked value so need to call op(my_data, received_data) */ smpi_op_apply( op, outgoing_data + recv_offset*true_extent, incoming_data + recv_offset*true_extent, &size, &datatype); buf0_was_inout = !buf0_was_inout; } /* the next round of send/recv needs to happen within the block (of size "size") that we just received and reduced */ send_offset = recv_offset; } xbt_assert(size == recvcounts[rank]); /* copy the reduced data to the recvbuf */ result_ptr = (char *)(buf0_was_inout ? tmp_buf0 : tmp_buf1) + recv_offset * true_extent; mpi_errno = smpi_datatype_copy(result_ptr, size, datatype, recvbuf, size, datatype); smpi_free_tmp_buffer(tmp_buf0_save); smpi_free_tmp_buffer(tmp_buf1_save); if (mpi_errno) return(mpi_errno); return MPI_SUCCESS; } int smpi_coll_tuned_reduce_scatter_mpich_rdb(void *sendbuf, void *recvbuf, int recvcounts[], MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { int rank, comm_size, i; MPI_Aint extent, true_extent, true_lb; int *disps; void *tmp_recvbuf, *tmp_results; int mpi_errno = MPI_SUCCESS; int dis[2], blklens[2], total_count, dst; int mask, dst_tree_root, my_tree_root, j, k; int received; MPI_Datatype sendtype, recvtype; int nprocs_completed, tmp_mask, tree_root, is_commutative=0; comm_size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); extent =smpi_datatype_get_extent(datatype); smpi_datatype_extent(datatype, &true_lb, &true_extent); if (smpi_op_is_commute(op)) { is_commutative = 1; } disps = (int*)xbt_malloc( comm_size * sizeof(int)); total_count = 0; for (i=0; i> i; dst_tree_root <<= i; my_tree_root = rank >> i; my_tree_root <<= i; /* At step 1, processes exchange (n-n/p) amount of data; at step 2, (n-2n/p) amount of data; at step 3, (n-4n/p) amount of data, and so forth. We use derived datatypes for this. At each step, a process does not need to send data indexed from my_tree_root to my_tree_root+mask-1. Similarly, a process won't receive data indexed from dst_tree_root to dst_tree_root+mask-1. */ /* calculate sendtype */ blklens[0] = blklens[1] = 0; for (j=0; j comm_size) { nprocs_completed = comm_size - my_tree_root - mask; /* nprocs_completed is the number of processes in this subtree that have all the data. Send data to others in a tree fashion. First find root of current tree that is being divided into two. k is the number of least-significant bits in this process's rank that must be zeroed out to find the rank of the root */ j = mask; k = 0; while (j) { j >>= 1; k++; } k--; tmp_mask = mask >> 1; while (tmp_mask) { dst = rank ^ tmp_mask; tree_root = rank >> k; tree_root <<= k; /* send only if this proc has data and destination doesn't have data. at any step, multiple processes can send if they have the data */ if ((dst > rank) && (rank < tree_root + nprocs_completed) && (dst >= tree_root + nprocs_completed)) { /* send the current result */ smpi_mpi_send(tmp_recvbuf, 1, recvtype, dst, COLL_TAG_SCATTER, comm); } /* recv only if this proc. doesn't have data and sender has data */ else if ((dst < rank) && (dst < tree_root + nprocs_completed) && (rank >= tree_root + nprocs_completed)) { smpi_mpi_recv(tmp_recvbuf, 1, recvtype, dst, COLL_TAG_SCATTER, comm, MPI_STATUS_IGNORE); received = 1; } tmp_mask >>= 1; k--; } } /* The following reduction is done here instead of after the MPIC_Sendrecv_ft or MPIC_Recv_ft above. This is because to do it above, in the noncommutative case, we would need an extra temp buffer so as not to overwrite temp_recvbuf, because temp_recvbuf may have to be communicated to other processes in the non-power-of-two case. To avoid that extra allocation, we do the reduce here. */ if (received) { if (is_commutative || (dst_tree_root < my_tree_root)) { { smpi_op_apply(op, tmp_recvbuf, tmp_results, &blklens[0], &datatype); smpi_op_apply(op, ((char *)tmp_recvbuf + dis[1]*extent), ((char *)tmp_results + dis[1]*extent), &blklens[1], &datatype); } } else { { smpi_op_apply(op, tmp_results, tmp_recvbuf, &blklens[0], &datatype); smpi_op_apply(op, ((char *)tmp_results + dis[1]*extent), ((char *)tmp_recvbuf + dis[1]*extent), &blklens[1], &datatype); } /* copy result back into tmp_results */ mpi_errno = smpi_datatype_copy(tmp_recvbuf, 1, recvtype, tmp_results, 1, recvtype); if (mpi_errno) return(mpi_errno); } } smpi_datatype_unuse(sendtype); smpi_datatype_unuse(recvtype); mask <<= 1; i++; } /* now copy final results from tmp_results to recvbuf */ mpi_errno = smpi_datatype_copy(((char *)tmp_results+disps[rank]*extent), recvcounts[rank], datatype, recvbuf, recvcounts[rank], datatype); if (mpi_errno) return(mpi_errno); xbt_free(disps); smpi_free_tmp_buffer(tmp_recvbuf); smpi_free_tmp_buffer(tmp_results); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/alltoall-pair-light-barrier.c000644 001750 001750 00000005246 13031031271 025403 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** * Function: alltoall_pair_light_barrier * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works in P - 1 steps. In step i, node j exchanges data with node i ^ j. Light barriers are inserted between communications in different phases. * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_alltoall_pair_light_barrier(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint send_chunk, recv_chunk; MPI_Status s; int i, src, dst, rank, num_procs, next_partner; int tag = COLL_TAG_ALLTOALL; /*, failure = 0; */ char send_sync = 'a', recv_sync = 'b'; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "alltoall pair algorithm can't be used with non power of two number of processes ! "); send_chunk = smpi_datatype_get_extent(send_type); recv_chunk = smpi_datatype_get_extent(recv_type); send_chunk *= send_count; recv_chunk *= recv_count; smpi_mpi_sendrecv(send_ptr + rank * send_chunk, send_count, send_type, rank, tag, recv_ptr + rank * recv_chunk, recv_count, recv_type, rank, tag, comm, &s); for (i = 1; i < num_procs; i++) { src = dst = rank ^ i; smpi_mpi_sendrecv(send_ptr + dst * send_chunk, send_count, send_type, dst, tag, recv_ptr + src * recv_chunk, recv_count, recv_type, src, tag, comm, &s); if ((i + 1) < num_procs) { next_partner = rank ^ (i + 1); smpi_mpi_sendrecv(&send_sync, 1, MPI_CHAR, next_partner, tag, &recv_sync, 1, MPI_CHAR, next_partner, tag, comm, &s); } } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allgather-smp-simple.c000644 001750 001750 00000011354 13031031271 024141 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" int smpi_coll_tuned_allgather_smp_simple(void *send_buf, int scount, MPI_Datatype stype, void *recv_buf, int rcount, MPI_Datatype rtype, MPI_Comm comm) { int src, dst, comm_size, rank; comm_size = smpi_comm_size(comm); if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } int num_core=1; if (smpi_comm_is_uniform(comm)){ num_core = smpi_comm_size(smpi_comm_get_intra_comm(comm)); } if(comm_size%num_core) THROWF(arg_error,0, "allgather SMP simple algorithm can't be used with non multiple of NUM_CORE=%d number of processes ! ", num_core); rank = smpi_comm_rank(comm); MPI_Aint rextent, sextent; rextent = smpi_datatype_get_extent(rtype); sextent = smpi_datatype_get_extent(stype); int tag = COLL_TAG_ALLGATHER; MPI_Status status; int i, send_offset, recv_offset; int intra_rank, inter_rank; intra_rank = rank % num_core; inter_rank = rank / num_core; int inter_comm_size = (comm_size + num_core - 1) / num_core; int num_core_in_current_smp = num_core; // the last SMP node may have fewer number of running processes than all others if (inter_rank == (inter_comm_size - 1)) { num_core_in_current_smp = comm_size - (inter_rank * num_core); } //INTRA-SMP-ALLGATHER recv_offset = rank * rextent * rcount; smpi_mpi_sendrecv(send_buf, scount, stype, rank, tag, ((char *) recv_buf + recv_offset), rcount, rtype, rank, tag, comm, &status); for (i = 1; i < num_core_in_current_smp; i++) { dst = (inter_rank * num_core) + (intra_rank + i) % (num_core_in_current_smp); src = (inter_rank * num_core) + (intra_rank - i + num_core_in_current_smp) % (num_core_in_current_smp); recv_offset = src * rextent * rcount; smpi_mpi_sendrecv(send_buf, scount, stype, dst, tag, ((char *) recv_buf + recv_offset), rcount, rtype, src, tag, comm, &status); } // INTER-SMP-ALLGATHER // Every root of each SMP node post INTER-Sendrecv, then do INTRA-Bcast for each receiving message if (intra_rank == 0) { MPI_Request *reqs, *req_ptr; int num_req = (inter_comm_size - 1) * 2; reqs = (MPI_Request *) xbt_malloc(num_req * sizeof(MPI_Request)); req_ptr = reqs; MPI_Status *stat; stat = (MPI_Status *) xbt_malloc(num_req * sizeof(MPI_Status)); for (i = 1; i < inter_comm_size; i++) { //dst = ((inter_rank+i)%inter_comm_size) * num_core; src = ((inter_rank - i + inter_comm_size) % inter_comm_size) * num_core; //send_offset = (rank * sextent * scount); recv_offset = (src * sextent * scount); // smpi_mpi_sendrecv((recv_buf+send_offset), (scount * num_core), stype, dst, tag, // (recv_buf+recv_offset), (rcount * num_core), rtype, src, tag, comm, &status); //MPIC_Isend((recv_buf+send_offset), (scount * num_core), stype, dst, tag, comm, req_ptr++); *(req_ptr++) = smpi_mpi_irecv(((char *) recv_buf + recv_offset), (rcount * num_core), rtype, src, tag, comm); } for (i = 1; i < inter_comm_size; i++) { dst = ((inter_rank + i) % inter_comm_size) * num_core; //src = ((inter_rank-i+inter_comm_size)%inter_comm_size) * num_core; send_offset = (rank * sextent * scount); //recv_offset = (src * sextent * scount); // smpi_mpi_sendrecv((recv_buf+send_offset), (scount * num_core), stype, dst, tag, // (recv_buf+recv_offset), (rcount * num_core), rtype, src, tag, comm, &status); *(req_ptr++) = smpi_mpi_isend(((char *) recv_buf + send_offset), (scount * num_core), stype, dst, tag, comm); //MPIC_Irecv((recv_buf+recv_offset), (rcount * num_core), rtype, src, tag, comm, req_ptr++); } smpi_mpi_waitall(num_req, reqs, stat); free(reqs); free(stat); } //INTRA-BCAST (use flat tree) if (intra_rank == 0) { for (i = 1; i < num_core_in_current_smp; i++) { //printf("rank = %d, num = %d send to %d\n",rank, num_core_in_current_smp, (rank + i)); smpi_mpi_send(recv_buf, (scount * comm_size), stype, (rank + i), tag, comm); } } else { //printf("rank = %d recv from %d\n",rank, (inter_rank * num_core)); smpi_mpi_recv(recv_buf, (rcount * comm_size), rtype, (inter_rank * num_core), tag, comm, &status); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allgather-mvapich-smp.c000644 001750 001750 00000014261 13031031271 024277 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2009 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * * Additional copyrights may follow */ /* -*- Mode: C; c-basic-offset:4 ; -*- */ /* Copyright (c) 2001-2014, The Ohio State University. All rights * reserved. * * This file is part of the MVAPICH2 software package developed by the * team members of The Ohio State University's Network-Based Computing * Laboratory (NBCL), headed by Professor Dhabaleswar K. (DK) Panda. * * For detailed copyright and licensing information, please refer to the * copyright file COPYRIGHT in the top level MVAPICH2 directory. */ /* * * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ #include "colls_private.h" int smpi_coll_tuned_allgather_mvapich2_smp(void *sendbuf,int sendcnt, MPI_Datatype sendtype, void *recvbuf, int recvcnt,MPI_Datatype recvtype, MPI_Comm comm) { int rank, size; int local_rank, local_size; int leader_comm_size = 0; int mpi_errno = MPI_SUCCESS; MPI_Aint recvtype_extent = 0; /* Datatype extent */ MPI_Comm shmem_comm, leader_comm; if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } if(!smpi_comm_is_uniform(comm) || !smpi_comm_is_blocked(comm)) THROWF(arg_error,0, "allgather MVAPICH2 smp algorithm can't be used with irregular deployment. Please insure that processes deployed on the same node are contiguous and that each node has the same number of processes"); if (recvcnt == 0) { return MPI_SUCCESS; } rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); /* extract the rank,size information for the intra-node communicator */ recvtype_extent=smpi_datatype_get_extent(recvtype); shmem_comm = smpi_comm_get_intra_comm(comm); local_rank = smpi_comm_rank(shmem_comm); local_size = smpi_comm_size(shmem_comm); if (local_rank == 0) { /* Node leader. Extract the rank, size information for the leader communicator */ leader_comm = smpi_comm_get_leaders_comm(comm); if(leader_comm==MPI_COMM_NULL){ leader_comm = MPI_COMM_WORLD; } leader_comm_size = smpi_comm_size(leader_comm); } /*If there is just one node, after gather itself, * root has all the data and it can do bcast*/ if(local_rank == 0) { mpi_errno = mpi_coll_gather_fun(sendbuf, sendcnt,sendtype, (void*)((char*)recvbuf + (rank * recvcnt * recvtype_extent)), recvcnt, recvtype, 0, shmem_comm); } else { /*Since in allgather all the processes could have * its own data in place*/ if(sendbuf == MPI_IN_PLACE) { mpi_errno = mpi_coll_gather_fun((void*)((char*)recvbuf + (rank * recvcnt * recvtype_extent)), recvcnt , recvtype, recvbuf, recvcnt, recvtype, 0, shmem_comm); } else { mpi_errno = mpi_coll_gather_fun(sendbuf, sendcnt,sendtype, recvbuf, recvcnt, recvtype, 0, shmem_comm); } } /* Exchange the data between the node leaders*/ if (local_rank == 0 && (leader_comm_size > 1)) { /*When data in each socket is different*/ if (smpi_comm_is_uniform(comm) != 1) { int *displs = NULL; int *recvcnts = NULL; int *node_sizes = NULL; int i = 0; node_sizes = smpi_comm_get_non_uniform_map(comm); displs = xbt_malloc(sizeof (int) * leader_comm_size); recvcnts = xbt_malloc(sizeof (int) * leader_comm_size); if (!displs || !recvcnts) { return MPI_ERR_OTHER; } recvcnts[0] = node_sizes[0] * recvcnt; displs[0] = 0; for (i = 1; i < leader_comm_size; i++) { displs[i] = displs[i - 1] + node_sizes[i - 1] * recvcnt; recvcnts[i] = node_sizes[i] * recvcnt; } void* sendbuf=((char*)recvbuf)+smpi_datatype_get_extent(recvtype)*displs[smpi_comm_rank(leader_comm)]; mpi_errno = mpi_coll_allgatherv_fun(sendbuf, (recvcnt*local_size), recvtype, recvbuf, recvcnts, displs, recvtype, leader_comm); xbt_free(displs); xbt_free(recvcnts); } else { void* sendtmpbuf=((char*)recvbuf)+smpi_datatype_get_extent(recvtype)*(recvcnt*local_size)*smpi_comm_rank(leader_comm); mpi_errno = smpi_coll_tuned_allgather_mpich(sendtmpbuf, (recvcnt*local_size), recvtype, recvbuf, (recvcnt*local_size), recvtype, leader_comm); } } /*Bcast the entire data from node leaders to all other cores*/ mpi_errno = mpi_coll_bcast_fun (recvbuf, recvcnt * size, recvtype, 0, shmem_comm); return mpi_errno; } SimGrid-3.14.159/src/smpi/colls/alltoall-pair.c000644 001750 001750 00000005756 13031031271 022660 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** * Function: alltoall_pair * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works when P is power of two. In each phase of P - 1 phases, nodes in pair communicate their data. * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_alltoall_pair_rma(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint send_chunk, recv_chunk; MPI_Win win; int assert = 0; int i, dst, rank, num_procs; char *send_ptr = (char *) send_buff; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); send_chunk = smpi_datatype_get_extent(send_type); recv_chunk = smpi_datatype_get_extent(recv_type); win=smpi_mpi_win_create(recv_buff, num_procs * recv_chunk * send_count, recv_chunk, 0, comm); send_chunk *= send_count; recv_chunk *= recv_count; smpi_mpi_win_fence(assert, win); for (i = 0; i < num_procs; i++) { dst = rank ^ i; smpi_mpi_put(send_ptr + dst * send_chunk, send_count, send_type, dst, rank /* send_chunk*/, send_count, send_type, win); } smpi_mpi_win_fence(assert, win); smpi_mpi_win_free(&win); return 0; } int smpi_coll_tuned_alltoall_pair(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint send_chunk, recv_chunk; MPI_Status s; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALL; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "alltoall pair algorithm can't be used with non power of two number of processes ! "); send_chunk = smpi_datatype_get_extent(send_type); recv_chunk = smpi_datatype_get_extent(recv_type); send_chunk *= send_count; recv_chunk *= recv_count; for (i = 0; i < num_procs; i++) { src = dst = rank ^ i; smpi_mpi_sendrecv(send_ptr + dst * send_chunk, send_count, send_type, dst, tag, recv_ptr + src * recv_chunk, recv_count, recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allgather-3dmesh.c000644 001750 001750 00000016371 13031031271 023242 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** Copyright (c) 2006, Ahmad Faraj & Xin Yuan, All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Florida State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ************************************************************************* * Any results obtained from executing this software require the * * acknowledgment and citation of the software and its owners. * * The full citation is given below: * * * * A. Faraj and X. Yuan. "Automatic Generation and Tuning of MPI * * Collective Communication Routines." The 19th ACM International * * Conference on Supercomputing (ICS), Cambridge, Massachusetts, * * June 20-22, 2005. * ************************************************************************* *****************************************************************************/ /***************************************************************************** * Function: is_2dmesh * return: int * num: the number of processors in a communicator * i: x dimension * j: y dimension * k: z dimension * descp: takes a number and tries to find a factoring of x*y*z mesh out of it ****************************************************************************/ #ifndef THREED #define THREED static int is_3dmesh(int num, int *i, int *j, int *k) { int x, max = num / 3; x = cbrt(num); *i = *j = *k = 0; while (x <= max) { if ((num % (x * x)) == 0) { *i = *j = x; *k = num / (x * x); return 1; } x++; } return 0; } #endif /***************************************************************************** * Function: allgather_3dmesh_shoot * return: int * send_buff: send input buffer * send_count: number of elements to send * send_type: data type of elements being sent * recv_buff: receive output buffer * recv_count: number of elements to received * recv_type: data type of elements being received * comm: communication * Descrp: Function realizes the allgather operation using the 2dmesh * algorithm. Allgather ommunication occurs first in the x dimension, y * dimension, and then in the z dimension. Communication in each dimension * follows "simple" * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_allgather_3dmesh(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Request *req, *req_ptr; MPI_Aint extent; int i, src, dst, rank, num_procs, block_size, my_z_base; int my_z, X, Y, Z, send_offset, recv_offset; int two_dsize, my_row_base, my_col_base, src_row_base, src_z_base, num_reqs; int tag = COLL_TAG_ALLGATHER; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); extent = smpi_datatype_get_extent(send_type); if (!is_3dmesh(num_procs, &X, &Y, &Z)) THROWF(arg_error,0, "allgather_3dmesh algorithm can't be used with this number of processes! "); num_reqs = X; if (Y > X) num_reqs = Y; if (Z > Y) num_reqs = Z; two_dsize = X * Y; my_z = rank / two_dsize; my_row_base = (rank / X) * X; my_col_base = (rank % Y) + (my_z * two_dsize); my_z_base = my_z * two_dsize; block_size = extent * send_count; req = (MPI_Request *) xbt_malloc(num_reqs * sizeof(MPI_Request)); req_ptr = req; // do local allgather/local copy recv_offset = rank * block_size; smpi_datatype_copy(send_buff, send_count, send_type, (char *)recv_buff + recv_offset, recv_count, recv_type); // do rowwise comm for (i = 0; i < Y; i++) { src = i + my_row_base; if (src == rank) continue; recv_offset = src * block_size; *(req_ptr++) = smpi_mpi_irecv((char *)recv_buff + recv_offset, send_count, recv_type, src, tag, comm); } for (i = 0; i < Y; i++) { dst = i + my_row_base; if (dst == rank) continue; smpi_mpi_send(send_buff, send_count, send_type, dst, tag, comm); } smpi_mpi_waitall(Y - 1, req, MPI_STATUSES_IGNORE); req_ptr = req; // do colwise comm, it does not matter here if i*X or i *Y since X == Y for (i = 0; i < X; i++) { src = (i * Y + my_col_base); if (src == rank) continue; src_row_base = (src / X) * X; recv_offset = src_row_base * block_size; *(req_ptr++) = smpi_mpi_irecv((char *)recv_buff + recv_offset, recv_count * Y, recv_type, src, tag, comm); } send_offset = my_row_base * block_size; for (i = 0; i < X; i++) { dst = (i * Y + my_col_base); if (dst == rank) continue; smpi_mpi_send((char *)recv_buff + send_offset, send_count * Y, send_type, dst, tag, comm); } smpi_mpi_waitall(X - 1, req, MPI_STATUSES_IGNORE); req_ptr = req; for (i = 1; i < Z; i++) { src = (rank + i * two_dsize) % num_procs; src_z_base = (src / two_dsize) * two_dsize; recv_offset = (src_z_base * block_size); *(req_ptr++) = smpi_mpi_irecv((char *)recv_buff + recv_offset, recv_count * two_dsize, recv_type, src, tag, comm); } for (i = 1; i < Z; i++) { dst = (rank + i * two_dsize) % num_procs; send_offset = my_z_base * block_size; smpi_mpi_send((char *)recv_buff + send_offset, send_count * two_dsize, send_type, dst, tag, comm); } smpi_mpi_waitall(Z - 1, req, MPI_STATUSES_IGNORE); free(req); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allgatherv-mpich-ring.c000644 001750 001750 00000010030 13031031271 024264 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ #include "colls_private.h" /***************************************************************************** * Function: allgather_mpich_ring * return: int * inputs: * send_buff: send input buffer * send_count: number of elements to send * send_type: data type of elements being sent * recv_buff: receive output buffer * recv_count: number of elements to received * recv_type: data type of elements being received * comm: communication ****************************************************************************/ int smpi_coll_tuned_allgatherv_mpich_ring(void *sendbuf, int sendcount, MPI_Datatype send_type, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvtype, MPI_Comm comm) { char * sbuf = NULL, * rbuf = NULL; int soffset, roffset; int torecv=0, tosend=0, min, rank, comm_size; int sendnow, recvnow; int sidx, ridx; MPI_Status status; MPI_Aint recvtype_extent; int right, left, total_count, i; rank= smpi_comm_rank(comm); comm_size=smpi_comm_size(comm); recvtype_extent= smpi_datatype_get_extent( recvtype); total_count = 0; for (i=0; i recvcounts[i]) min = recvcounts[i]; if (min * recvtype_extent < 32768*8) min = 32768*8 / recvtype_extent; /* Handle the case where the datatype extent is larger than * the pipeline size. */ if (!min) min = 1; sidx = rank; ridx = left; soffset = 0; roffset = 0; while (tosend || torecv) { /* While we have data to send or receive */ sendnow = ((recvcounts[sidx] - soffset) > min) ? min : (recvcounts[sidx] - soffset); recvnow = ((recvcounts[ridx] - roffset) > min) ? min : (recvcounts[ridx] - roffset); sbuf = (char *)recvbuf + ((displs[sidx] + soffset) * recvtype_extent); rbuf = (char *)recvbuf + ((displs[ridx] + roffset) * recvtype_extent); /* Protect against wrap-around of indices */ if (!tosend) sendnow = 0; if (!torecv) recvnow = 0; /* Communicate */ if (!sendnow && !recvnow) { /* Don't do anything. This case is possible if two * consecutive processes contribute 0 bytes each. */ } else if (!sendnow) { /* If there's no data to send, just do a recv call */ smpi_mpi_recv(rbuf, recvnow, recvtype, left, COLL_TAG_ALLGATHERV, comm, &status); torecv -= recvnow; } else if (!recvnow) { /* If there's no data to receive, just do a send call */ smpi_mpi_send(sbuf, sendnow, recvtype, right, COLL_TAG_ALLGATHERV, comm); tosend -= sendnow; } else { /* There's data to be sent and received */ smpi_mpi_sendrecv(sbuf, sendnow, recvtype, right, COLL_TAG_ALLGATHERV, rbuf, recvnow, recvtype, left, COLL_TAG_ALLGATHERV, comm, &status); tosend -= sendnow; torecv -= recvnow; } soffset += sendnow; roffset += recvnow; if (soffset == recvcounts[sidx]) { soffset = 0; sidx = (sidx + comm_size - 1) % comm_size; } if (roffset == recvcounts[ridx]) { roffset = 0; ridx = (ridx + comm_size - 1) % comm_size; } } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/scatter-mvapich-two-level.c000644 001750 001750 00000042415 13031031272 025123 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2009 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * * Additional copyrights may follow */ /* -*- Mode: C; c-basic-offset:4 ; -*- */ /* Copyright (c) 2001-2014, The Ohio State University. All rights * reserved. * * This file is part of the MVAPICH2 software package developed by the * team members of The Ohio State University's Network-Based Computing * Laboratory (NBCL), headed by Professor Dhabaleswar K. (DK) Panda. * * For detailed copyright and licensing information, please refer to the * copyright file COPYRIGHT in the top level MVAPICH2 directory. */ /* * * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ #include "colls_private.h" #define MPIR_Scatter_MV2_Binomial smpi_coll_tuned_scatter_ompi_binomial #define MPIR_Scatter_MV2_Direct smpi_coll_tuned_scatter_ompi_basic_linear extern int (*MV2_Scatter_intra_function) (void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm); int smpi_coll_tuned_scatter_mvapich2_two_level_direct(void *sendbuf, int sendcnt, MPI_Datatype sendtype, void *recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm) { int comm_size, rank; int local_rank, local_size; int leader_comm_rank = -1, leader_comm_size = -1; int mpi_errno = MPI_SUCCESS; int recvtype_size, sendtype_size, nbytes; void *tmp_buf = NULL; void *leader_scatter_buf = NULL; MPI_Status status; int leader_root, leader_of_root = -1; MPI_Comm shmem_comm, leader_comm; //if not set (use of the algo directly, without mvapich2 selector) if(MV2_Scatter_intra_function==NULL) MV2_Scatter_intra_function=smpi_coll_tuned_scatter_mpich; if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } comm_size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); if (((rank == root) && (recvcnt == 0)) || ((rank != root) && (sendcnt == 0))) { return MPI_SUCCESS; } /* extract the rank,size information for the intra-node * communicator */ shmem_comm = smpi_comm_get_intra_comm(comm); local_rank = smpi_comm_rank(shmem_comm); local_size = smpi_comm_size(shmem_comm); if (local_rank == 0) { /* Node leader. Extract the rank, size information for the leader * communicator */ leader_comm = smpi_comm_get_leaders_comm(comm); leader_comm_size = smpi_comm_size(leader_comm); leader_comm_rank = smpi_comm_rank(leader_comm); } if (local_size == comm_size) { /* purely intra-node scatter. Just use the direct algorithm and we are done */ mpi_errno = MPIR_Scatter_MV2_Direct(sendbuf, sendcnt, sendtype, recvbuf, recvcnt, recvtype, root, comm); } else { recvtype_size=smpi_datatype_size(recvtype); sendtype_size=smpi_datatype_size(sendtype); if (rank == root) { nbytes = sendcnt * sendtype_size; } else { nbytes = recvcnt * recvtype_size; } if (local_rank == 0) { /* Node leader, allocate tmp_buffer */ tmp_buf = smpi_get_tmp_sendbuffer(nbytes * local_size); } leader_comm = smpi_comm_get_leaders_comm(comm); int* leaders_map = smpi_comm_get_leaders_map(comm); leader_of_root = smpi_group_rank(smpi_comm_group(comm),leaders_map[root]); leader_root = smpi_group_rank(smpi_comm_group(leader_comm),leaders_map[root]); /* leader_root is the rank of the leader of the root in leader_comm. * leader_root is to be used as the root of the inter-leader gather ops */ if ((local_rank == 0) && (root != rank) && (leader_of_root == rank)) { /* The root of the scatter operation is not the node leader. Recv * data from the node leader */ leader_scatter_buf = smpi_get_tmp_sendbuffer(nbytes * comm_size); smpi_mpi_recv(leader_scatter_buf, nbytes * comm_size, MPI_BYTE, root, COLL_TAG_SCATTER, comm, &status); } if (rank == root && local_rank != 0) { /* The root of the scatter operation is not the node leader. Send * data to the node leader */ smpi_mpi_send(sendbuf, sendcnt * comm_size, sendtype, leader_of_root, COLL_TAG_SCATTER, comm ); } if (leader_comm_size > 1 && local_rank == 0) { if (!smpi_comm_is_uniform(comm)) { int *displs = NULL; int *sendcnts = NULL; int *node_sizes; int i = 0; node_sizes = smpi_comm_get_non_uniform_map(comm); if (root != leader_of_root) { if (leader_comm_rank == leader_root) { displs = xbt_malloc(sizeof (int) * leader_comm_size); sendcnts = xbt_malloc(sizeof (int) * leader_comm_size); sendcnts[0] = node_sizes[0] * nbytes; displs[0] = 0; for (i = 1; i < leader_comm_size; i++) { displs[i] = displs[i - 1] + node_sizes[i - 1] * nbytes; sendcnts[i] = node_sizes[i] * nbytes; } } smpi_mpi_scatterv(leader_scatter_buf, sendcnts, displs, MPI_BYTE, tmp_buf, nbytes * local_size, MPI_BYTE, leader_root, leader_comm); } else { if (leader_comm_rank == leader_root) { displs = xbt_malloc(sizeof (int) * leader_comm_size); sendcnts = xbt_malloc(sizeof (int) * leader_comm_size); sendcnts[0] = node_sizes[0] * sendcnt; displs[0] = 0; for (i = 1; i < leader_comm_size; i++) { displs[i] = displs[i - 1] + node_sizes[i - 1] * sendcnt; sendcnts[i] = node_sizes[i] * sendcnt; } } smpi_mpi_scatterv(sendbuf, sendcnts, displs, sendtype, tmp_buf, nbytes * local_size, MPI_BYTE, leader_root, leader_comm); } if (leader_comm_rank == leader_root) { xbt_free(displs); xbt_free(sendcnts); } } else { if (leader_of_root != root) { mpi_errno = MPIR_Scatter_MV2_Direct(leader_scatter_buf, nbytes * local_size, MPI_BYTE, tmp_buf, nbytes * local_size, MPI_BYTE, leader_root, leader_comm); } else { mpi_errno = MPIR_Scatter_MV2_Direct(sendbuf, sendcnt * local_size, sendtype, tmp_buf, nbytes * local_size, MPI_BYTE, leader_root, leader_comm); } } } /* The leaders are now done with the inter-leader part. Scatter the data within the nodes */ if (rank == root && recvbuf == MPI_IN_PLACE) { mpi_errno = MV2_Scatter_intra_function(tmp_buf, nbytes, MPI_BYTE, (void *)sendbuf, sendcnt, sendtype, 0, shmem_comm); } else { mpi_errno = MV2_Scatter_intra_function(tmp_buf, nbytes, MPI_BYTE, recvbuf, recvcnt, recvtype, 0, shmem_comm); } } /* check if multiple threads are calling this collective function */ if (comm_size != local_size && local_rank == 0) { smpi_free_tmp_buffer(tmp_buf); if (leader_of_root == rank && root != rank) { smpi_free_tmp_buffer(leader_scatter_buf); } } return (mpi_errno); } int smpi_coll_tuned_scatter_mvapich2_two_level_binomial(void *sendbuf, int sendcnt, MPI_Datatype sendtype, void *recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm) { int comm_size, rank; int local_rank, local_size; int leader_comm_rank = -1, leader_comm_size = -1; int mpi_errno = MPI_SUCCESS; int recvtype_size, sendtype_size, nbytes; void *tmp_buf = NULL; void *leader_scatter_buf = NULL; MPI_Status status; int leader_root = -1, leader_of_root = -1; MPI_Comm shmem_comm, leader_comm; //if not set (use of the algo directly, without mvapich2 selector) if(MV2_Scatter_intra_function==NULL) MV2_Scatter_intra_function=smpi_coll_tuned_scatter_mpich; if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } comm_size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); if (((rank == root) && (recvcnt == 0)) || ((rank != root) && (sendcnt == 0))) { return MPI_SUCCESS; } /* extract the rank,size information for the intra-node * communicator */ shmem_comm = smpi_comm_get_intra_comm(comm); local_rank = smpi_comm_rank(shmem_comm); local_size = smpi_comm_size(shmem_comm); if (local_rank == 0) { /* Node leader. Extract the rank, size information for the leader * communicator */ leader_comm = smpi_comm_get_leaders_comm(comm); leader_comm_size = smpi_comm_size(leader_comm); leader_comm_rank = smpi_comm_rank(leader_comm); } if (local_size == comm_size) { /* purely intra-node scatter. Just use the direct algorithm and we are done */ mpi_errno = MPIR_Scatter_MV2_Direct(sendbuf, sendcnt, sendtype, recvbuf, recvcnt, recvtype, root, comm); } else { recvtype_size=smpi_datatype_size(recvtype); sendtype_size=smpi_datatype_size(sendtype); if (rank == root) { nbytes = sendcnt * sendtype_size; } else { nbytes = recvcnt * recvtype_size; } if (local_rank == 0) { /* Node leader, allocate tmp_buffer */ tmp_buf = smpi_get_tmp_sendbuffer(nbytes * local_size); } leader_comm = smpi_comm_get_leaders_comm(comm); int* leaders_map = smpi_comm_get_leaders_map(comm); leader_of_root = smpi_group_rank(smpi_comm_group(comm),leaders_map[root]); leader_root = smpi_group_rank(smpi_comm_group(leader_comm),leaders_map[root]); /* leader_root is the rank of the leader of the root in leader_comm. * leader_root is to be used as the root of the inter-leader gather ops */ if ((local_rank == 0) && (root != rank) && (leader_of_root == rank)) { /* The root of the scatter operation is not the node leader. Recv * data from the node leader */ leader_scatter_buf = smpi_get_tmp_sendbuffer(nbytes * comm_size); smpi_mpi_recv(leader_scatter_buf, nbytes * comm_size, MPI_BYTE, root, COLL_TAG_SCATTER, comm, &status); } if (rank == root && local_rank != 0) { /* The root of the scatter operation is not the node leader. Send * data to the node leader */ smpi_mpi_send(sendbuf, sendcnt * comm_size, sendtype, leader_of_root, COLL_TAG_SCATTER, comm); } if (leader_comm_size > 1 && local_rank == 0) { if (!smpi_comm_is_uniform(comm)) { int *displs = NULL; int *sendcnts = NULL; int *node_sizes; int i = 0; node_sizes = smpi_comm_get_non_uniform_map(comm); if (root != leader_of_root) { if (leader_comm_rank == leader_root) { displs = xbt_malloc(sizeof (int) * leader_comm_size); sendcnts = xbt_malloc(sizeof (int) * leader_comm_size); sendcnts[0] = node_sizes[0] * nbytes; displs[0] = 0; for (i = 1; i < leader_comm_size; i++) { displs[i] = displs[i - 1] + node_sizes[i - 1] * nbytes; sendcnts[i] = node_sizes[i] * nbytes; } } smpi_mpi_scatterv(leader_scatter_buf, sendcnts, displs, MPI_BYTE, tmp_buf, nbytes * local_size, MPI_BYTE, leader_root, leader_comm); } else { if (leader_comm_rank == leader_root) { displs = xbt_malloc(sizeof (int) * leader_comm_size); sendcnts = xbt_malloc(sizeof (int) * leader_comm_size); sendcnts[0] = node_sizes[0] * sendcnt; displs[0] = 0; for (i = 1; i < leader_comm_size; i++) { displs[i] = displs[i - 1] + node_sizes[i - 1] * sendcnt; sendcnts[i] = node_sizes[i] * sendcnt; } } smpi_mpi_scatterv(sendbuf, sendcnts, displs, sendtype, tmp_buf, nbytes * local_size, MPI_BYTE, leader_root, leader_comm); } if (leader_comm_rank == leader_root) { xbt_free(displs); xbt_free(sendcnts); } } else { if (leader_of_root != root) { mpi_errno = MPIR_Scatter_MV2_Binomial(leader_scatter_buf, nbytes * local_size, MPI_BYTE, tmp_buf, nbytes * local_size, MPI_BYTE, leader_root, leader_comm); } else { mpi_errno = MPIR_Scatter_MV2_Binomial(sendbuf, sendcnt * local_size, sendtype, tmp_buf, nbytes * local_size, MPI_BYTE, leader_root, leader_comm); } } } /* The leaders are now done with the inter-leader part. Scatter the data within the nodes */ if (rank == root && recvbuf == MPI_IN_PLACE) { mpi_errno = MV2_Scatter_intra_function(tmp_buf, nbytes, MPI_BYTE, (void *)sendbuf, sendcnt, sendtype, 0, shmem_comm); } else { mpi_errno = MV2_Scatter_intra_function(tmp_buf, nbytes, MPI_BYTE, recvbuf, recvcnt, recvtype, 0, shmem_comm); } } /* check if multiple threads are calling this collective function */ if (comm_size != local_size && local_rank == 0) { smpi_free_tmp_buffer(tmp_buf); if (leader_of_root == rank && root != rank) { smpi_free_tmp_buffer(leader_scatter_buf); } } return (mpi_errno); } SimGrid-3.14.159/src/smpi/colls/smpi_openmpi_selector.c000644 001750 001750 00000063742 13031031272 024522 0ustar00mquinsonmquinson000000 000000 /* selector for collective algorithms based on openmpi's default coll_tuned_decision_fixed selector */ /* Copyright (c) 2009-2010, 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" int smpi_coll_tuned_allreduce_ompi(void *sbuf, void *rbuf, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { size_t dsize, block_dsize; int comm_size = smpi_comm_size(comm); const size_t intermediate_message = 10000; /** * Decision function based on MX results from the Grig cluster at UTK. * * Currently, linear, recursive doubling, and nonoverlapping algorithms * can handle both commutative and non-commutative operations. * Ring algorithm does not support non-commutative operations. */ dsize = smpi_datatype_size(dtype); block_dsize = dsize * count; if (block_dsize < intermediate_message) { return (smpi_coll_tuned_allreduce_rdb (sbuf, rbuf, count, dtype, op, comm)); } if( smpi_op_is_commute(op) && (count > comm_size) ) { const size_t segment_size = 1 << 20; /* 1 MB */ if ((comm_size * segment_size >= block_dsize)) { //FIXME: ok, these are not the right algorithms, try to find closer ones // lr is a good match for allreduce_ring (difference is mainly the use of sendrecv) return smpi_coll_tuned_allreduce_lr(sbuf, rbuf, count, dtype, op, comm); } else { return (smpi_coll_tuned_allreduce_ompi_ring_segmented (sbuf, rbuf, count, dtype, op, comm /*segment_size*/)); } } return (smpi_coll_tuned_allreduce_redbcast(sbuf, rbuf, count, dtype, op, comm)); } int smpi_coll_tuned_alltoall_ompi( void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, MPI_Comm comm) { int communicator_size; size_t dsize, block_dsize; communicator_size = smpi_comm_size(comm); /* Decision function based on measurement on Grig cluster at the University of Tennessee (2GB MX) up to 64 nodes. Has better performance for messages of intermediate sizes than the old one */ /* determine block size */ dsize = smpi_datatype_size(sdtype); block_dsize = dsize * scount; if ((block_dsize < 200) && (communicator_size > 12)) { return smpi_coll_tuned_alltoall_bruck(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } else if (block_dsize < 3000) { return smpi_coll_tuned_alltoall_basic_linear(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } return smpi_coll_tuned_alltoall_ring (sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } int smpi_coll_tuned_alltoallv_ompi(void *sbuf, int *scounts, int *sdisps, MPI_Datatype sdtype, void *rbuf, int *rcounts, int *rdisps, MPI_Datatype rdtype, MPI_Comm comm ) { /* For starters, just keep the original algorithm. */ return smpi_coll_tuned_alltoallv_ompi_basic_linear(sbuf, scounts, sdisps, sdtype, rbuf, rcounts, rdisps,rdtype, comm); } int smpi_coll_tuned_barrier_ompi(MPI_Comm comm) { int communicator_size = smpi_comm_size(comm); if( 2 == communicator_size ) return smpi_coll_tuned_barrier_ompi_two_procs(comm); /* * Basic optimisation. If we have a power of 2 number of nodes*/ /* * the use the recursive doubling algorithm, otherwise*/ /* * bruck is the one we want.*/ { int has_one = 0; for( ; communicator_size > 0; communicator_size >>= 1 ) { if( communicator_size & 0x1 ) { if( has_one ) return smpi_coll_tuned_barrier_ompi_bruck(comm); has_one = 1; } } } return smpi_coll_tuned_barrier_ompi_recursivedoubling(comm); } int smpi_coll_tuned_bcast_ompi(void *buff, int count, MPI_Datatype datatype, int root, MPI_Comm comm ) { /* Decision function based on MX results for messages up to 36MB and communicator sizes up to 64 nodes */ const size_t small_message_size = 2048; const size_t intermediate_message_size = 370728; const double a_p16 = 3.2118e-6; /* [1 / byte] */ const double b_p16 = 8.7936; const double a_p64 = 2.3679e-6; /* [1 / byte] */ const double b_p64 = 1.1787; const double a_p128 = 1.6134e-6; /* [1 / byte] */ const double b_p128 = 2.1102; int communicator_size; //int segsize = 0; size_t message_size, dsize; communicator_size = smpi_comm_size(comm); /* else we need data size for decision function */ dsize = smpi_datatype_size(datatype); message_size = dsize * (unsigned long)count; /* needed for decision */ /* Handle messages of small and intermediate size, and single-element broadcasts */ if ((message_size < small_message_size) || (count <= 1)) { /* Binomial without segmentation */ return smpi_coll_tuned_bcast_binomial_tree (buff, count, datatype, root, comm); } else if (message_size < intermediate_message_size) { // SplittedBinary with 1KB segments return smpi_coll_tuned_bcast_ompi_split_bintree(buff, count, datatype, root, comm); } //Handle large message sizes else if (communicator_size < (a_p128 * message_size + b_p128)) { //Pipeline with 128KB segments //segsize = 1024 << 7; return smpi_coll_tuned_bcast_ompi_pipeline (buff, count, datatype, root, comm); } else if (communicator_size < 13) { // Split Binary with 8KB segments return smpi_coll_tuned_bcast_ompi_split_bintree(buff, count, datatype, root, comm); } else if (communicator_size < (a_p64 * message_size + b_p64)) { // Pipeline with 64KB segments //segsize = 1024 << 6; return smpi_coll_tuned_bcast_ompi_pipeline (buff, count, datatype, root, comm); } else if (communicator_size < (a_p16 * message_size + b_p16)) { //Pipeline with 16KB segments //segsize = 1024 << 4; return smpi_coll_tuned_bcast_ompi_pipeline (buff, count, datatype, root, comm); } /* Pipeline with 8KB segments */ //segsize = 1024 << 3; return smpi_coll_tuned_bcast_flattree_pipeline (buff, count, datatype, root, comm /*segsize*/); #if 0 /* this is based on gige measurements */ if (communicator_size < 4) { return smpi_coll_tuned_bcast_intra_basic_linear (buff, count, datatype, root, comm, module); } if (communicator_size == 4) { if (message_size < 524288) segsize = 0; else segsize = 16384; return smpi_coll_tuned_bcast_intra_bintree (buff, count, datatype, root, comm, module, segsize); } if (communicator_size <= 8 && message_size < 4096) { return smpi_coll_tuned_bcast_intra_basic_linear (buff, count, datatype, root, comm, module); } if (communicator_size > 8 && message_size >= 32768 && message_size < 524288) { segsize = 16384; return smpi_coll_tuned_bcast_intra_bintree (buff, count, datatype, root, comm, module, segsize); } if (message_size >= 524288) { segsize = 16384; return smpi_coll_tuned_bcast_intra_pipeline (buff, count, datatype, root, comm, module, segsize); } segsize = 0; /* once tested can swap this back in */ /* return smpi_coll_tuned_bcast_intra_bmtree (buff, count, datatype, root, comm, segsize); */ return smpi_coll_tuned_bcast_intra_bintree (buff, count, datatype, root, comm, module, segsize); #endif /* 0 */ } int smpi_coll_tuned_reduce_ompi( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm ) { int communicator_size=0; //int segsize = 0; size_t message_size, dsize; const double a1 = 0.6016 / 1024.0; /* [1/B] */ const double b1 = 1.3496; const double a2 = 0.0410 / 1024.0; /* [1/B] */ const double b2 = 9.7128; const double a3 = 0.0422 / 1024.0; /* [1/B] */ const double b3 = 1.1614; //const double a4 = 0.0033 / 1024.0; [1/B] //const double b4 = 1.6761; /* no limit on # of outstanding requests */ //const int max_requests = 0; communicator_size = smpi_comm_size(comm); /* need data size for decision function */ dsize=smpi_datatype_size(datatype); message_size = dsize * count; /* needed for decision */ /** * If the operation is non commutative we currently have choice of linear * or in-order binary tree algorithm. */ if( !smpi_op_is_commute(op) ) { if ((communicator_size < 12) && (message_size < 2048)) { return smpi_coll_tuned_reduce_ompi_basic_linear (sendbuf, recvbuf, count, datatype, op, root, comm/*, module*/); } return smpi_coll_tuned_reduce_ompi_in_order_binary (sendbuf, recvbuf, count, datatype, op, root, comm/*, module, 0, max_requests*/); } if ((communicator_size < 8) && (message_size < 512)){ /* Linear_0K */ return smpi_coll_tuned_reduce_ompi_basic_linear (sendbuf, recvbuf, count, datatype, op, root, comm); } else if (((communicator_size < 8) && (message_size < 20480)) || (message_size < 2048) || (count <= 1)) { /* Binomial_0K */ //segsize = 0; return smpi_coll_tuned_reduce_ompi_binomial(sendbuf, recvbuf, count, datatype, op, root, comm/*, module, segsize, max_requests*/); } else if (communicator_size > (a1 * message_size + b1)) { // Binomial_1K //segsize = 1024; return smpi_coll_tuned_reduce_ompi_binomial(sendbuf, recvbuf, count, datatype, op, root, comm/*, module, segsize, max_requests*/); } else if (communicator_size > (a2 * message_size + b2)) { // Pipeline_1K //segsize = 1024; return smpi_coll_tuned_reduce_ompi_pipeline (sendbuf, recvbuf, count, datatype, op, root, comm/*, module, segsize, max_requests*/); } else if (communicator_size > (a3 * message_size + b3)) { // Binary_32K //segsize = 32*1024; return smpi_coll_tuned_reduce_ompi_binary( sendbuf, recvbuf, count, datatype, op, root, comm/*, module, segsize, max_requests*/); } // if (communicator_size > (a4 * message_size + b4)) { // Pipeline_32K // segsize = 32*1024; // } else { // Pipeline_64K // segsize = 64*1024; // } return smpi_coll_tuned_reduce_ompi_pipeline (sendbuf, recvbuf, count, datatype, op, root, comm/*, module, segsize, max_requests*/); #if 0 /* for small messages use linear algorithm */ if (message_size <= 4096) { segsize = 0; fanout = communicator_size - 1; /* when linear implemented or taken from basic put here, right now using chain as a linear system */ /* it is implemented and I shouldn't be calling a chain with a fanout bigger than MAXTREEFANOUT from topo.h! */ return smpi_coll_tuned_reduce_intra_basic_linear (sendbuf, recvbuf, count, datatype, op, root, comm, module); /* return smpi_coll_tuned_reduce_intra_chain (sendbuf, recvbuf, count, datatype, op, root, comm, segsize, fanout); */ } if (message_size < 524288) { if (message_size <= 65536 ) { segsize = 32768; fanout = 8; } else { segsize = 1024; fanout = communicator_size/2; } /* later swap this for a binary tree */ /* fanout = 2; */ return smpi_coll_tuned_reduce_intra_chain (sendbuf, recvbuf, count, datatype, op, root, comm, module, segsize, fanout, max_requests); } segsize = 1024; return smpi_coll_tuned_reduce_intra_pipeline (sendbuf, recvbuf, count, datatype, op, root, comm, module, segsize, max_requests); #endif /* 0 */ } int smpi_coll_tuned_reduce_scatter_ompi( void *sbuf, void *rbuf, int *rcounts, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm ) { int comm_size, i, pow2; size_t total_message_size, dsize; const double a = 0.0012; const double b = 8.0; const size_t small_message_size = 12 * 1024; const size_t large_message_size = 256 * 1024; int zerocounts = 0; XBT_DEBUG("smpi_coll_tuned_reduce_scatter_ompi"); comm_size = smpi_comm_size(comm); // We need data size for decision function dsize=smpi_datatype_size(dtype); total_message_size = 0; for (i = 0; i < comm_size; i++) { total_message_size += rcounts[i]; if (0 == rcounts[i]) { zerocounts = 1; } } if( !smpi_op_is_commute(op) || (zerocounts)) { smpi_mpi_reduce_scatter (sbuf, rbuf, rcounts, dtype, op, comm); return MPI_SUCCESS; } total_message_size *= dsize; // compute the nearest power of 2 for (pow2 = 1; pow2 < comm_size; pow2 <<= 1); if ((total_message_size <= small_message_size) || ((total_message_size <= large_message_size) && (pow2 == comm_size)) || (comm_size >= a * total_message_size + b)) { return smpi_coll_tuned_reduce_scatter_ompi_basic_recursivehalving(sbuf, rbuf, rcounts, dtype, op, comm); } return smpi_coll_tuned_reduce_scatter_ompi_ring(sbuf, rbuf, rcounts, dtype, op, comm); } int smpi_coll_tuned_allgather_ompi(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, MPI_Comm comm ) { int communicator_size, pow2_size; size_t dsize, total_dsize; communicator_size = smpi_comm_size(comm); /* Special case for 2 processes */ if (communicator_size == 2) { return smpi_coll_tuned_allgather_pair (sbuf, scount, sdtype, rbuf, rcount, rdtype, comm/*, module*/); } /* Determine complete data size */ dsize=smpi_datatype_size(sdtype); total_dsize = dsize * scount * communicator_size; for (pow2_size = 1; pow2_size < communicator_size; pow2_size <<=1); /* Decision based on MX 2Gb results from Grig cluster at The University of Tennesse, Knoxville - if total message size is less than 50KB use either bruck or recursive doubling for non-power of two and power of two nodes, respectively. - else use ring and neighbor exchange algorithms for odd and even number of nodes, respectively. */ if (total_dsize < 50000) { if (pow2_size == communicator_size) { return smpi_coll_tuned_allgather_rdb(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } else { return smpi_coll_tuned_allgather_bruck(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } } else { if (communicator_size % 2) { return smpi_coll_tuned_allgather_ring(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } else { return smpi_coll_tuned_allgather_ompi_neighborexchange(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } } #if defined(USE_MPICH2_DECISION) /* Decision as in MPICH-2 presented in Thakur et.al. "Optimization of Collective Communication Operations in MPICH", International Journal of High Performance Computing Applications, Vol. 19, No. 1, 49-66 (2005) - for power-of-two processes and small and medium size messages (up to 512KB) use recursive doubling - for non-power-of-two processes and small messages (80KB) use bruck, - for everything else use ring. */ if ((pow2_size == communicator_size) && (total_dsize < 524288)) { return smpi_coll_tuned_allgather_rdb(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } else if (total_dsize <= 81920) { return smpi_coll_tuned_allgather_bruck(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } return smpi_coll_tuned_allgather_ring(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); #endif /* defined(USE_MPICH2_DECISION) */ } int smpi_coll_tuned_allgatherv_ompi(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int *rcounts, int *rdispls, MPI_Datatype rdtype, MPI_Comm comm ) { int i; int communicator_size; size_t dsize, total_dsize; communicator_size = smpi_comm_size(comm); /* Special case for 2 processes */ if (communicator_size == 2) { return smpi_coll_tuned_allgatherv_pair(sbuf, scount, sdtype, rbuf, rcounts, rdispls, rdtype, comm); } /* Determine complete data size */ dsize=smpi_datatype_size(sdtype); total_dsize = 0; for (i = 0; i < communicator_size; i++) { total_dsize += dsize * rcounts[i]; } /* Decision based on allgather decision. */ if (total_dsize < 50000) { /* return smpi_coll_tuned_allgatherv_intra_bruck(sbuf, scount, sdtype, rbuf, rcounts, rdispls, rdtype, comm, module);*/ return smpi_coll_tuned_allgatherv_ring(sbuf, scount, sdtype, rbuf, rcounts, rdispls, rdtype, comm); } else { if (communicator_size % 2) { return smpi_coll_tuned_allgatherv_ring(sbuf, scount, sdtype, rbuf, rcounts, rdispls, rdtype, comm); } else { return smpi_coll_tuned_allgatherv_ompi_neighborexchange(sbuf, scount, sdtype, rbuf, rcounts, rdispls, rdtype, comm); } } } int smpi_coll_tuned_gather_ompi(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, int root, MPI_Comm comm ) { //const int large_segment_size = 32768; //const int small_segment_size = 1024; //const size_t large_block_size = 92160; const size_t intermediate_block_size = 6000; const size_t small_block_size = 1024; const int large_communicator_size = 60; const int small_communicator_size = 10; int communicator_size, rank; size_t dsize, block_size; XBT_DEBUG("smpi_coll_tuned_gather_ompi"); communicator_size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); // Determine block size if (rank == root) { dsize = smpi_datatype_size(rdtype); block_size = dsize * rcount; } else { dsize = smpi_datatype_size(sdtype); block_size = dsize * scount; } /* if (block_size > large_block_size) {*/ /* return smpi_coll_tuned_gather_ompi_linear_sync (sbuf, scount, sdtype, */ /* rbuf, rcount, rdtype, */ /* root, comm);*/ /* } else*/ if (block_size > intermediate_block_size) { return smpi_coll_tuned_gather_ompi_linear_sync (sbuf, scount, sdtype, rbuf, rcount, rdtype, root, comm); } else if ((communicator_size > large_communicator_size) || ((communicator_size > small_communicator_size) && (block_size < small_block_size))) { return smpi_coll_tuned_gather_ompi_binomial (sbuf, scount, sdtype, rbuf, rcount, rdtype, root, comm); } // Otherwise, use basic linear return smpi_coll_tuned_gather_ompi_basic_linear (sbuf, scount, sdtype, rbuf, rcount, rdtype, root, comm); } int smpi_coll_tuned_scatter_ompi(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, int root, MPI_Comm comm ) { const size_t small_block_size = 300; const int small_comm_size = 10; int communicator_size, rank; size_t dsize, block_size; XBT_DEBUG("smpi_coll_tuned_scatter_ompi"); communicator_size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); // Determine block size if (root == rank) { dsize=smpi_datatype_size(sdtype); block_size = dsize * scount; } else { dsize=smpi_datatype_size(rdtype); block_size = dsize * rcount; } if ((communicator_size > small_comm_size) && (block_size < small_block_size)) { if(rank!=root){ sbuf=xbt_malloc(rcount*smpi_datatype_get_extent(rdtype)); scount=rcount; sdtype=rdtype; } int ret=smpi_coll_tuned_scatter_ompi_binomial (sbuf, scount, sdtype, rbuf, rcount, rdtype, root, comm); if(rank!=root){ xbt_free(sbuf); } return ret; } return smpi_coll_tuned_scatter_ompi_basic_linear (sbuf, scount, sdtype, rbuf, rcount, rdtype, root, comm); } SimGrid-3.14.159/src/smpi/colls/reduce-binomial.c000644 001750 001750 00000005354 13031031272 023155 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" //#include int smpi_coll_tuned_reduce_binomial(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { MPI_Status status; int comm_size, rank; int mask, relrank, source; int dst; int tag = COLL_TAG_REDUCE; MPI_Aint extent; void *tmp_buf; MPI_Aint true_lb, true_extent; if (count == 0) return 0; rank = smpi_comm_rank(comm); comm_size = smpi_comm_size(comm); extent = smpi_datatype_get_extent(datatype); tmp_buf = (void *) smpi_get_tmp_sendbuffer(count * extent); int is_commutative = smpi_op_is_commute(op); mask = 1; int lroot; if (is_commutative) lroot = root; else lroot = 0; relrank = (rank - lroot + comm_size) % comm_size; smpi_datatype_extent(datatype, &true_lb, &true_extent); /* adjust for potential negative lower bound in datatype */ tmp_buf = (void *)((char*)tmp_buf - true_lb); /* If I'm not the root, then my recvbuf may not be valid, therefore I have to allocate a temporary one */ if (rank != root) { recvbuf = (void *) smpi_get_tmp_recvbuffer(count*(MAX(extent,true_extent))); recvbuf = (void *)((char*)recvbuf - true_lb); } if ((rank != root) || (sendbuf != MPI_IN_PLACE)) { smpi_datatype_copy(sendbuf, count, datatype, recvbuf,count, datatype); } while (mask < comm_size) { /* Receive */ if ((mask & relrank) == 0) { source = (relrank | mask); if (source < comm_size) { source = (source + lroot) % comm_size; smpi_mpi_recv(tmp_buf, count, datatype, source, tag, comm, &status); if (is_commutative) { smpi_op_apply(op, tmp_buf, recvbuf, &count, &datatype); } else { smpi_op_apply(op, recvbuf, tmp_buf, &count, &datatype); smpi_datatype_copy(tmp_buf, count, datatype,recvbuf, count, datatype); } } } else { dst = ((relrank & (~mask)) + lroot) % comm_size; smpi_mpi_send(recvbuf, count, datatype, dst, tag, comm); break; } mask <<= 1; } if (!is_commutative && (root != 0)){ if (rank == 0){ smpi_mpi_send(recvbuf, count, datatype, root,tag, comm); }else if (rank == root){ smpi_mpi_recv(recvbuf, count, datatype, 0, tag, comm, &status); } } if (rank != root) { smpi_free_tmp_buffer(recvbuf); } smpi_free_tmp_buffer(tmp_buf); return 0; } SimGrid-3.14.159/src/smpi/colls/allgather-GB.c000644 001750 001750 00000001560 13031031271 022341 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" // Allgather - gather/bcast algorithm int smpi_coll_tuned_allgather_GB(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { int num_procs; num_procs = smpi_comm_size(comm); mpi_coll_gather_fun(send_buff, send_count, send_type, recv_buff, recv_count, recv_type, 0, comm); mpi_coll_bcast_fun(recv_buff, (recv_count * num_procs), recv_type, 0, comm); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allgather-rhv.c000644 001750 001750 00000006107 13031031271 022652 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" // now only work with power of two processes int smpi_coll_tuned_allgather_rhv(void *sbuf, int send_count, MPI_Datatype send_type, void *rbuf, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Status status; MPI_Aint s_extent, r_extent; // local int variables int i, dst, send_base_offset, recv_base_offset, send_chunk, recv_chunk, send_offset, recv_offset; int tag = COLL_TAG_ALLGATHER; int mask; int curr_count; // get size of the communicator, followed by rank unsigned int num_procs = smpi_comm_size(comm); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "allgather rhv algorithm can't be used with non power of two number of processes ! "); unsigned int rank = smpi_comm_rank(comm); // get size of single element's type for send buffer and recv buffer s_extent = smpi_datatype_get_extent(send_type); r_extent = smpi_datatype_get_extent(recv_type); // multiply size of each element by number of elements to send or recv send_chunk = s_extent * send_count; recv_chunk = r_extent * recv_count; if (send_chunk != recv_chunk) { XBT_WARN("MPI_allgather_rhv use default MPI_allgather."); smpi_mpi_allgather(sbuf, send_count, send_type, rbuf, recv_count, recv_type, comm); return MPI_SUCCESS; } // compute starting offset location to perform local copy int size = num_procs / 2; int base_offset = 0; mask = 1; while (mask < num_procs) { if (rank & mask) { base_offset += size; } mask <<= 1; size /= 2; } // printf("node %d base_offset %d\n",rank,base_offset); //perform a remote copy dst = base_offset; smpi_mpi_sendrecv(sbuf, send_count, send_type, dst, tag, (char *)rbuf + base_offset * recv_chunk, recv_count, recv_type, dst, tag, comm, &status); mask >>= 1; i = 1; int phase = 0; curr_count = recv_count; while (mask >= 1) { // destination pair for both send and recv dst = rank ^ mask; // compute offsets send_base_offset = base_offset; if (rank & mask) { recv_base_offset = base_offset - i; base_offset -= i; } else { recv_base_offset = base_offset + i; } send_offset = send_base_offset * recv_chunk; recv_offset = recv_base_offset * recv_chunk; // printf("node %d send to %d in phase %d s_offset = %d r_offset = %d count = %d\n",rank,dst,phase, send_base_offset, recv_base_offset, curr_count); smpi_mpi_sendrecv((char *)rbuf + send_offset, curr_count, recv_type, dst, tag, (char *)rbuf + recv_offset, curr_count, recv_type, dst, tag, comm, &status); curr_count *= 2; i *= 2; mask >>= 1; phase++; } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/bcast-flattree-pipeline.c000644 001750 001750 00000003604 13031031272 024615 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" int flattree_segment_in_byte = 8192; int smpi_coll_tuned_bcast_flattree_pipeline(void *buff, int count, MPI_Datatype data_type, int root, MPI_Comm comm) { int i, j, rank, num_procs; int tag = COLL_TAG_BCAST; MPI_Aint extent; extent = smpi_datatype_get_extent(data_type); int segment = flattree_segment_in_byte / extent; segment = segment == 0 ? 1 :segment; int pipe_length = count / segment; int increment = segment * extent; if (pipe_length==0) { XBT_WARN("MPI_bcast_flattree_pipeline use default MPI_bcast_flattree."); return smpi_coll_tuned_bcast_flattree(buff, count, data_type, root, comm); } rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); MPI_Request *request_array; MPI_Status *status_array; request_array = (MPI_Request *) xbt_malloc(pipe_length * sizeof(MPI_Request)); status_array = (MPI_Status *) xbt_malloc(pipe_length * sizeof(MPI_Status)); if (rank != root) { for (i = 0; i < pipe_length; i++) { request_array[i] = smpi_mpi_irecv((char *)buff + (i * increment), segment, data_type, root, tag, comm); } smpi_mpi_waitall(pipe_length, request_array, status_array); } else { // Root sends data to all others for (j = 0; j < num_procs; j++) { if (j == rank) continue; else { for (i = 0; i < pipe_length; i++) { smpi_mpi_send((char *)buff + (i * increment), segment, data_type, j, tag, comm); } } } } free(request_array); free(status_array); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/bcast-ompi-pipeline.c000644 001750 001750 00000017503 13031031272 023756 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" #include "coll_tuned_topo.h" #define MAXTREEFANOUT 32 int smpi_coll_tuned_bcast_ompi_pipeline( void* buffer, int original_count, MPI_Datatype datatype, int root, MPI_Comm comm) { int count_by_segment = original_count; size_t type_size; int segsize =1024 << 7; //mca_coll_tuned_module_t *tuned_module = (mca_coll_tuned_module_t*) module; //mca_coll_tuned_comm_t *data = tuned_module->tuned_data; // return ompi_coll_tuned_bcast_intra_generic( buffer, count, datatype, root, comm, module, // count_by_segment, data->cached_pipeline ); ompi_coll_tree_t * tree = ompi_coll_tuned_topo_build_chain( 1, comm, root ); int i; int rank, size; int segindex; int num_segments; /* Number of segments */ int sendcount; /* number of elements sent in this segment */ size_t realsegsize; char *tmpbuf; ptrdiff_t extent; MPI_Request recv_reqs[2] = {MPI_REQUEST_NULL, MPI_REQUEST_NULL}; MPI_Request *send_reqs = NULL; int req_index; /** * Determine number of elements sent per operation. */ type_size = smpi_datatype_size(datatype); size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); if(size==1)return MPI_SUCCESS; const double a_p16 = 3.2118e-6; /* [1 / byte] */ const double b_p16 = 8.7936; const double a_p64 = 2.3679e-6; /* [1 / byte] */ const double b_p64 = 1.1787; const double a_p128 = 1.6134e-6; /* [1 / byte] */ const double b_p128 = 2.1102; size_t message_size; /* else we need data size for decision function */ message_size = type_size * (unsigned long)original_count; /* needed for decision */ if (size < (a_p128 * message_size + b_p128)) { //Pipeline with 128KB segments segsize = 1024 << 7; }else if (size < (a_p64 * message_size + b_p64)) { // Pipeline with 64KB segments segsize = 1024 << 6; }else if (size < (a_p16 * message_size + b_p16)) { //Pipeline with 16KB segments segsize = 1024 << 4; } COLL_TUNED_COMPUTED_SEGCOUNT( segsize, type_size, count_by_segment ); XBT_DEBUG("coll:tuned:bcast_intra_pipeline rank %d ss %5d type_size %lu count_by_segment %d", smpi_comm_rank(comm), segsize, (unsigned long)type_size, count_by_segment); extent = smpi_datatype_get_extent (datatype); num_segments = (original_count + count_by_segment - 1) / count_by_segment; realsegsize = count_by_segment * extent; /* Set the buffer pointers */ tmpbuf = (char *) buffer; if( tree->tree_nextsize != 0 ) { send_reqs = xbt_new(MPI_Request, tree->tree_nextsize ); } /* Root code */ if( rank == root ) { /* For each segment: - send segment to all children. The last segment may have less elements than other segments. */ sendcount = count_by_segment; for( segindex = 0; segindex < num_segments; segindex++ ) { if( segindex == (num_segments - 1) ) { sendcount = original_count - segindex * count_by_segment; } for( i = 0; i < tree->tree_nextsize; i++ ) { send_reqs[i] = smpi_mpi_isend(tmpbuf, sendcount, datatype, tree->tree_next[i], COLL_TAG_BCAST, comm); } /* complete the sends before starting the next sends */ smpi_mpi_waitall( tree->tree_nextsize, send_reqs, MPI_STATUSES_IGNORE ); /* update tmp buffer */ tmpbuf += realsegsize; } } /* Intermediate nodes code */ else if( tree->tree_nextsize > 0 ) { /* Create the pipeline. 1) Post the first receive 2) For segments 1 .. num_segments - post new receive - wait on the previous receive to complete - send this data to children 3) Wait on the last segment 4) Compute number of elements in last segment. 5) Send the last segment to children */ req_index = 0; recv_reqs[req_index]=smpi_mpi_irecv(tmpbuf, count_by_segment, datatype, tree->tree_prev, COLL_TAG_BCAST, comm); for( segindex = 1; segindex < num_segments; segindex++ ) { req_index = req_index ^ 0x1; /* post new irecv */ recv_reqs[req_index]= smpi_mpi_irecv( tmpbuf + realsegsize, count_by_segment, datatype, tree->tree_prev, COLL_TAG_BCAST, comm); /* wait for and forward the previous segment to children */ smpi_mpi_wait( &recv_reqs[req_index ^ 0x1], MPI_STATUSES_IGNORE ); for( i = 0; i < tree->tree_nextsize; i++ ) { send_reqs[i]=smpi_mpi_isend(tmpbuf, count_by_segment, datatype, tree->tree_next[i], COLL_TAG_BCAST, comm ); } /* complete the sends before starting the next iteration */ smpi_mpi_waitall( tree->tree_nextsize, send_reqs, MPI_STATUSES_IGNORE ); /* Update the receive buffer */ tmpbuf += realsegsize; } /* Process the last segment */ smpi_mpi_wait( &recv_reqs[req_index], MPI_STATUSES_IGNORE ); sendcount = original_count - (num_segments - 1) * count_by_segment; for( i = 0; i < tree->tree_nextsize; i++ ) { send_reqs[i] = smpi_mpi_isend(tmpbuf, sendcount, datatype, tree->tree_next[i], COLL_TAG_BCAST, comm); } smpi_mpi_waitall( tree->tree_nextsize, send_reqs, MPI_STATUSES_IGNORE ); } /* Leaf nodes */ else { /* Receive all segments from parent in a loop: 1) post irecv for the first segment 2) for segments 1 .. num_segments - post irecv for the next segment - wait on the previous segment to arrive 3) wait for the last segment */ req_index = 0; recv_reqs[req_index] = smpi_mpi_irecv(tmpbuf, count_by_segment, datatype, tree->tree_prev, COLL_TAG_BCAST, comm); for( segindex = 1; segindex < num_segments; segindex++ ) { req_index = req_index ^ 0x1; tmpbuf += realsegsize; /* post receive for the next segment */ recv_reqs[req_index] = smpi_mpi_irecv(tmpbuf, count_by_segment, datatype, tree->tree_prev, COLL_TAG_BCAST, comm); /* wait on the previous segment */ smpi_mpi_wait( &recv_reqs[req_index ^ 0x1], MPI_STATUS_IGNORE ); } smpi_mpi_wait( &recv_reqs[req_index], MPI_STATUS_IGNORE ); } if( NULL != send_reqs ) free(send_reqs); xbt_free(tree); return (MPI_SUCCESS); } SimGrid-3.14.159/src/smpi/colls/reduce-flat-tree.c000644 001750 001750 00000003351 13031031272 023241 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" //#include int smpi_coll_tuned_reduce_flat_tree(void *sbuf, void *rbuf, int count, MPI_Datatype dtype, MPI_Op op, int root, MPI_Comm comm) { int i, tag = COLL_TAG_REDUCE; int size; int rank; MPI_Aint extent; char *origin = 0; char *inbuf; MPI_Status status; rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); /* If not root, send data to the root. */ extent = smpi_datatype_get_extent(dtype); if (rank != root) { smpi_mpi_send(sbuf, count, dtype, root, tag, comm); return 0; } /* Root receives and reduces messages. Allocate buffer to receive messages. */ if (size > 1) origin = (char *) smpi_get_tmp_recvbuffer(count * extent); /* Initialize the receive buffer. */ if (rank == (size - 1)) smpi_mpi_sendrecv(sbuf, count, dtype, rank, tag, rbuf, count, dtype, rank, tag, comm, &status); else smpi_mpi_recv(rbuf, count, dtype, size - 1, tag, comm, &status); /* Loop receiving and calling reduction function (C or Fortran). */ for (i = size - 2; i >= 0; --i) { if (rank == i) inbuf = sbuf; else { smpi_mpi_recv(origin, count, dtype, i, tag, comm, &status); inbuf = origin; } /* Call reduction function. */ smpi_op_apply(op, inbuf, rbuf, &count, &dtype); } if (origin) smpi_free_tmp_buffer(origin); /* All done */ return 0; } SimGrid-3.14.159/src/smpi/colls/reduce-ompi.c000644 001750 001750 00000065352 13031031272 022333 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2009 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * * Additional copyrights may follow */ #include "colls_private.h" #include "coll_tuned_topo.h" int smpi_coll_tuned_ompi_reduce_generic( void* sendbuf, void* recvbuf, int original_count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm, ompi_coll_tree_t* tree, int count_by_segment, int max_outstanding_reqs ); /** * This is a generic implementation of the reduce protocol. It used the tree * provided as an argument and execute all operations using a segment of * count times a datatype. * For the last communication it will update the count in order to limit * the number of datatype to the original count (original_count) * * Note that for non-commutative operations we cannot save memory copy * for the first block: thus we must copy sendbuf to accumbuf on intermediate * to keep the optimized loop happy. */ int smpi_coll_tuned_ompi_reduce_generic( void* sendbuf, void* recvbuf, int original_count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm, ompi_coll_tree_t* tree, int count_by_segment, int max_outstanding_reqs ) { char *inbuf[2] = {NULL, NULL}, *inbuf_free[2] = {NULL, NULL}; char *accumbuf = NULL, *accumbuf_free = NULL; char *local_op_buffer = NULL, *sendtmpbuf = NULL; ptrdiff_t extent, lower_bound, segment_increment; MPI_Request reqs[2] = {MPI_REQUEST_NULL, MPI_REQUEST_NULL}; int num_segments, line, ret, segindex, i, rank; int recvcount, prevcount, inbi; /** * Determine number of segments and number of elements * sent per operation */ smpi_datatype_extent( datatype, &lower_bound, &extent); num_segments = (original_count + count_by_segment - 1) / count_by_segment; segment_increment = count_by_segment * extent; sendtmpbuf = (char*) sendbuf; if( sendbuf == MPI_IN_PLACE ) { sendtmpbuf = (char *)recvbuf; } XBT_DEBUG( "coll:tuned:reduce_generic count %d, msg size %ld, segsize %ld, max_requests %d", original_count, (unsigned long)(num_segments * segment_increment), (unsigned long)segment_increment, max_outstanding_reqs); rank = smpi_comm_rank(comm); /* non-leaf nodes - wait for children to send me data & forward up (if needed) */ if( tree->tree_nextsize > 0 ) { ptrdiff_t true_extent, real_segment_size; true_extent=smpi_datatype_get_extent( datatype); /* handle non existant recv buffer (i.e. its NULL) and protect the recv buffer on non-root nodes */ accumbuf = (char*)recvbuf; if( (NULL == accumbuf) || (root != rank) ) { /* Allocate temporary accumulator buffer. */ accumbuf_free = (char*)smpi_get_tmp_sendbuffer(true_extent + (original_count - 1) * extent); if (accumbuf_free == NULL) { line = __LINE__; ret = -1; goto error_hndl; } accumbuf = accumbuf_free - lower_bound; } /* If this is a non-commutative operation we must copy sendbuf to the accumbuf, in order to simplfy the loops */ if (!smpi_op_is_commute(op)) { smpi_datatype_copy( (char*)sendtmpbuf, original_count, datatype, (char*)accumbuf, original_count, datatype); } /* Allocate two buffers for incoming segments */ real_segment_size = true_extent + (count_by_segment - 1) * extent; inbuf_free[0] = (char*) smpi_get_tmp_recvbuffer(real_segment_size); if( inbuf_free[0] == NULL ) { line = __LINE__; ret = -1; goto error_hndl; } inbuf[0] = inbuf_free[0] - lower_bound; /* if there is chance to overlap communication - allocate second buffer */ if( (num_segments > 1) || (tree->tree_nextsize > 1) ) { inbuf_free[1] = (char*) smpi_get_tmp_recvbuffer(real_segment_size); if( inbuf_free[1] == NULL ) { line = __LINE__; ret = -1; goto error_hndl; } inbuf[1] = inbuf_free[1] - lower_bound; } /* reset input buffer index and receive count */ inbi = 0; recvcount = 0; /* for each segment */ for( segindex = 0; segindex <= num_segments; segindex++ ) { prevcount = recvcount; /* recvcount - number of elements in current segment */ recvcount = count_by_segment; if( segindex == (num_segments-1) ) recvcount = original_count - count_by_segment * segindex; /* for each child */ for( i = 0; i < tree->tree_nextsize; i++ ) { /** * We try to overlap communication: * either with next segment or with the next child */ /* post irecv for current segindex on current child */ if( segindex < num_segments ) { void* local_recvbuf = inbuf[inbi]; if( 0 == i ) { /* for the first step (1st child per segment) and * commutative operations we might be able to irecv * directly into the accumulate buffer so that we can * reduce(op) this with our sendbuf in one step as * ompi_op_reduce only has two buffer pointers, * this avoids an extra memory copy. * * BUT if the operation is non-commutative or * we are root and are USING MPI_IN_PLACE this is wrong! */ if( (smpi_op_is_commute(op)) && !((MPI_IN_PLACE == sendbuf) && (rank == tree->tree_root)) ) { local_recvbuf = accumbuf + segindex * segment_increment; } } reqs[inbi]=smpi_mpi_irecv(local_recvbuf, recvcount, datatype, tree->tree_next[i], COLL_TAG_REDUCE, comm ); } /* wait for previous req to complete, if any. if there are no requests reqs[inbi ^1] will be MPI_REQUEST_NULL. */ /* wait on data from last child for previous segment */ smpi_mpi_waitall( 1, &reqs[inbi ^ 1], MPI_STATUSES_IGNORE ); local_op_buffer = inbuf[inbi ^ 1]; if( i > 0 ) { /* our first operation is to combine our own [sendbuf] data * with the data we recvd from down stream (but only * the operation is commutative and if we are not root and * not using MPI_IN_PLACE) */ if( 1 == i ) { if( (smpi_op_is_commute(op)) && !((MPI_IN_PLACE == sendbuf) && (rank == tree->tree_root)) ) { local_op_buffer = sendtmpbuf + segindex * segment_increment; } } /* apply operation */ smpi_op_apply(op, local_op_buffer, accumbuf + segindex * segment_increment, &recvcount, &datatype ); } else if ( segindex > 0 ) { void* accumulator = accumbuf + (segindex-1) * segment_increment; if( tree->tree_nextsize <= 1 ) { if( (smpi_op_is_commute(op)) && !((MPI_IN_PLACE == sendbuf) && (rank == tree->tree_root)) ) { local_op_buffer = sendtmpbuf + (segindex-1) * segment_increment; } } smpi_op_apply(op, local_op_buffer, accumulator, &prevcount, &datatype ); /* all reduced on available data this step (i) complete, * pass to the next process unless you are the root. */ if (rank != tree->tree_root) { /* send combined/accumulated data to parent */ smpi_mpi_send( accumulator, prevcount, datatype, tree->tree_prev, COLL_TAG_REDUCE, comm); } /* we stop when segindex = number of segments (i.e. we do num_segment+1 steps for pipelining */ if (segindex == num_segments) break; } /* update input buffer index */ inbi = inbi ^ 1; } /* end of for each child */ } /* end of for each segment */ /* clean up */ smpi_free_tmp_buffer(inbuf_free[0]); smpi_free_tmp_buffer(inbuf_free[1]); smpi_free_tmp_buffer(accumbuf_free); } /* leaf nodes Depending on the value of max_outstanding_reqs and the number of segments we have two options: - send all segments using blocking send to the parent, or - avoid overflooding the parent nodes by limiting the number of outstanding requests to max_oustanding_reqs. TODO/POSSIBLE IMPROVEMENT: If there is a way to determine the eager size for the current communication, synchronization should be used only when the message/segment size is smaller than the eager size. */ else { /* If the number of segments is less than a maximum number of oustanding requests or there is no limit on the maximum number of outstanding requests, we send data to the parent using blocking send */ if ((0 == max_outstanding_reqs) || (num_segments <= max_outstanding_reqs)) { segindex = 0; while ( original_count > 0) { if (original_count < count_by_segment) { count_by_segment = original_count; } smpi_mpi_send((char*)sendbuf + segindex * segment_increment, count_by_segment, datatype, tree->tree_prev, COLL_TAG_REDUCE, comm) ; segindex++; original_count -= count_by_segment; } } /* Otherwise, introduce flow control: - post max_outstanding_reqs non-blocking synchronous send, - for remaining segments - wait for a ssend to complete, and post the next one. - wait for all outstanding sends to complete. */ else { int creq = 0; MPI_Request* sreq = NULL; sreq = (MPI_Request*) calloc( max_outstanding_reqs, sizeof(MPI_Request ) ); if (NULL == sreq) { line = __LINE__; ret = -1; goto error_hndl; } /* post first group of requests */ for (segindex = 0; segindex < max_outstanding_reqs; segindex++) { sreq[segindex]=smpi_mpi_isend((char*)sendbuf + segindex * segment_increment, count_by_segment, datatype, tree->tree_prev, COLL_TAG_REDUCE, comm); original_count -= count_by_segment; } creq = 0; while ( original_count > 0 ) { /* wait on a posted request to complete */ smpi_mpi_wait(&sreq[creq], MPI_STATUS_IGNORE); sreq[creq] = MPI_REQUEST_NULL; if( original_count < count_by_segment ) { count_by_segment = original_count; } sreq[creq]=smpi_mpi_isend((char*)sendbuf + segindex * segment_increment, count_by_segment, datatype, tree->tree_prev, COLL_TAG_REDUCE, comm ); creq = (creq + 1) % max_outstanding_reqs; segindex++; original_count -= count_by_segment; } /* Wait on the remaining request to complete */ smpi_mpi_waitall( max_outstanding_reqs, sreq, MPI_STATUSES_IGNORE ); /* free requests */ free(sreq); } } free(tree); return MPI_SUCCESS; error_hndl: /* error handler */ XBT_DEBUG("ERROR_HNDL: node %d file %s line %d error %d\n", rank, __FILE__, line, ret ); if( inbuf_free[0] != NULL ) free(inbuf_free[0]); if( inbuf_free[1] != NULL ) free(inbuf_free[1]); if( accumbuf_free != NULL ) free(accumbuf); return ret; } /* Attention: this version of the reduce operations does not work for: - non-commutative operations - segment sizes which are not multiplies of the extent of the datatype meaning that at least one datatype must fit in the segment ! */ int smpi_coll_tuned_reduce_ompi_chain( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm ) { uint32_t segsize=64*1024; int segcount = count; size_t typelng; int fanout = smpi_comm_size(comm)/2; XBT_DEBUG("coll:tuned:reduce_intra_chain rank %d fo %d ss %5d", smpi_comm_rank(comm), fanout, segsize); /** * Determine number of segments and number of elements * sent per operation */ typelng = smpi_datatype_size( datatype); COLL_TUNED_COMPUTED_SEGCOUNT( segsize, typelng, segcount ); return smpi_coll_tuned_ompi_reduce_generic( sendbuf, recvbuf, count, datatype, op, root, comm, ompi_coll_tuned_topo_build_chain(fanout, comm, root), segcount, 0 ); } int smpi_coll_tuned_reduce_ompi_pipeline( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm ) { uint32_t segsize; int segcount = count; size_t typelng; // COLL_TUNED_UPDATE_PIPELINE( comm, tuned_module, root ); /** * Determine number of segments and number of elements * sent per operation */ const double a2 = 0.0410 / 1024.0; /* [1/B] */ const double b2 = 9.7128; const double a4 = 0.0033 / 1024.0; /* [1/B] */ const double b4 = 1.6761; typelng= smpi_datatype_size( datatype); int communicator_size = smpi_comm_size(comm); size_t message_size = typelng * count; if (communicator_size > (a2 * message_size + b2)) { // Pipeline_1K segsize = 1024; }else if (communicator_size > (a4 * message_size + b4)) { // Pipeline_32K segsize = 32*1024; } else { // Pipeline_64K segsize = 64*1024; } XBT_DEBUG("coll:tuned:reduce_intra_pipeline rank %d ss %5d", smpi_comm_rank(comm), segsize); COLL_TUNED_COMPUTED_SEGCOUNT( segsize, typelng, segcount ); return smpi_coll_tuned_ompi_reduce_generic( sendbuf, recvbuf, count, datatype, op, root, comm, ompi_coll_tuned_topo_build_chain( 1, comm, root), segcount, 0); } int smpi_coll_tuned_reduce_ompi_binary( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { uint32_t segsize; int segcount = count; size_t typelng; /** * Determine number of segments and number of elements * sent per operation */ typelng=smpi_datatype_size( datatype ); // Binary_32K segsize = 32*1024; XBT_DEBUG("coll:tuned:reduce_intra_binary rank %d ss %5d", smpi_comm_rank(comm), segsize); COLL_TUNED_COMPUTED_SEGCOUNT( segsize, typelng, segcount ); return smpi_coll_tuned_ompi_reduce_generic( sendbuf, recvbuf, count, datatype, op, root, comm, ompi_coll_tuned_topo_build_tree(2, comm, root), segcount, 0); } int smpi_coll_tuned_reduce_ompi_binomial( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { uint32_t segsize=0; int segcount = count; size_t typelng; const double a1 = 0.6016 / 1024.0; /* [1/B] */ const double b1 = 1.3496; // COLL_TUNED_UPDATE_IN_ORDER_BMTREE( comm, tuned_module, root ); /** * Determine number of segments and number of elements * sent per operation */ typelng= smpi_datatype_size( datatype); int communicator_size = smpi_comm_size(comm); size_t message_size = typelng * count; if (((communicator_size < 8) && (message_size < 20480)) || (message_size < 2048) || (count <= 1)) { /* Binomial_0K */ segsize = 0; } else if (communicator_size > (a1 * message_size + b1)) { // Binomial_1K segsize = 1024; } XBT_DEBUG("coll:tuned:reduce_intra_binomial rank %d ss %5d", smpi_comm_rank(comm), segsize); COLL_TUNED_COMPUTED_SEGCOUNT( segsize, typelng, segcount ); return smpi_coll_tuned_ompi_reduce_generic( sendbuf, recvbuf, count, datatype, op, root, comm, ompi_coll_tuned_topo_build_in_order_bmtree(comm, root), segcount, 0); } /* * reduce_intra_in_order_binary * * Function: Logarithmic reduce operation for non-commutative operations. * Acecpts: same as MPI_Reduce() * Returns: MPI_SUCCESS or error code */ int smpi_coll_tuned_reduce_ompi_in_order_binary( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { uint32_t segsize=0; int ret; int rank, size, io_root; int segcount = count; void *use_this_sendbuf = NULL, *use_this_recvbuf = NULL; size_t typelng; rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); XBT_DEBUG("coll:tuned:reduce_intra_in_order_binary rank %d ss %5d", rank, segsize); /** * Determine number of segments and number of elements * sent per operation */ typelng=smpi_datatype_size( datatype); COLL_TUNED_COMPUTED_SEGCOUNT( segsize, typelng, segcount ); /* An in-order binary tree must use root (size-1) to preserve the order of operations. Thus, if root is not rank (size - 1), then we must handle 1. MPI_IN_PLACE option on real root, and 2. we must allocate temporary recvbuf on rank (size - 1). Note that generic function must be careful not to switch order of operations for non-commutative ops. */ io_root = size - 1; use_this_sendbuf = sendbuf; use_this_recvbuf = recvbuf; if (io_root != root) { ptrdiff_t text, ext; char *tmpbuf = NULL; ext=smpi_datatype_get_extent(datatype); text=smpi_datatype_get_extent(datatype); if ((root == rank) && (MPI_IN_PLACE == sendbuf)) { tmpbuf = (char *) smpi_get_tmp_sendbuffer(text + (count - 1) * ext); if (NULL == tmpbuf) { return MPI_ERR_INTERN; } smpi_datatype_copy ( (char*)recvbuf, count, datatype, (char*)tmpbuf, count, datatype); use_this_sendbuf = tmpbuf; } else if (io_root == rank) { tmpbuf = (char *) smpi_get_tmp_recvbuffer(text + (count - 1) * ext); if (NULL == tmpbuf) { return MPI_ERR_INTERN; } use_this_recvbuf = tmpbuf; } } /* Use generic reduce with in-order binary tree topology and io_root */ ret = smpi_coll_tuned_ompi_reduce_generic( use_this_sendbuf, use_this_recvbuf, count, datatype, op, io_root, comm, ompi_coll_tuned_topo_build_in_order_bintree(comm), segcount, 0 ); if (MPI_SUCCESS != ret) { return ret; } /* Clean up */ if (io_root != root) { if (root == rank) { /* Receive result from rank io_root to recvbuf */ smpi_mpi_recv(recvbuf, count, datatype, io_root, COLL_TAG_REDUCE, comm, MPI_STATUS_IGNORE); if (MPI_IN_PLACE == sendbuf) { smpi_free_tmp_buffer(use_this_sendbuf); } } else if (io_root == rank) { /* Send result from use_this_recvbuf to root */ smpi_mpi_send(use_this_recvbuf, count, datatype, root, COLL_TAG_REDUCE, comm); smpi_free_tmp_buffer(use_this_recvbuf); } } return MPI_SUCCESS; } /* * Linear functions are copied from the BASIC coll module * they do not segment the message and are simple implementations * but for some small number of nodes and/or small data sizes they * are just as fast as tuned/tree based segmenting operations * and as such may be selected by the decision functions * These are copied into this module due to the way we select modules * in V1. i.e. in V2 we will handle this differently and so will not * have to duplicate code. * GEF Oct05 after asking Jeff. */ /* copied function (with appropriate renaming) starts here */ /* * reduce_lin_intra * * Function: - reduction using O(N) algorithm * Accepts: - same as MPI_Reduce() * Returns: - MPI_SUCCESS or error code */ int smpi_coll_tuned_reduce_ompi_basic_linear(void *sbuf, void *rbuf, int count, MPI_Datatype dtype, MPI_Op op, int root, MPI_Comm comm) { int i, rank, size; ptrdiff_t true_extent, lb, extent; char *free_buffer = NULL; char *pml_buffer = NULL; char *inplace_temp = NULL; char *inbuf; /* Initialize */ rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); XBT_DEBUG("coll:tuned:reduce_intra_basic_linear rank %d", rank); /* If not root, send data to the root. */ if (rank != root) { smpi_mpi_send(sbuf, count, dtype, root, COLL_TAG_REDUCE, comm); return MPI_SUCCESS; } /* see discussion in ompi_coll_basic_reduce_lin_intra about extent and true extent */ /* for reducing buffer allocation lengths.... */ smpi_datatype_extent(dtype, &lb, &extent); true_extent = smpi_datatype_get_extent(dtype); if (MPI_IN_PLACE == sbuf) { sbuf = rbuf; inplace_temp = (char*)smpi_get_tmp_recvbuffer(true_extent + (count - 1) * extent); if (NULL == inplace_temp) { return -1; } rbuf = inplace_temp - lb; } if (size > 1) { free_buffer = (char*)smpi_get_tmp_recvbuffer(true_extent + (count - 1) * extent); pml_buffer = free_buffer - lb; } /* Initialize the receive buffer. */ if (rank == (size - 1)) { smpi_datatype_copy((char*)sbuf, count, dtype,(char*)rbuf, count, dtype); } else { smpi_mpi_recv(rbuf, count, dtype, size - 1, COLL_TAG_REDUCE, comm, MPI_STATUS_IGNORE); } /* Loop receiving and calling reduction function (C or Fortran). */ for (i = size - 2; i >= 0; --i) { if (rank == i) { inbuf = (char*)sbuf; } else { smpi_mpi_recv(pml_buffer, count, dtype, i, COLL_TAG_REDUCE, comm, MPI_STATUS_IGNORE); inbuf = pml_buffer; } /* Perform the reduction */ smpi_op_apply(op, inbuf, rbuf, &count, &dtype); } if (NULL != inplace_temp) { smpi_datatype_copy(inplace_temp, count, dtype,(char*)sbuf ,count , dtype); smpi_free_tmp_buffer(inplace_temp); } if (NULL != free_buffer) { smpi_free_tmp_buffer(free_buffer); } /* All done */ return MPI_SUCCESS; } /* copied function (with appropriate renaming) ends here */ SimGrid-3.14.159/src/smpi/colls/reduce-NTSL.c000644 001750 001750 00000012514 13031031272 022137 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" //#include int reduce_NTSL_segment_size_in_byte = 8192; /* Non-topology-specific pipelined linear-bcast function 0->1, 1->2 ,2->3, ....., ->last node : in a pipeline fashion */ int smpi_coll_tuned_reduce_NTSL(void *buf, void *rbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { int tag = COLL_TAG_REDUCE; MPI_Status status; MPI_Request *send_request_array; MPI_Request *recv_request_array; MPI_Status *send_status_array; MPI_Status *recv_status_array; int rank, size; int i; MPI_Aint extent; extent = smpi_datatype_get_extent(datatype); rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); /* source node and destination nodes (same through out the functions) */ int to = (rank - 1 + size) % size; int from = (rank + 1) % size; /* segment is segment size in number of elements (not bytes) */ int segment = reduce_NTSL_segment_size_in_byte / extent; /* pipeline length */ int pipe_length = count / segment; /* use for buffer offset for sending and receiving data = segment size in byte */ int increment = segment * extent; /* if the input size is not divisible by segment size => the small remainder will be done with native implementation */ int remainder = count % segment; /* if root is not zero send to rank zero first this can be modified to make it faster by using logical src, dst. */ /* if (root != 0) { if (rank == root){ smpi_mpi_send(buf,count,datatype,0,tag,comm); } else if (rank == 0) { smpi_mpi_recv(buf,count,datatype,root,tag,comm,&status); } } */ char *tmp_buf; tmp_buf = (char *) smpi_get_tmp_sendbuffer(count * extent); smpi_mpi_sendrecv(buf, count, datatype, rank, tag, rbuf, count, datatype, rank, tag, comm, &status); /* when a message is smaller than a block size => no pipeline */ if (count <= segment) { if (rank == root) { smpi_mpi_recv(tmp_buf, count, datatype, from, tag, comm, &status); smpi_op_apply(op, tmp_buf, rbuf, &count, &datatype); } else if (rank == ((root - 1 + size) % size)) { smpi_mpi_send(rbuf, count, datatype, to, tag, comm); } else { smpi_mpi_recv(tmp_buf, count, datatype, from, tag, comm, &status); smpi_op_apply(op, tmp_buf, rbuf, &count, &datatype); smpi_mpi_send(rbuf, count, datatype, to, tag, comm); } smpi_free_tmp_buffer(tmp_buf); return MPI_SUCCESS; } /* pipeline */ else { send_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); recv_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); send_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); recv_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); /* root recv data */ if (rank == root) { for (i = 0; i < pipe_length; i++) { recv_request_array[i] = smpi_mpi_irecv((char *) tmp_buf + (i * increment), segment, datatype, from, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { smpi_mpi_wait(&recv_request_array[i], &status); smpi_op_apply(op, tmp_buf + (i * increment), (char *)rbuf + (i * increment), &segment, &datatype); } } /* last node only sends data */ else if (rank == ((root - 1 + size) % size)) { for (i = 0; i < pipe_length; i++) { send_request_array[i] = smpi_mpi_isend((char *)rbuf + (i * increment), segment, datatype, to, (tag + i), comm); } smpi_mpi_waitall((pipe_length), send_request_array, send_status_array); } /* intermediate nodes relay (receive, reduce, then send) data */ else { for (i = 0; i < pipe_length; i++) { recv_request_array[i] = smpi_mpi_irecv((char *) tmp_buf + (i * increment), segment, datatype, from, (tag + i), comm); } for (i = 0; i < pipe_length; i++) { smpi_mpi_wait(&recv_request_array[i], &status); smpi_op_apply(op, tmp_buf + (i * increment), (char *)rbuf + (i * increment), &segment, &datatype); send_request_array[i] = smpi_mpi_isend((char *) rbuf + (i * increment), segment, datatype, to, (tag + i), comm); } smpi_mpi_waitall((pipe_length), send_request_array, send_status_array); } free(send_request_array); free(recv_request_array); free(send_status_array); free(recv_status_array); } /* end pipeline */ /* when count is not divisible by block size, use default BCAST for the remainder */ if ((remainder != 0) && (count > segment)) { XBT_WARN("MPI_reduce_NTSL use default MPI_reduce."); smpi_mpi_reduce((char *)buf + (pipe_length * increment), (char *)rbuf + (pipe_length * increment), remainder, datatype, op, root, comm); } free(tmp_buf); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/reduce-arrival-pattern-aware.c000644 001750 001750 00000025522 13031031272 025572 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" //#include int reduce_arrival_pattern_aware_segment_size_in_byte = 8192; #ifndef HEADER_SIZE #define HEADER_SIZE 1024 #endif #ifndef MAX_NODE #define MAX_NODE 1024 #endif /* Non-topology-specific pipelined linear-reduce function */ int smpi_coll_tuned_reduce_arrival_pattern_aware(void *buf, void *rbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { int rank = smpi_comm_rank(comm); int tag = -COLL_TAG_REDUCE; MPI_Status status; MPI_Request request; MPI_Request *send_request_array; MPI_Request *recv_request_array; MPI_Status *send_status_array; MPI_Status *recv_status_array; MPI_Status temp_status_array[MAX_NODE]; int size = smpi_comm_size(comm); int i; int sent_count; int header_index; int flag_array[MAX_NODE]; int already_received[MAX_NODE]; int header_buf[HEADER_SIZE]; char temp_buf[MAX_NODE]; MPI_Aint extent, lb; smpi_datatype_extent(datatype, &lb, &extent); /* source and destination */ int to, from; /* segment is segment size in number of elements (not bytes) */ int segment = reduce_arrival_pattern_aware_segment_size_in_byte / extent; /* pipeline length */ int pipe_length = count / segment; /* use for buffer offset for sending and receiving data = segment size in byte */ int increment = segment * extent; /* if the input size is not divisible by segment size => the small remainder will be done with native implementation */ int remainder = count % segment; /* value == 0 means root has not send data (or header) to the node yet */ for (i = 0; i < MAX_NODE; i++) { already_received[i] = 0; } char *tmp_buf; tmp_buf = (char *) smpi_get_tmp_sendbuffer(count * extent); smpi_mpi_sendrecv(buf, count, datatype, rank, tag, rbuf, count, datatype, rank, tag, comm, &status); /* when a message is smaller than a block size => no pipeline */ if (count <= segment) { if (rank == 0) { sent_count = 0; while (sent_count < (size - 1)) { for (i = 1; i < size; i++) { if (already_received[i] == 0) { smpi_mpi_iprobe(i, MPI_ANY_TAG, comm, &flag_array[i], MPI_STATUSES_IGNORE); simcall_process_sleep(0.0001); } } header_index = 0; /* recv 1-byte message */ for (i = 0; i < size; i++) { if (i == rank) continue; /* 1-byte message arrive */ if ((flag_array[i] == 1) && (already_received[i] == 0)) { smpi_mpi_recv(temp_buf, 1, MPI_CHAR, i, tag, comm, &status); header_buf[header_index] = i; header_index++; sent_count++; //printf("root send to %d recv from %d : data = ",to,from); /* for (i=0;i<=header_index;i++) { printf("%d ",header_buf[i]); } printf("\n"); */ /* will receive in the next step */ already_received[i] = 1; } } /* send header followed by receive and reduce data */ if (header_index != 0) { header_buf[header_index] = -1; to = header_buf[0]; from = header_buf[header_index - 1]; smpi_mpi_send(header_buf, HEADER_SIZE, MPI_INT, to, tag, comm); smpi_mpi_recv(tmp_buf, count, datatype, from, tag, comm, &status); smpi_op_apply(op, tmp_buf, rbuf, &count, &datatype); } } /* while loop */ } /* root */ /* non-root */ else { /* send 1-byte message to root */ smpi_mpi_send(temp_buf, 1, MPI_CHAR, 0, tag, comm); /* wait for header and data, forward when required */ smpi_mpi_recv(header_buf, HEADER_SIZE, MPI_INT, MPI_ANY_SOURCE, tag, comm, &status); // smpi_mpi_recv(buf,count,datatype,MPI_ANY_SOURCE,tag,comm,&status); /* search for where it is */ int myordering = 0; while (rank != header_buf[myordering]) { myordering++; } /* forward header */ if (header_buf[myordering + 1] != -1) { smpi_mpi_send(header_buf, HEADER_SIZE, MPI_INT, header_buf[myordering + 1], tag, comm); } //printf("node %d ordering %d\n",rank,myordering); /* receive, reduce, and forward data */ /* send only */ if (myordering == 0) { if (header_buf[myordering + 1] == -1) { to = 0; } else { to = header_buf[myordering + 1]; } smpi_mpi_send(rbuf, count, datatype, to, tag, comm); } /* recv, reduce, send */ else { if (header_buf[myordering + 1] == -1) { to = 0; } else { to = header_buf[myordering + 1]; } from = header_buf[myordering - 1]; smpi_mpi_recv(tmp_buf, count, datatype, from, tag, comm, &status); smpi_op_apply(op, tmp_buf, rbuf, &count, &datatype); smpi_mpi_send(rbuf, count, datatype, to, tag, comm); } } /* non-root */ } /* pipeline bcast */ else { // printf("node %d start\n",rank); send_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); recv_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); send_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); recv_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); if (rank == 0) { sent_count = 0; int will_send[MAX_NODE]; for (i = 0; i < MAX_NODE; i++) will_send[i] = 0; /* loop until all data are received (sent) */ while (sent_count < (size - 1)) { int k; for (k = 0; k < 1; k++) { for (i = 1; i < size; i++) { //if (i == rank) //continue; if ((already_received[i] == 0) && (will_send[i] == 0)) { smpi_mpi_iprobe(i, MPI_ANY_TAG, comm, &flag_array[i], &temp_status_array[i]); if (flag_array[i] == 1) { will_send[i] = 1; smpi_mpi_recv(&temp_buf[i], 1, MPI_CHAR, i, tag, comm, &status); //printf("recv from %d\n",i); i = 1; } } } } /* end of probing */ header_index = 0; /* recv 1-byte message */ for (i = 1; i < size; i++) { //if (i==rank) //continue; /* message arrived in this round (put in the header) */ if ((will_send[i] == 1) && (already_received[i] == 0)) { header_buf[header_index] = i; header_index++; sent_count++; /* will send in the next step */ already_received[i] = 1; } } /* send header followed by data */ if (header_index != 0) { header_buf[header_index] = -1; to = header_buf[0]; /* send header */ smpi_mpi_send(header_buf, HEADER_SIZE, MPI_INT, to, tag, comm); /* recv data - pipeline */ from = header_buf[header_index - 1]; for (i = 0; i < pipe_length; i++) { smpi_mpi_recv(tmp_buf + (i * increment), segment, datatype, from, tag, comm, &status); smpi_op_apply(op, tmp_buf + (i * increment), (char *)rbuf + (i * increment), &segment, &datatype); } } } /* while loop (sent_count < size-1 ) */ } /* root */ /* none root */ else { /* send 1-byte message to root */ smpi_mpi_send(temp_buf, 1, MPI_CHAR, 0, tag, comm); /* wait for header forward when required */ request=smpi_mpi_irecv(header_buf, HEADER_SIZE, MPI_INT, MPI_ANY_SOURCE, tag, comm); smpi_mpi_wait(&request, MPI_STATUS_IGNORE); /* search for where it is */ int myordering = 0; while (rank != header_buf[myordering]) { myordering++; } /* send header when required */ if (header_buf[myordering + 1] != -1) { smpi_mpi_send(header_buf, HEADER_SIZE, MPI_INT, header_buf[myordering + 1], tag, comm); } /* (receive, reduce), and send data */ if (header_buf[myordering + 1] == -1) { to = 0; } else { to = header_buf[myordering + 1]; } /* send only */ if (myordering == 0) { for (i = 0; i < pipe_length; i++) { send_request_array[i]= smpi_mpi_isend((char *)rbuf + (i * increment), segment, datatype, to, tag, comm); } smpi_mpi_waitall((pipe_length), send_request_array, send_status_array); } /* receive, reduce, and send */ else { from = header_buf[myordering - 1]; for (i = 0; i < pipe_length; i++) { recv_request_array[i]=smpi_mpi_irecv(tmp_buf + (i * increment), segment, datatype, from, tag, comm); } for (i = 0; i < pipe_length; i++) { smpi_mpi_wait(&recv_request_array[i], MPI_STATUS_IGNORE); smpi_op_apply(op, tmp_buf + (i * increment), (char *)rbuf + (i * increment), &segment, &datatype); send_request_array[i]=smpi_mpi_isend((char *)rbuf + (i * increment), segment, datatype, to, tag, comm); } smpi_mpi_waitall((pipe_length), send_request_array, send_status_array); } } /* non-root */ free(send_request_array); free(recv_request_array); free(send_status_array); free(recv_status_array); //printf("node %d done\n",rank); } /* end pipeline */ /* if root is not zero send root after finished this can be modified to make it faster by using logical src, dst. */ if (root != 0) { if (rank == 0) { smpi_mpi_send(rbuf, count, datatype, root, tag, comm); } else if (rank == root) { smpi_mpi_recv(rbuf, count, datatype, 0, tag, comm, &status); } } /* when count is not divisible by block size, use default BCAST for the remainder */ if ((remainder != 0) && (count > segment)) { smpi_mpi_reduce((char *)buf + (pipe_length * increment), (char *)rbuf + (pipe_length * increment), remainder, datatype, op, root, comm); } smpi_free_tmp_buffer(tmp_buf); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/bcast-NTSB.c000644 001750 001750 00000015034 13031031271 021751 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" int bcast_NTSB_segment_size_in_byte = 8192; int smpi_coll_tuned_bcast_NTSB(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int tag = COLL_TAG_BCAST; MPI_Status status; int rank, size; int i; MPI_Request *send_request_array; MPI_Request *recv_request_array; MPI_Status *send_status_array; MPI_Status *recv_status_array; MPI_Aint extent; extent = smpi_datatype_get_extent(datatype); rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); /* source node and destination nodes (same through out the functions) */ int from = (rank - 1) / 2; int to_left = rank * 2 + 1; int to_right = rank * 2 + 2; if (to_left >= size) to_left = -1; if (to_right >= size) to_right = -1; /* segment is segment size in number of elements (not bytes) */ int segment = bcast_NTSB_segment_size_in_byte / extent; segment = segment == 0 ? 1 :segment; /* pipeline length */ int pipe_length = count / segment; /* use for buffer offset for sending and receiving data = segment size in byte */ int increment = segment * extent; /* if the input size is not divisible by segment size => the small remainder will be done with native implementation */ int remainder = count % segment; /* if root is not zero send to rank zero first */ if (root != 0) { if (rank == root) { smpi_mpi_send(buf, count, datatype, 0, tag, comm); } else if (rank == 0) { smpi_mpi_recv(buf, count, datatype, root, tag, comm, &status); } } /* when a message is smaller than a block size => no pipeline */ if (count <= segment) { /* case: root */ if (rank == 0) { /* case root has only a left child */ if (to_right == -1) { smpi_mpi_send(buf, count, datatype, to_left, tag, comm); } /* case root has both left and right children */ else { smpi_mpi_send(buf, count, datatype, to_left, tag, comm); smpi_mpi_send(buf, count, datatype, to_right, tag, comm); } } /* case: leaf ==> receive only */ else if (to_left == -1) { smpi_mpi_recv(buf, count, datatype, from, tag, comm, &status); } /* case: intermidiate node with only left child ==> relay message */ else if (to_right == -1) { smpi_mpi_recv(buf, count, datatype, from, tag, comm, &status); smpi_mpi_send(buf, count, datatype, to_left, tag, comm); } /* case: intermidiate node with both left and right children ==> relay message */ else { smpi_mpi_recv(buf, count, datatype, from, tag, comm, &status); smpi_mpi_send(buf, count, datatype, to_left, tag, comm); smpi_mpi_send(buf, count, datatype, to_right, tag, comm); } return MPI_SUCCESS; } // pipelining else { send_request_array = (MPI_Request *) xbt_malloc(2 * (size + pipe_length) * sizeof(MPI_Request)); recv_request_array = (MPI_Request *) xbt_malloc((size + pipe_length) * sizeof(MPI_Request)); send_status_array = (MPI_Status *) xbt_malloc(2 * (size + pipe_length) * sizeof(MPI_Status)); recv_status_array = (MPI_Status *) xbt_malloc((size + pipe_length) * sizeof(MPI_Status)); /* case: root */ if (rank == 0) { /* case root has only a left child */ if (to_right == -1) { for (i = 0; i < pipe_length; i++) { send_request_array[i] = smpi_mpi_isend((char *) buf + (i * increment), segment, datatype, to_left, tag + i, comm); } smpi_mpi_waitall((pipe_length), send_request_array, send_status_array); } /* case root has both left and right children */ else { for (i = 0; i < pipe_length; i++) { send_request_array[i] = smpi_mpi_isend((char *) buf + (i * increment), segment, datatype, to_left, tag + i, comm); send_request_array[i + pipe_length] = smpi_mpi_isend((char *) buf + (i * increment), segment, datatype, to_right, tag + i, comm); } smpi_mpi_waitall((2 * pipe_length), send_request_array, send_status_array); } } /* case: leaf ==> receive only */ else if (to_left == -1) { for (i = 0; i < pipe_length; i++) { recv_request_array[i] = smpi_mpi_irecv((char *) buf + (i * increment), segment, datatype, from, tag + i, comm); } smpi_mpi_waitall((pipe_length), recv_request_array, recv_status_array); } /* case: intermidiate node with only left child ==> relay message */ else if (to_right == -1) { for (i = 0; i < pipe_length; i++) { recv_request_array[i] = smpi_mpi_irecv((char *) buf + (i * increment), segment, datatype, from, tag + i, comm); } for (i = 0; i < pipe_length; i++) { smpi_mpi_wait(&recv_request_array[i], &status); send_request_array[i] = smpi_mpi_isend((char *) buf + (i * increment), segment, datatype, to_left, tag + i, comm); } smpi_mpi_waitall(pipe_length, send_request_array, send_status_array); } /* case: intermidiate node with both left and right children ==> relay message */ else { for (i = 0; i < pipe_length; i++) { recv_request_array[i] = smpi_mpi_irecv((char *) buf + (i * increment), segment, datatype, from, tag + i, comm); } for (i = 0; i < pipe_length; i++) { smpi_mpi_wait(&recv_request_array[i], &status); send_request_array[i] = smpi_mpi_isend((char *) buf + (i * increment), segment, datatype, to_left, tag + i, comm); send_request_array[i + pipe_length] = smpi_mpi_isend((char *) buf + (i * increment), segment, datatype, to_right, tag + i, comm); } smpi_mpi_waitall((2 * pipe_length), send_request_array, send_status_array); } free(send_request_array); free(recv_request_array); free(send_status_array); free(recv_status_array); } /* end pipeline */ /* when count is not divisible by block size, use default BCAST for the remainder */ if ((remainder != 0) && (count > segment)) { XBT_WARN("MPI_bcast_NTSB use default MPI_bcast."); smpi_mpi_bcast((char *) buf + (pipe_length * increment), remainder, datatype, root, comm); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/alltoall-pair-one-barrier.c000644 001750 001750 00000004145 13031031271 025052 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** * Function: alltoall_pair * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works when P is power of two. In each phase of P - 1 phases, nodes in pair communicate their data. * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_alltoall_pair_one_barrier(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint send_chunk, recv_chunk; MPI_Status s; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALL; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "alltoall pair algorithm can't be used with non power of two number of processes ! "); send_chunk = smpi_datatype_get_extent(send_type); recv_chunk = smpi_datatype_get_extent(recv_type); send_chunk *= send_count; recv_chunk *= recv_count; mpi_coll_barrier_fun(comm); for (i = 0; i < num_procs; i++) { src = dst = rank ^ i; smpi_mpi_sendrecv(send_ptr + dst * send_chunk, send_count, send_type, dst, tag, recv_ptr + src * recv_chunk, recv_count, recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/gather-ompi.c000644 001750 001750 00000032421 13031031272 022325 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2009 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * * Additional copyrights may follow */ #include "colls_private.h" #include "coll_tuned_topo.h" /* Todo: gather_intra_generic, gather_intra_binary, gather_intra_chain, * gather_intra_pipeline, segmentation? */ int smpi_coll_tuned_gather_ompi_binomial(void *sbuf, int scount, MPI_Datatype sdtype, void *rbuf, int rcount, MPI_Datatype rdtype, int root, MPI_Comm comm) { int line = -1; int i; int rank; int vrank; int size; int total_recv = 0; char *ptmp = NULL; char *tempbuf = NULL; int err; ompi_coll_tree_t* bmtree; MPI_Status status; MPI_Aint sextent, slb, strue_lb, strue_extent; MPI_Aint rextent, rlb, rtrue_lb, rtrue_extent; size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); XBT_DEBUG( "smpi_coll_tuned_gather_ompi_binomial rank %d", rank); /* create the binomial tree */ // COLL_TUNED_UPDATE_IN_ORDER_BMTREE( comm, tuned_module, root ); bmtree = ompi_coll_tuned_topo_build_in_order_bmtree(comm, root); // data->cached_in_order_bmtree; smpi_datatype_extent(sdtype, &slb, &sextent); smpi_datatype_extent(sdtype, &strue_lb, &strue_extent); vrank = (rank - root + size) % size; if (rank == root) { smpi_datatype_extent(rdtype, &rlb, &rextent); smpi_datatype_extent(rdtype, &rtrue_lb, &rtrue_extent); if (0 == root){ /* root on 0, just use the recv buffer */ ptmp = (char *) rbuf; if (sbuf != MPI_IN_PLACE) { err = smpi_datatype_copy(sbuf, scount, sdtype, ptmp, rcount, rdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } } } else { /* root is not on 0, allocate temp buffer for recv, * rotate data at the end */ tempbuf = (char *) smpi_get_tmp_recvbuffer(rtrue_extent + (rcount*size - 1) * rextent); if (NULL == tempbuf) { err= MPI_ERR_OTHER; line = __LINE__; goto err_hndl; } ptmp = tempbuf - rlb; if (sbuf != MPI_IN_PLACE) { /* copy from sbuf to temp buffer */ err = smpi_datatype_copy(sbuf, scount, sdtype, ptmp, rcount, rdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } } else { /* copy from rbuf to temp buffer */ err = smpi_datatype_copy((char *) rbuf + rank*rextent*rcount, rcount, rdtype, ptmp, rcount, rdtype ); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } } } total_recv = rcount; } else if (!(vrank % 2)) { /* other non-leaf nodes, allocate temp buffer for data received from * children, the most we need is half of the total data elements due * to the property of binimoal tree */ tempbuf = (char *) smpi_get_tmp_sendbuffer(strue_extent + (scount*size - 1) * sextent); if (NULL == tempbuf) { err= MPI_ERR_OTHER; line = __LINE__; goto err_hndl; } ptmp = tempbuf - slb; /* local copy to tempbuf */ err = smpi_datatype_copy(sbuf, scount, sdtype, ptmp, scount, sdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } /* use sdtype,scount as rdtype,rdcount since they are ignored on * non-root procs */ rdtype = sdtype; rcount = scount; rextent = sextent; total_recv = rcount; } else { /* leaf nodes, no temp buffer needed, use sdtype,scount as * rdtype,rdcount since they are ignored on non-root procs */ ptmp = (char *) sbuf; total_recv = scount; } if (!(vrank % 2)) { /* all non-leaf nodes recv from children */ for (i = 0; i < bmtree->tree_nextsize; i++) { int mycount = 0, vkid; /* figure out how much data I have to send to this child */ vkid = (bmtree->tree_next[i] - root + size) % size; mycount = vkid - vrank; if (mycount > (size - vkid)) mycount = size - vkid; mycount *= rcount; XBT_DEBUG( "smpi_coll_tuned_gather_ompi_binomial rank %d recv %d mycount = %d", rank, bmtree->tree_next[i], mycount); smpi_mpi_recv(ptmp + total_recv*rextent, mycount, rdtype, bmtree->tree_next[i], COLL_TAG_GATHER, comm, &status); total_recv += mycount; } } if (rank != root) { /* all nodes except root send to parents */ XBT_DEBUG( "smpi_coll_tuned_gather_ompi_binomial rank %d send %d count %d\n", rank, bmtree->tree_prev, total_recv); smpi_mpi_send(ptmp, total_recv, sdtype, bmtree->tree_prev, COLL_TAG_GATHER, comm); } if (rank == root) { if (root != 0) { /* rotate received data on root if root != 0 */ err = smpi_datatype_copy(ptmp, rcount*(size - root), rdtype, (char *) rbuf + rextent*root*rcount, rcount*(size - root), rdtype ); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } err = smpi_datatype_copy( ptmp + rextent*rcount*(size-root), rcount*root,rdtype, (char *) rbuf,rcount*root,rdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } smpi_free_tmp_buffer(tempbuf); } } else if (!(vrank % 2)) { /* other non-leaf nodes */ smpi_free_tmp_buffer(tempbuf); } xbt_free(bmtree); return MPI_SUCCESS; err_hndl: if (NULL != tempbuf) smpi_free_tmp_buffer(tempbuf); XBT_DEBUG( "%s:%4d\tError occurred %d, rank %2d", __FILE__, line, err, rank); return err; } /* * gather_intra_linear_sync * * Function: - synchronized gather operation with * Accepts: - same arguments as MPI_Gather(), first segment size * Returns: - MPI_SUCCESS or error code */ int smpi_coll_tuned_gather_ompi_linear_sync(void *sbuf, int scount, MPI_Datatype sdtype, void *rbuf, int rcount, MPI_Datatype rdtype, int root, MPI_Comm comm) { int i; int ret, line; int rank, size; int first_segment_count; size_t typelng; MPI_Aint extent; MPI_Aint lb; int first_segment_size=0; size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); size_t dsize, block_size; if (rank == root) { dsize= smpi_datatype_size(rdtype); block_size = dsize * rcount; } else { dsize=smpi_datatype_size(sdtype); block_size = dsize * scount; } if (block_size > 92160){ first_segment_size = 32768; }else{ first_segment_size = 1024; } XBT_DEBUG( "smpi_coll_tuned_gather_ompi_linear_sync rank %d, segment %d", rank, first_segment_size); if (rank != root) { /* Non-root processes: - receive zero byte message from the root, - send the first segment of the data synchronously, - send the second segment of the data. */ typelng= smpi_datatype_size(sdtype); smpi_datatype_extent(sdtype, &lb, &extent); first_segment_count = scount; COLL_TUNED_COMPUTED_SEGCOUNT( (size_t) first_segment_size, typelng, first_segment_count ); smpi_mpi_recv(sbuf, 0, MPI_BYTE, root, COLL_TAG_GATHER, comm, MPI_STATUS_IGNORE); smpi_mpi_send(sbuf, first_segment_count, sdtype, root, COLL_TAG_GATHER, comm); smpi_mpi_send((char*)sbuf + extent * first_segment_count, (scount - first_segment_count), sdtype, root, COLL_TAG_GATHER, comm); } else { /* Root process, - For every non-root node: - post irecv for the first segment of the message - send zero byte message to signal node to send the message - post irecv for the second segment of the message - wait for the first segment to complete - Copy local data if necessary - Waitall for all the second segments to complete. */ char *ptmp; MPI_Request *reqs = NULL, first_segment_req; reqs = (MPI_Request *) calloc(size, sizeof(MPI_Request )); if (NULL == reqs) { ret = -1; line = __LINE__; goto error_hndl; } typelng=smpi_datatype_size(rdtype); smpi_datatype_extent(rdtype, &lb, &extent); first_segment_count = rcount; COLL_TUNED_COMPUTED_SEGCOUNT( (size_t)first_segment_size, typelng, first_segment_count ); for (i = 0; i < size; ++i) { if (i == rank) { /* skip myself */ reqs[i] = MPI_REQUEST_NULL; continue; } /* irecv for the first segment from i */ ptmp = (char*)rbuf + i * rcount * extent; first_segment_req = smpi_mpi_irecv(ptmp, first_segment_count, rdtype, i, COLL_TAG_GATHER, comm ); /* send sync message */ smpi_mpi_send(rbuf, 0, MPI_BYTE, i, COLL_TAG_GATHER, comm); /* irecv for the second segment */ ptmp = (char*)rbuf + (i * rcount + first_segment_count) * extent; reqs[i]=smpi_mpi_irecv(ptmp, (rcount - first_segment_count), rdtype, i, COLL_TAG_GATHER, comm ); /* wait on the first segment to complete */ smpi_mpi_wait(&first_segment_req, MPI_STATUS_IGNORE); } /* copy local data if necessary */ if (MPI_IN_PLACE != sbuf) { ret = smpi_datatype_copy(sbuf, scount, sdtype, (char*)rbuf + rank * rcount * extent, rcount, rdtype); if (ret != MPI_SUCCESS) { line = __LINE__; goto error_hndl; } } /* wait all second segments to complete */ ret = smpi_mpi_waitall(size, reqs, MPI_STATUSES_IGNORE); if (ret != MPI_SUCCESS) { line = __LINE__; goto error_hndl; } free(reqs); } /* All done */ return MPI_SUCCESS; error_hndl: XBT_DEBUG( "ERROR_HNDL: node %d file %s line %d error %d\n", rank, __FILE__, line, ret ); return ret; } /* * Linear functions are copied from the BASIC coll module * they do not segment the message and are simple implementations * but for some small number of nodes and/or small data sizes they * are just as fast as tuned/tree based segmenting operations * and as such may be selected by the decision functions * These are copied into this module due to the way we select modules * in V1. i.e. in V2 we will handle this differently and so will not * have to duplicate code. * JPG following the examples from other coll_tuned implementations. Dec06. */ /* copied function (with appropriate renaming) starts here */ /* * gather_intra * * Function: - basic gather operation * Accepts: - same arguments as MPI_Gather() * Returns: - MPI_SUCCESS or error code */ int smpi_coll_tuned_gather_ompi_basic_linear(void *sbuf, int scount, MPI_Datatype sdtype, void *rbuf, int rcount, MPI_Datatype rdtype, int root, MPI_Comm comm) { int i; int err; int rank; int size; char *ptmp; MPI_Aint incr; MPI_Aint extent; MPI_Aint lb; size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); /* Everyone but root sends data and returns. */ XBT_DEBUG( "ompi_coll_tuned_gather_intra_basic_linear rank %d", rank); if (rank != root) { smpi_mpi_send(sbuf, scount, sdtype, root, COLL_TAG_GATHER, comm); return MPI_SUCCESS; } /* I am the root, loop receiving the data. */ smpi_datatype_extent(rdtype, &lb, &extent); incr = extent * rcount; for (i = 0, ptmp = (char *) rbuf; i < size; ++i, ptmp += incr) { if (i == rank) { if (MPI_IN_PLACE != sbuf) { err = smpi_datatype_copy(sbuf, scount, sdtype, ptmp, rcount, rdtype); } else { err = MPI_SUCCESS; } } else { smpi_mpi_recv(ptmp, rcount, rdtype, i, COLL_TAG_GATHER, comm, MPI_STATUS_IGNORE); err = MPI_SUCCESS; } if (MPI_SUCCESS != err) { return err; } } /* All done */ return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allgatherv-GB.c000644 001750 001750 00000002010 13031031271 022516 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" // Allgather - gather/bcast algorithm int smpi_coll_tuned_allgatherv_GB(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { smpi_mpi_gatherv(send_buff, send_count, send_type, recv_buff, recv_counts, recv_disps, recv_type, 0, comm); int num_procs, i, current, max = 0; num_procs = smpi_comm_size(comm); for (i = 0; i < num_procs; i++) { current = recv_disps[i] + recv_counts[i]; if (current > max) max = current; } mpi_coll_bcast_fun(recv_buff, max, recv_type, 0, comm); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/bcast-mvapich-smp.c000644 001750 001750 00000034543 13031031272 023436 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2009 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * * Additional copyrights may follow */ /* -*- Mode: C; c-basic-offset:4 ; -*- */ /* Copyright (c) 2001-2014, The Ohio State University. All rights * reserved. * * This file is part of the MVAPICH2 software package developed by the * team members of The Ohio State University's Network-Based Computing * Laboratory (NBCL), headed by Professor Dhabaleswar K. (DK) Panda. * * For detailed copyright and licensing information, please refer to the * copyright file COPYRIGHT in the top level MVAPICH2 directory. */ /* * * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ #include "colls_private.h" extern int (*MV2_Bcast_function) (void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm_ptr); extern int (*MV2_Bcast_intra_node_function) (void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm_ptr); extern int zcpy_knomial_factor; extern int mv2_pipelined_zcpy_knomial_factor; extern int bcast_segment_size; extern int mv2_inter_node_knomial_factor; extern int mv2_intra_node_knomial_factor; extern int mv2_bcast_two_level_system_size; #define INTRA_NODE_ROOT 0 #define MPIR_Pipelined_Bcast_Zcpy_MV2 smpi_coll_tuned_bcast_mpich #define MPIR_Pipelined_Bcast_MV2 smpi_coll_tuned_bcast_mpich #define MPIR_Bcast_binomial_MV2 smpi_coll_tuned_bcast_binomial_tree #define MPIR_Bcast_scatter_ring_allgather_shm_MV2 smpi_coll_tuned_bcast_scatter_LR_allgather #define MPIR_Bcast_scatter_doubling_allgather_MV2 smpi_coll_tuned_bcast_scatter_rdb_allgather #define MPIR_Bcast_scatter_ring_allgather_MV2 smpi_coll_tuned_bcast_scatter_LR_allgather #define MPIR_Shmem_Bcast_MV2 smpi_coll_tuned_bcast_mpich #define MPIR_Bcast_tune_inter_node_helper_MV2 smpi_coll_tuned_bcast_mvapich2_inter_node #define MPIR_Bcast_inter_node_helper_MV2 smpi_coll_tuned_bcast_mvapich2_inter_node #define MPIR_Knomial_Bcast_intra_node_MV2 smpi_coll_tuned_bcast_mvapich2_knomial_intra_node #define MPIR_Bcast_intra_MV2 smpi_coll_tuned_bcast_mvapich2_intra_node extern int zcpy_knomial_factor; extern int mv2_pipelined_zcpy_knomial_factor; extern int bcast_segment_size; extern int mv2_inter_node_knomial_factor; extern int mv2_intra_node_knomial_factor; #define mv2_bcast_two_level_system_size 64 #define mv2_bcast_short_msg 16384 #define mv2_bcast_large_msg 512*1024 #define mv2_knomial_intra_node_threshold 131072 #define mv2_scatter_rd_inter_leader_bcast 1 int smpi_coll_tuned_bcast_mvapich2_inter_node(void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int rank; int mpi_errno = MPI_SUCCESS; MPI_Comm shmem_comm, leader_comm; int local_rank, local_size, global_rank = -1; int leader_root, leader_of_root; rank = smpi_comm_rank(comm); //comm_size = smpi_comm_size(comm); if (MV2_Bcast_function==NULL){ MV2_Bcast_function=smpi_coll_tuned_bcast_mpich; } if (MV2_Bcast_intra_node_function==NULL){ MV2_Bcast_intra_node_function= smpi_coll_tuned_bcast_mpich; } if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } shmem_comm = smpi_comm_get_intra_comm(comm); local_rank = smpi_comm_rank(shmem_comm); local_size = smpi_comm_size(shmem_comm); leader_comm = smpi_comm_get_leaders_comm(comm); if ((local_rank == 0) && (local_size > 1)) { global_rank = smpi_comm_rank(leader_comm); } int* leaders_map = smpi_comm_get_leaders_map(comm); leader_of_root = smpi_group_rank(smpi_comm_group(comm),leaders_map[root]); leader_root = smpi_group_rank(smpi_comm_group(leader_comm),leaders_map[root]); if (local_size > 1) { if ((local_rank == 0) && (root != rank) && (leader_root == global_rank)) { smpi_mpi_recv(buffer, count, datatype, root, COLL_TAG_BCAST, comm, MPI_STATUS_IGNORE); } if ((local_rank != 0) && (root == rank)) { smpi_mpi_send(buffer, count, datatype, leader_of_root, COLL_TAG_BCAST, comm); } } #if defined(_MCST_SUPPORT_) if (comm_ptr->ch.is_mcast_ok) { mpi_errno = MPIR_Mcast_inter_node_MV2(buffer, count, datatype, root, comm_ptr, errflag); if (mpi_errno == MPI_SUCCESS) { goto fn_exit; } else { goto fn_fail; } } #endif /* if (local_rank == 0) { leader_comm = smpi_comm_get_leaders_comm(comm); root = leader_root; } if (MV2_Bcast_function == &MPIR_Pipelined_Bcast_MV2) { mpi_errno = MPIR_Pipelined_Bcast_MV2(buffer, count, datatype, root, comm); } else if (MV2_Bcast_function == &MPIR_Bcast_scatter_ring_allgather_shm_MV2) { mpi_errno = MPIR_Bcast_scatter_ring_allgather_shm_MV2(buffer, count, datatype, root, comm); } else */{ if (local_rank == 0) { /* if (MV2_Bcast_function == &MPIR_Knomial_Bcast_inter_node_wrapper_MV2) { mpi_errno = MPIR_Knomial_Bcast_inter_node_wrapper_MV2(buffer, count, datatype, root, comm); } else {*/ mpi_errno = MV2_Bcast_function(buffer, count, datatype, leader_root, leader_comm); // } } } return mpi_errno; } int smpi_coll_tuned_bcast_mvapich2_knomial_intra_node(void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int local_size = 0, rank; int mpi_errno = MPI_SUCCESS; MPI_Request *reqarray = NULL; MPI_Status *starray = NULL; int src, dst, mask, relative_rank; int k; if (MV2_Bcast_function==NULL){ MV2_Bcast_function=smpi_coll_tuned_bcast_mpich; } if (MV2_Bcast_intra_node_function==NULL){ MV2_Bcast_intra_node_function= smpi_coll_tuned_bcast_mpich; } if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } local_size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); reqarray=(MPI_Request *)xbt_malloc(2 * mv2_intra_node_knomial_factor * sizeof (MPI_Request)); starray=(MPI_Status *)xbt_malloc(2 * mv2_intra_node_knomial_factor * sizeof (MPI_Status)); /* intra-node k-nomial bcast */ if (local_size > 1) { relative_rank = (rank >= root) ? rank - root : rank - root + local_size; mask = 0x1; while (mask < local_size) { if (relative_rank % (mv2_intra_node_knomial_factor * mask)) { src = relative_rank / (mv2_intra_node_knomial_factor * mask) * (mv2_intra_node_knomial_factor * mask) + root; if (src >= local_size) { src -= local_size; } smpi_mpi_recv(buffer, count, datatype, src, COLL_TAG_BCAST, comm, MPI_STATUS_IGNORE); break; } mask *= mv2_intra_node_knomial_factor; } mask /= mv2_intra_node_knomial_factor; while (mask > 0) { int reqs = 0; for (k = 1; k < mv2_intra_node_knomial_factor; k++) { if (relative_rank + mask * k < local_size) { dst = rank + mask * k; if (dst >= local_size) { dst -= local_size; } reqarray[reqs++]=smpi_mpi_isend(buffer, count, datatype, dst, COLL_TAG_BCAST, comm); } } smpi_mpi_waitall(reqs, reqarray, starray); mask /= mv2_intra_node_knomial_factor; } } xbt_free(reqarray); xbt_free(starray); return mpi_errno; } int smpi_coll_tuned_bcast_mvapich2_intra_node(void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm) { int mpi_errno = MPI_SUCCESS; int comm_size; int two_level_bcast = 1; size_t nbytes = 0; int is_homogeneous, is_contig; MPI_Aint type_size; void *tmp_buf = NULL; MPI_Comm shmem_comm; if (count == 0) return MPI_SUCCESS; if (MV2_Bcast_function==NULL){ MV2_Bcast_function=smpi_coll_tuned_bcast_mpich; } if (MV2_Bcast_intra_node_function==NULL){ MV2_Bcast_intra_node_function= smpi_coll_tuned_bcast_mpich; } if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } comm_size = smpi_comm_size(comm); // rank = smpi_comm_rank(comm); /* if (HANDLE_GET_KIND(datatype) == HANDLE_KIND_BUILTIN)*/ is_contig = 1; /* else { MPID_Datatype_get_ptr(datatype, dtp); is_contig = dtp->is_contig; } */ is_homogeneous = 1; #ifdef MPID_HAS_HETERO if (comm_ptr->is_hetero) is_homogeneous = 0; #endif /* MPI_Type_size() might not give the accurate size of the packed * datatype for heterogeneous systems (because of padding, encoding, * etc). On the other hand, MPI_Pack_size() can become very * expensive, depending on the implementation, especially for * heterogeneous systems. We want to use MPI_Type_size() wherever * possible, and MPI_Pack_size() in other places. */ //if (is_homogeneous) { type_size=smpi_datatype_size(datatype); //} /* else {*/ /* MPIR_Pack_size_impl(1, datatype, &type_size);*/ /* }*/ nbytes = (size_t) (count) * (type_size); if (comm_size <= mv2_bcast_two_level_system_size) { if (nbytes > mv2_bcast_short_msg && nbytes < mv2_bcast_large_msg) { two_level_bcast = 1; } else { two_level_bcast = 0; } } if (two_level_bcast == 1 #if defined(_MCST_SUPPORT_) || comm_ptr->ch.is_mcast_ok #endif ) { if (!is_contig || !is_homogeneous) { tmp_buf=(void *)smpi_get_tmp_sendbuffer(nbytes); /* TODO: Pipeline the packing and communication */ // position = 0; /* if (rank == root) {*/ /* mpi_errno =*/ /* MPIR_Pack_impl(buffer, count, datatype, tmp_buf, nbytes, &position);*/ /* if (mpi_errno)*/ /* MPIU_ERR_POP(mpi_errno);*/ /* }*/ } shmem_comm = smpi_comm_get_intra_comm(comm); if (!is_contig || !is_homogeneous) { mpi_errno = MPIR_Bcast_inter_node_helper_MV2(tmp_buf, nbytes, MPI_BYTE, root, comm); } else { mpi_errno = MPIR_Bcast_inter_node_helper_MV2(buffer, count, datatype, root, comm); } /* We are now done with the inter-node phase */ if (nbytes <= mv2_knomial_intra_node_threshold) { if (!is_contig || !is_homogeneous) { mpi_errno = MPIR_Shmem_Bcast_MV2(tmp_buf, nbytes, MPI_BYTE, root, shmem_comm); } else { mpi_errno = MPIR_Shmem_Bcast_MV2(buffer, count, datatype, root, shmem_comm); } } else { if (!is_contig || !is_homogeneous) { mpi_errno = MPIR_Knomial_Bcast_intra_node_MV2(tmp_buf, nbytes, MPI_BYTE, INTRA_NODE_ROOT, shmem_comm); } else { mpi_errno = MPIR_Knomial_Bcast_intra_node_MV2(buffer, count, datatype, INTRA_NODE_ROOT, shmem_comm); } } } else { if (nbytes <= mv2_bcast_short_msg) { mpi_errno = MPIR_Bcast_binomial_MV2(buffer, count, datatype, root, comm); } else { if (mv2_scatter_rd_inter_leader_bcast) { mpi_errno = MPIR_Bcast_scatter_ring_allgather_MV2(buffer, count, datatype, root, comm); } else { mpi_errno = MPIR_Bcast_scatter_doubling_allgather_MV2(buffer, count, datatype, root, comm); } } } return mpi_errno; } SimGrid-3.14.159/src/smpi/colls/alltoallv-pair-light-barrier.c000644 001750 001750 00000005337 13031031271 025572 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** * Function: alltoall_pair_light_barrier * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works in P - 1 steps. In step i, node j exchanges data with node i ^ j. Light barriers are inserted between communications in different phases. * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_alltoallv_pair_light_barrier(void *send_buff, int *send_counts, int *send_disps, MPI_Datatype send_type, void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint send_chunk, recv_chunk; MPI_Status s; int i, src, dst, rank, num_procs, next_partner; int tag = COLL_TAG_ALLTOALLV; /*, failure = 0; */ char send_sync = 'a', recv_sync = 'b'; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "alltoallv pair algorithm can't be used with non power of two number of processes ! "); send_chunk = smpi_datatype_get_extent(send_type); recv_chunk = smpi_datatype_get_extent(recv_type); smpi_mpi_sendrecv(send_ptr + send_disps[rank] * send_chunk, send_counts[rank], send_type, rank, tag, recv_ptr + recv_disps[rank] * recv_chunk, recv_counts[rank], recv_type, rank, tag, comm, &s); for (i = 1; i < num_procs; i++) { src = dst = rank ^ i; smpi_mpi_sendrecv(send_ptr + send_disps[dst] * send_chunk, send_counts[dst], send_type, dst, tag, recv_ptr + recv_disps[src] *recv_chunk, recv_counts[dst], recv_type, src, tag, comm, &s); if ((i + 1) < num_procs) { next_partner = rank ^ (i + 1); smpi_mpi_sendrecv(&send_sync, 1, MPI_CHAR, next_partner, tag, &recv_sync, 1, MPI_CHAR, next_partner, tag, comm, &s); } } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/barrier-mvapich2-pair.c000644 001750 001750 00000006576 13031031271 024212 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2006 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * Copyright (c) 2008 Sun Microsystems, Inc. All rights reserved. * * Additional copyrights may follow */ /* -*- Mode: C; c-basic-offset:4 ; -*- */ /* * * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ /* Copyright (c) 2001-2014, The Ohio State University. All rights * reserved. * * This file is part of the MVAPICH2 software package developed by the * team members of The Ohio State University's Network-Based Computing * Laboratory (NBCL), headed by Professor Dhabaleswar K. (DK) Panda. * * For detailed copyright and licensing information, please refer to the * copyright file COPYRIGHT in the top level MVAPICH2 directory. * */ #include "colls_private.h" #include "coll_tuned_topo.h" int smpi_coll_tuned_barrier_mvapich2_pair(MPI_Comm comm) { int size, rank; int d, dst, src; int mpi_errno = MPI_SUCCESS; size = smpi_comm_size(comm); /* Trivial barriers return immediately */ if (size == 1) return MPI_SUCCESS; rank = smpi_comm_rank(comm); int N2_prev = 1; /* N2_prev = greatest power of two < size of Comm */ for( N2_prev = 1; N2_prev <= size; N2_prev <<= 1 ); N2_prev >>= 1; int surfeit = size - N2_prev; /* Perform a combine-like operation */ if (rank < N2_prev) { if (rank < surfeit) { /* get the fanin letter from the upper "half" process: */ dst = N2_prev + rank; smpi_mpi_recv(NULL, 0, MPI_BYTE, dst, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } /* combine on embedded N2_prev power-of-two processes */ for (d = 1; d < N2_prev; d <<= 1) { dst = (rank ^ d); smpi_mpi_sendrecv(NULL, 0, MPI_BYTE, dst, COLL_TAG_BARRIER, NULL, 0, MPI_BYTE, dst, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } /* fanout data to nodes above N2_prev... */ if (rank < surfeit) { dst = N2_prev + rank; smpi_mpi_send(NULL, 0, MPI_BYTE, dst, COLL_TAG_BARRIER, comm); } } else { /* fanin data to power of 2 subset */ src = rank - N2_prev; smpi_mpi_sendrecv(NULL, 0, MPI_BYTE, src, COLL_TAG_BARRIER, NULL, 0, MPI_BYTE, src, COLL_TAG_BARRIER, comm, MPI_STATUS_IGNORE); } return mpi_errno; } SimGrid-3.14.159/src/smpi/colls/alltoall-2dmesh.c000644 001750 001750 00000011733 13031031271 023077 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" #include /***************************************************************************** * Function: alltoall_2dmesh_shoot * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function realizes the alltoall operation using the 2dmesh algorithm. It actually performs allgather operation in x dimension then in the y dimension. Each node then extracts the needed data. The communication in each dimension follows "simple." * Auther: Ahmad Faraj ****************************************************************************/ static int alltoall_check_is_2dmesh(int num, int *i, int *j) { int x, max = num / 2; x = sqrt(num); while (x <= max) { if ((num % x) == 0) { *i = x; *j = num / x; if (*i > *j) { x = *i; *i = *j; *j = x; } return 1; } x++; } return 0; } int smpi_coll_tuned_alltoall_2dmesh(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Status *statuses, s; MPI_Request *reqs, *req_ptr;; MPI_Aint extent; char *tmp_buff1, *tmp_buff2; int i, j, src, dst, rank, num_procs, count, num_reqs; int X, Y, send_offset, recv_offset; int my_row_base, my_col_base, src_row_base, block_size; int tag = COLL_TAG_ALLTOALL; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); extent = smpi_datatype_get_extent(send_type); if (!alltoall_check_is_2dmesh(num_procs, &X, &Y)) return MPI_ERR_OTHER; my_row_base = (rank / Y) * Y; my_col_base = rank % Y; block_size = extent * send_count; tmp_buff1 = (char *) smpi_get_tmp_sendbuffer(block_size * num_procs * Y); tmp_buff2 = (char *) smpi_get_tmp_recvbuffer(block_size * Y); num_reqs = X; if (Y > X) num_reqs = Y; statuses = (MPI_Status *) xbt_malloc(num_reqs * sizeof(MPI_Status)); reqs = (MPI_Request *) xbt_malloc(num_reqs * sizeof(MPI_Request)); req_ptr = reqs; count = send_count * num_procs; for (i = 0; i < Y; i++) { src = i + my_row_base; if (src == rank) continue; recv_offset = (src % Y) * block_size * num_procs; *(req_ptr++) = smpi_mpi_irecv(tmp_buff1 + recv_offset, count, recv_type, src, tag, comm); } for (i = 0; i < Y; i++) { dst = i + my_row_base; if (dst == rank) continue; smpi_mpi_send(send_buff, count, send_type, dst, tag, comm); } smpi_mpi_waitall(Y - 1, reqs, statuses); req_ptr = reqs; for (i = 0; i < Y; i++) { send_offset = (rank * block_size) + (i * block_size * num_procs); recv_offset = (my_row_base * block_size) + (i * block_size); if (i + my_row_base == rank) smpi_mpi_sendrecv((char *) send_buff + recv_offset, send_count, send_type, rank, tag, (char *) recv_buff + recv_offset, recv_count, recv_type, rank, tag, comm, &s); else smpi_mpi_sendrecv(tmp_buff1 + send_offset, send_count, send_type, rank, tag, (char *) recv_buff + recv_offset, recv_count, recv_type, rank, tag, comm, &s); } for (i = 0; i < X; i++) { src = (i * Y + my_col_base); if (src == rank) continue; src_row_base = (src / Y) * Y; *(req_ptr++) = smpi_mpi_irecv((char *) recv_buff + src_row_base * block_size, recv_count * Y, recv_type, src, tag, comm); } for (i = 0; i < X; i++) { dst = (i * Y + my_col_base); if (dst == rank) continue; recv_offset = 0; for (j = 0; j < Y; j++) { send_offset = (dst + j * num_procs) * block_size; if (j + my_row_base == rank) smpi_mpi_sendrecv((char *) send_buff + dst * block_size, send_count, send_type, rank, tag, tmp_buff2 + recv_offset, recv_count, recv_type, rank, tag, comm, &s); else smpi_mpi_sendrecv(tmp_buff1 + send_offset, send_count, send_type, rank, tag, tmp_buff2 + recv_offset, recv_count, recv_type, rank, tag, comm, &s); recv_offset += block_size; } smpi_mpi_send(tmp_buff2, send_count * Y, send_type, dst, tag, comm); } smpi_mpi_waitall(X - 1, reqs, statuses); free(reqs); free(statuses); smpi_free_tmp_buffer(tmp_buff1); smpi_free_tmp_buffer(tmp_buff2); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/bcast-scatter-rdb-allgather.c000644 001750 001750 00000021562 13031031272 025362 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** Copyright (c) 2006, Ahmad Faraj & Xin Yuan, All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Florida State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ************************************************************************* * Any results obtained from executing this software require the * * acknowledgment and citation of the software and its owners. * * The full citation is given below: * * * * A. Faraj and X. Yuan. "Automatic Generation and Tuning of MPI * * Collective Communication Routines." The 19th ACM International * * Conference on Supercomputing (ICS), Cambridge, Massachusetts, * * June 20-22, 2005. * ************************************************************************* *****************************************************************************/ /***************************************************************************** * Function: bcast_scatter_rdb_allgather * Return: int * Inputs: buff: send input buffer count: number of elements to send data_type: data type of elements being sent root: source of data comm: communicator * Descrp: broadcasts using a scatter followed by rdb allgather. * Auther: MPICH / modified by Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_bcast_scatter_rdb_allgather(void *buff, int count, MPI_Datatype data_type, int root, MPI_Comm comm) { MPI_Aint extent; MPI_Status status; int i, j, k, src, dst, rank, num_procs, send_offset, recv_offset; int mask, relative_rank, curr_size, recv_size = 0, send_size, nbytes; int scatter_size, tree_root, relative_dst, dst_tree_root; int my_tree_root, offset, tmp_mask, num_procs_completed; int tag = COLL_TAG_BCAST; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); extent = smpi_datatype_get_extent(data_type); nbytes = extent * count; scatter_size = (nbytes + num_procs - 1) / num_procs; // ceiling division curr_size = (rank == root) ? nbytes : 0; // root starts with all the data relative_rank = (rank >= root) ? rank - root : rank - root + num_procs; mask = 0x1; while (mask < num_procs) { if (relative_rank & mask) { src = rank - mask; if (src < 0) src += num_procs; recv_size = nbytes - relative_rank * scatter_size; // recv_size is larger than what might actually be sent by the // sender. We don't need compute the exact value because MPI // allows you to post a larger recv. if (recv_size <= 0) curr_size = 0; // this process doesn't receive any data // because of uneven division else { smpi_mpi_recv((char *)buff + relative_rank * scatter_size, recv_size, MPI_BYTE, src, tag, comm, &status); curr_size = smpi_mpi_get_count(&status, MPI_BYTE); } break; } mask <<= 1; } // This process is responsible for all processes that have bits // set from the LSB upto (but not including) mask. Because of // the "not including", we start by shifting mask back down // one. mask >>= 1; while (mask > 0) { if (relative_rank + mask < num_procs) { send_size = curr_size - scatter_size * mask; // mask is also the size of this process's subtree if (send_size > 0) { dst = rank + mask; if (dst >= num_procs) dst -= num_procs; smpi_mpi_send((char *)buff + scatter_size * (relative_rank + mask), send_size, MPI_BYTE, dst, tag, comm); curr_size -= send_size; } } mask >>= 1; } // done scatter now do allgather mask = 0x1; i = 0; while (mask < num_procs) { relative_dst = relative_rank ^ mask; dst = (relative_dst + root) % num_procs; /* find offset into send and recv buffers. zero out the least significant "i" bits of relative_rank and relative_dst to find root of src and dst subtrees. Use ranks of roots as index to send from and recv into buffer */ dst_tree_root = relative_dst >> i; dst_tree_root <<= i; my_tree_root = relative_rank >> i; my_tree_root <<= i; send_offset = my_tree_root * scatter_size; recv_offset = dst_tree_root * scatter_size; if (relative_dst < num_procs) { smpi_mpi_sendrecv((char *)buff + send_offset, curr_size, MPI_BYTE, dst, tag, (char *)buff + recv_offset, scatter_size * mask, MPI_BYTE, dst, tag, comm, &status); recv_size = smpi_mpi_get_count(&status, MPI_BYTE); curr_size += recv_size; } /* if some processes in this process's subtree in this step did not have any destination process to communicate with because of non-power-of-two, we need to send them the data that they would normally have received from those processes. That is, the haves in this subtree must send to the havenots. We use a logarithmic recursive-halfing algorithm for this. */ if (dst_tree_root + mask > num_procs) { num_procs_completed = num_procs - my_tree_root - mask; /* num_procs_completed is the number of processes in this subtree that have all the data. Send data to others in a tree fashion. First find root of current tree that is being divided into two. k is the number of least-significant bits in this process's rank that must be zeroed out to find the rank of the root */ j = mask; k = 0; while (j) { j >>= 1; k++; } k--; offset = scatter_size * (my_tree_root + mask); tmp_mask = mask >> 1; while (tmp_mask) { relative_dst = relative_rank ^ tmp_mask; dst = (relative_dst + root) % num_procs; tree_root = relative_rank >> k; tree_root <<= k; /* send only if this proc has data and destination doesn't have data. */ if ((relative_dst > relative_rank) && (relative_rank < tree_root + num_procs_completed) && (relative_dst >= tree_root + num_procs_completed)) { smpi_mpi_send((char *)buff + offset, recv_size, MPI_BYTE, dst, tag, comm); /* recv_size was set in the previous receive. that's the amount of data to be sent now. */ } /* recv only if this proc. doesn't have data and sender has data */ else if ((relative_dst < relative_rank) && (relative_dst < tree_root + num_procs_completed) && (relative_rank >= tree_root + num_procs_completed)) { smpi_mpi_recv((char *)buff + offset, scatter_size * num_procs_completed, MPI_BYTE, dst, tag, comm, &status); /* num_procs_completed is also equal to the no. of processes whose data we don't have */ recv_size = smpi_mpi_get_count(&status, MPI_BYTE); curr_size += recv_size; } tmp_mask >>= 1; k--; } } mask <<= 1; i++; } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/bcast-flattree.c000644 001750 001750 00000002215 13031031272 023007 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" int smpi_coll_tuned_bcast_flattree(void *buff, int count, MPI_Datatype data_type, int root, MPI_Comm comm) { MPI_Request *req_ptr; MPI_Request *reqs; int i, rank, num_procs; int tag = COLL_TAG_BCAST; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); if (rank != root) { smpi_mpi_recv(buff, count, data_type, root, tag, comm, MPI_STATUS_IGNORE); } else { reqs = (MPI_Request *) xbt_malloc((num_procs - 1) * sizeof(MPI_Request)); req_ptr = reqs; // Root sends data to all others for (i = 0; i < num_procs; i++) { if (i == rank) continue; *(req_ptr++) = smpi_mpi_isend(buff, count, data_type, i, tag, comm); } // wait on all requests smpi_mpi_waitall(num_procs - 1, reqs, MPI_STATUSES_IGNORE); free(reqs); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/reduce-mvapich-two-level.c000644 001750 001750 00000026670 13031031272 024732 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2009 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * * Additional copyrights may follow */ /* -*- Mode: C; c-basic-offset:4 ; -*- */ /* Copyright (c) 2001-2014, The Ohio State University. All rights * reserved. * * This file is part of the MVAPICH2 software package developed by the * team members of The Ohio State University's Network-Based Computing * Laboratory (NBCL), headed by Professor Dhabaleswar K. (DK) Panda. * * For detailed copyright and licensing information, please refer to the * copyright file COPYRIGHT in the top level MVAPICH2 directory. */ /* * * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ #include "colls_private.h" #define MV2_INTRA_SHMEM_REDUCE_MSG 2048 #define mv2_g_shmem_coll_max_msg_size (1 << 17) #define SHMEM_COLL_BLOCK_SIZE (local_size * mv2_g_shmem_coll_max_msg_size) #define mv2_use_knomial_reduce 1 #define MPIR_Reduce_inter_knomial_wrapper_MV2 smpi_coll_tuned_reduce_mvapich2_knomial #define MPIR_Reduce_intra_knomial_wrapper_MV2 smpi_coll_tuned_reduce_mvapich2_knomial #define MPIR_Reduce_binomial_MV2 smpi_coll_tuned_reduce_binomial #define MPIR_Reduce_redscat_gather_MV2 smpi_coll_tuned_reduce_scatter_gather #define MPIR_Reduce_shmem_MV2 smpi_coll_tuned_reduce_ompi_basic_linear extern int (*MV2_Reduce_function)( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm_ptr); extern int (*MV2_Reduce_intra_function)( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm_ptr); /*Fn pointers for collectives */ static int (*reduce_fn)(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm); int smpi_coll_tuned_reduce_mvapich2_two_level( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { int mpi_errno = MPI_SUCCESS; int my_rank, total_size, local_rank, local_size; int leader_comm_rank = -1, leader_comm_size = 0; MPI_Comm shmem_comm, leader_comm; int leader_root, leader_of_root; void *in_buf = NULL, *out_buf = NULL, *tmp_buf = NULL; MPI_Aint true_lb, true_extent, extent; int is_commutative = 0, stride = 0; int intra_node_root=0; //if not set (use of the algo directly, without mvapich2 selector) if(MV2_Reduce_function==NULL) MV2_Reduce_function=smpi_coll_tuned_reduce_mpich; if(MV2_Reduce_intra_function==NULL) MV2_Reduce_intra_function=smpi_coll_tuned_reduce_mpich; if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } my_rank = smpi_comm_rank(comm); total_size = smpi_comm_size(comm); shmem_comm = smpi_comm_get_intra_comm(comm); local_rank = smpi_comm_rank(shmem_comm); local_size = smpi_comm_size(shmem_comm); leader_comm = smpi_comm_get_leaders_comm(comm); int* leaders_map = smpi_comm_get_leaders_map(comm); leader_of_root = smpi_group_rank(smpi_comm_group(comm),leaders_map[root]); leader_root = smpi_group_rank(smpi_comm_group(leader_comm),leaders_map[root]); is_commutative=smpi_op_is_commute(op); smpi_datatype_extent(datatype, &true_lb, &true_extent); extent =smpi_datatype_get_extent(datatype); stride = count * MAX(extent, true_extent); if (local_size == total_size) { /* First handle the case where there is only one node */ if (stride <= MV2_INTRA_SHMEM_REDUCE_MSG && is_commutative == 1) { if (local_rank == 0 ) { tmp_buf=(void *)smpi_get_tmp_sendbuffer( count * (MAX(extent, true_extent))); tmp_buf = (void *) ((char *) tmp_buf - true_lb); } if (sendbuf != MPI_IN_PLACE) { in_buf = (void *)sendbuf; } else { in_buf = recvbuf; } if (local_rank == 0) { if( my_rank != root) { out_buf = tmp_buf; } else { out_buf = recvbuf; if(in_buf == out_buf) { in_buf = MPI_IN_PLACE; out_buf = recvbuf; } } } else { in_buf = (void *)sendbuf; out_buf = NULL; } if (count * (MAX(extent, true_extent)) < SHMEM_COLL_BLOCK_SIZE) { mpi_errno = MPIR_Reduce_shmem_MV2(in_buf, out_buf, count, datatype, op, 0, shmem_comm); } else { mpi_errno = MPIR_Reduce_intra_knomial_wrapper_MV2(in_buf, out_buf, count, datatype, op, 0, shmem_comm); } if (local_rank == 0 && root != my_rank) { smpi_mpi_send(out_buf, count, datatype, root, COLL_TAG_REDUCE+1, comm); } if ((local_rank != 0) && (root == my_rank)) { smpi_mpi_recv(recvbuf, count, datatype, leader_of_root, COLL_TAG_REDUCE+1, comm, MPI_STATUS_IGNORE); } } else { if(mv2_use_knomial_reduce == 1) { reduce_fn = &MPIR_Reduce_intra_knomial_wrapper_MV2; } else { reduce_fn = &MPIR_Reduce_binomial_MV2; } mpi_errno = reduce_fn(sendbuf, recvbuf, count, datatype, op, root, comm); } /* We are done */ if(tmp_buf!=NULL) smpi_free_tmp_buffer((void *) ((char *) tmp_buf + true_lb)); goto fn_exit; } if (local_rank == 0) { leader_comm = smpi_comm_get_leaders_comm(comm); if(leader_comm==MPI_COMM_NULL){ leader_comm = MPI_COMM_WORLD; } leader_comm_size = smpi_comm_size(leader_comm); leader_comm_rank = smpi_comm_rank(leader_comm); tmp_buf=(void *)smpi_get_tmp_sendbuffer(count * (MAX(extent, true_extent))); tmp_buf = (void *) ((char *) tmp_buf - true_lb); } if (sendbuf != MPI_IN_PLACE) { in_buf = (void *)sendbuf; } else { in_buf = recvbuf; } if (local_rank == 0) { out_buf = tmp_buf; } else { out_buf = NULL; } if(local_size > 1) { /* Lets do the intra-node reduce operations, if we have more than one * process in the node */ /*Fix the input and outbuf buffers for the intra-node reduce. *Node leaders will have the reduced data in tmp_buf after *this step*/ if (MV2_Reduce_intra_function == & MPIR_Reduce_shmem_MV2) { if (is_commutative == 1 && (count * (MAX(extent, true_extent)) < SHMEM_COLL_BLOCK_SIZE)) { mpi_errno = MV2_Reduce_intra_function(in_buf, out_buf, count, datatype, op, intra_node_root, shmem_comm); } else { mpi_errno = MPIR_Reduce_intra_knomial_wrapper_MV2(in_buf, out_buf, count, datatype, op, intra_node_root, shmem_comm); } } else { mpi_errno = MV2_Reduce_intra_function(in_buf, out_buf, count, datatype, op, intra_node_root, shmem_comm); } } else { smpi_free_tmp_buffer((void *) ((char *) tmp_buf + true_lb)); tmp_buf = in_buf; } /* Now work on the inter-leader phase. Data is in tmp_buf */ if (local_rank == 0 && leader_comm_size > 1) { /*The leader of root will have the global reduced data in tmp_buf or recv_buf at the end of the reduce */ if (leader_comm_rank == leader_root) { if (my_rank == root) { /* I am the root of the leader-comm, and the * root of the reduce op. So, I will write the * final result directly into my recvbuf */ if(tmp_buf != recvbuf) { in_buf = tmp_buf; out_buf = recvbuf; } else { in_buf = (char *)smpi_get_tmp_sendbuffer(count* smpi_datatype_get_extent(datatype)); smpi_datatype_copy(tmp_buf, count, datatype, in_buf, count, datatype); //in_buf = MPI_IN_PLACE; out_buf = recvbuf; } } else { in_buf = (char *)smpi_get_tmp_sendbuffer(count* smpi_datatype_get_extent(datatype)); smpi_datatype_copy(tmp_buf, count, datatype, in_buf, count, datatype); //in_buf = MPI_IN_PLACE; out_buf = tmp_buf; } } else { in_buf = tmp_buf; out_buf = NULL; } /* inter-leader communication */ mpi_errno = MV2_Reduce_function(in_buf, out_buf, count, datatype, op, leader_root, leader_comm); } if (local_size > 1) { /* Send the message to the root if the leader is not the * root of the reduce operation. The reduced data is in tmp_buf */ if ((local_rank == 0) && (root != my_rank) && (leader_root == leader_comm_rank)) { smpi_mpi_send(tmp_buf, count, datatype, root, COLL_TAG_REDUCE+1, comm); } if ((local_rank != 0) && (root == my_rank)) { smpi_mpi_recv(recvbuf, count, datatype, leader_of_root, COLL_TAG_REDUCE+1, comm, MPI_STATUS_IGNORE); } smpi_free_tmp_buffer((void *) ((char *) tmp_buf + true_lb)); if (leader_comm_rank == leader_root) { if (my_rank != root || (my_rank == root && tmp_buf == recvbuf)) { smpi_free_tmp_buffer(in_buf); } } } fn_exit: return mpi_errno; } SimGrid-3.14.159/src/smpi/colls/allreduce-rab1.c000644 001750 001750 00000006360 13031031271 022676 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" //#include // NP pow of 2 for now int smpi_coll_tuned_allreduce_rab1(void *sbuff, void *rbuff, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { MPI_Status status; MPI_Aint extent; int tag = COLL_TAG_ALLREDUCE, send_size, newcnt, share; unsigned int pof2 = 1, mask; int send_idx, recv_idx, dst, send_cnt, recv_cnt; void *recv, *tmp_buf; unsigned int rank = smpi_comm_rank(comm); unsigned int nprocs = smpi_comm_size(comm); if((nprocs&(nprocs-1))) THROWF(arg_error,0, "allreduce rab1 algorithm can't be used with non power of two number of processes ! "); extent = smpi_datatype_get_extent(dtype); pof2 = 1; while (pof2 <= nprocs) pof2 <<= 1; pof2 >>= 1; send_idx = recv_idx = 0; // uneven count if ((count % nprocs)) { send_size = (count + nprocs) / nprocs; newcnt = send_size * nprocs; recv = (void *) smpi_get_tmp_recvbuffer(extent * newcnt); tmp_buf = (void *) smpi_get_tmp_sendbuffer(extent * newcnt); memcpy(recv, sbuff, extent * count); mask = pof2 / 2; share = newcnt / pof2; while (mask > 0) { dst = rank ^ mask; send_cnt = recv_cnt = newcnt / (pof2 / mask); if (rank < dst) send_idx = recv_idx + (mask * share); else recv_idx = send_idx + (mask * share); smpi_mpi_sendrecv((char *) recv + send_idx * extent, send_cnt, dtype, dst, tag, tmp_buf, recv_cnt, dtype, dst, tag, comm, &status); smpi_op_apply(op, tmp_buf, (char *) recv + recv_idx * extent, &recv_cnt, &dtype); // update send_idx for next iteration send_idx = recv_idx; mask >>= 1; } memcpy(tmp_buf, (char *) recv + recv_idx * extent, recv_cnt * extent); mpi_coll_allgather_fun(tmp_buf, recv_cnt, dtype, recv, recv_cnt, dtype, comm); memcpy(rbuff, recv, count * extent); smpi_free_tmp_buffer(recv); smpi_free_tmp_buffer(tmp_buf); } else { tmp_buf = (void *) smpi_get_tmp_sendbuffer(extent * count); memcpy(rbuff, sbuff, count * extent); mask = pof2 / 2; share = count / pof2; while (mask > 0) { dst = rank ^ mask; send_cnt = recv_cnt = count / (pof2 / mask); if (rank < dst) send_idx = recv_idx + (mask * share); else recv_idx = send_idx + (mask * share); smpi_mpi_sendrecv((char *) rbuff + send_idx * extent, send_cnt, dtype, dst, tag, tmp_buf, recv_cnt, dtype, dst, tag, comm, &status); smpi_op_apply(op, tmp_buf, (char *) rbuff + recv_idx * extent, &recv_cnt, &dtype); // update send_idx for next iteration send_idx = recv_idx; mask >>= 1; } memcpy(tmp_buf, (char *) rbuff + recv_idx * extent, recv_cnt * extent); mpi_coll_allgather_fun(tmp_buf, recv_cnt, dtype, rbuff, recv_cnt, dtype, comm); smpi_free_tmp_buffer(tmp_buf); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allreduce-smp-binomial-pipeline.c000644 001750 001750 00000015774 13031031271 026254 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /* IMPLEMENTED BY PITCH PATARASUK Non-topoloty-specific (however, number of cores/node need to be changed) all-reduce operation designed for smp clusters It uses 2-layer communication: binomial for both intra-communication inter-communication The communication are done in a pipeline fashion */ /* this is a default segment size for pipelining, but it is typically passed as a command line argument */ int allreduce_smp_binomial_pipeline_segment_size = 4096; /* ** NOTE ** This code is modified from allreduce-smp-binomial.c by wrapping the code with pipeline effect as follow for (loop over pipelength) { smp-binomial main code; } */ /* ** NOTE ** Use -DMPICH2 if this code does not compile. MPICH1 code also work on MPICH2 on our cluster and the performance are similar. This code assume commutative and associative reduce operator (MPI_SUM, MPI_MAX, etc). */ /* This fucntion performs all-reduce operation as follow. ** in a pipeline fashion ** 1) binomial_tree reduce inside each SMP node 2) binomial_tree reduce intra-communication between root of each SMP node 3) binomial_tree bcast intra-communication between root of each SMP node 4) binomial_tree bcast inside each SMP node */ int smpi_coll_tuned_allreduce_smp_binomial_pipeline(void *send_buf, void *recv_buf, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { int comm_size, rank; void *tmp_buf; int tag = COLL_TAG_ALLREDUCE; int mask, src, dst; MPI_Status status; if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } int num_core=1; if (smpi_comm_is_uniform(comm)){ num_core = smpi_comm_size(smpi_comm_get_intra_comm(comm)); } comm_size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); MPI_Aint extent; extent = smpi_datatype_get_extent(dtype); tmp_buf = (void *) smpi_get_tmp_sendbuffer(count * extent); int intra_rank, inter_rank; intra_rank = rank % num_core; inter_rank = rank / num_core; int phase; int send_offset; int recv_offset; int pcount = allreduce_smp_binomial_pipeline_segment_size; if (pcount > count) { pcount = count; } /* size of processes participate in intra communications => should be equal to number of machines */ int inter_comm_size = (comm_size + num_core - 1) / num_core; /* copy input buffer to output buffer */ smpi_mpi_sendrecv(send_buf, count, dtype, rank, tag, recv_buf, count, dtype, rank, tag, comm, &status); /* compute pipe length */ int pipelength; pipelength = count / pcount; /* pipelining over pipelength (+3 is because we have 4 stages: reduce-intra, reduce-inter, bcast-inter, bcast-intra */ for (phase = 0; phase < pipelength + 3; phase++) { /* start binomial reduce intra communication inside each SMP node */ if (phase < pipelength) { mask = 1; while (mask < num_core) { if ((mask & intra_rank) == 0) { src = (inter_rank * num_core) + (intra_rank | mask); if (src < comm_size) { recv_offset = phase * pcount * extent; smpi_mpi_recv(tmp_buf, pcount, dtype, src, tag, comm, &status); smpi_op_apply(op, tmp_buf, (char *)recv_buf + recv_offset, &pcount, &dtype); } } else { send_offset = phase * pcount * extent; dst = (inter_rank * num_core) + (intra_rank & (~mask)); smpi_mpi_send((char *)recv_buf + send_offset, pcount, dtype, dst, tag, comm); break; } mask <<= 1; } } /* start binomial reduce inter-communication between each SMP nodes: each node only have one process that can communicate to other nodes */ if ((phase > 0) && (phase < (pipelength + 1))) { if (intra_rank == 0) { mask = 1; while (mask < inter_comm_size) { if ((mask & inter_rank) == 0) { src = (inter_rank | mask) * num_core; if (src < comm_size) { recv_offset = (phase - 1) * pcount * extent; smpi_mpi_recv(tmp_buf, pcount, dtype, src, tag, comm, &status); smpi_op_apply(op, tmp_buf, (char *)recv_buf + recv_offset, &pcount, &dtype); } } else { dst = (inter_rank & (~mask)) * num_core; send_offset = (phase - 1) * pcount * extent; smpi_mpi_send((char *)recv_buf + send_offset, pcount, dtype, dst, tag, comm); break; } mask <<= 1; } } } /* start binomial broadcast inter-communication between each SMP nodes: each node only have one process that can communicate to other nodes */ if ((phase > 1) && (phase < (pipelength + 2))) { if (intra_rank == 0) { mask = 1; while (mask < inter_comm_size) { if (inter_rank & mask) { src = (inter_rank - mask) * num_core; recv_offset = (phase - 2) * pcount * extent; smpi_mpi_recv((char *)recv_buf + recv_offset, pcount, dtype, src, tag, comm, &status); break; } mask <<= 1; } mask >>= 1; while (mask > 0) { if (inter_rank < inter_comm_size) { dst = (inter_rank + mask) * num_core; if (dst < comm_size) { //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); send_offset = (phase - 2) * pcount * extent; smpi_mpi_send((char *)recv_buf + send_offset, pcount, dtype, dst, tag, comm); } } mask >>= 1; } } } /* start binomial broadcast intra-communication inside each SMP nodes */ if (phase > 2) { int num_core_in_current_smp = num_core; if (inter_rank == (inter_comm_size - 1)) { num_core_in_current_smp = comm_size - (inter_rank * num_core); } mask = 1; while (mask < num_core_in_current_smp) { if (intra_rank & mask) { src = (inter_rank * num_core) + (intra_rank - mask); recv_offset = (phase - 3) * pcount * extent; smpi_mpi_recv((char *)recv_buf + recv_offset, pcount, dtype, src, tag, comm, &status); break; } mask <<= 1; } mask >>= 1; while (mask > 0) { dst = (inter_rank * num_core) + (intra_rank + mask); if (dst < comm_size) { send_offset = (phase - 3) * pcount * extent; smpi_mpi_send((char *)recv_buf + send_offset, pcount, dtype, dst, tag, comm); } mask >>= 1; } } } // for phase smpi_free_tmp_buffer(tmp_buf); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allgather-ompi-neighborexchange.c000644 001750 001750 00000015322 13031031271 026314 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * ompi_coll_tuned_allgather_intra_neighborexchange * * Function: allgather using N/2 steps (O(N)) * Accepts: Same arguments as MPI_Allgather * Returns: MPI_SUCCESS or error code * * Description: Neighbor Exchange algorithm for allgather. * Described by Chen et.al. in * "Performance Evaluation of Allgather Algorithms on * Terascale Linux Cluster with Fast Ethernet", * Proceedings of the Eighth International Conference on * High-Performance Computing inn Asia-Pacific Region * (HPCASIA'05), 2005 * * Rank r exchanges message with one of its neighbors and * forwards the data further in the next step. * * No additional memory requirements. * * Limitations: Algorithm works only on even number of processes. * For odd number of processes we switch to ring algorithm. * * Example on 6 nodes: * Initial state * # 0 1 2 3 4 5 * [0] [ ] [ ] [ ] [ ] [ ] * [ ] [1] [ ] [ ] [ ] [ ] * [ ] [ ] [2] [ ] [ ] [ ] * [ ] [ ] [ ] [3] [ ] [ ] * [ ] [ ] [ ] [ ] [4] [ ] * [ ] [ ] [ ] [ ] [ ] [5] * Step 0: * # 0 1 2 3 4 5 * [0] [0] [ ] [ ] [ ] [ ] * [1] [1] [ ] [ ] [ ] [ ] * [ ] [ ] [2] [2] [ ] [ ] * [ ] [ ] [3] [3] [ ] [ ] * [ ] [ ] [ ] [ ] [4] [4] * [ ] [ ] [ ] [ ] [5] [5] * Step 1: * # 0 1 2 3 4 5 * [0] [0] [0] [ ] [ ] [0] * [1] [1] [1] [ ] [ ] [1] * [ ] [2] [2] [2] [2] [ ] * [ ] [3] [3] [3] [3] [ ] * [4] [ ] [ ] [4] [4] [4] * [5] [ ] [ ] [5] [5] [5] * Step 2: * # 0 1 2 3 4 5 * [0] [0] [0] [0] [0] [0] * [1] [1] [1] [1] [1] [1] * [2] [2] [2] [2] [2] [2] * [3] [3] [3] [3] [3] [3] * [4] [4] [4] [4] [4] [4] * [5] [5] [5] [5] [5] [5] */ #include "colls_private.h" int smpi_coll_tuned_allgather_ompi_neighborexchange(void *sbuf, int scount, MPI_Datatype sdtype, void* rbuf, int rcount, MPI_Datatype rdtype, MPI_Comm comm ) { int line = -1; int rank, size; int neighbor[2], offset_at_step[2], recv_data_from[2], send_data_from; int i, even_rank; int err = 0; ptrdiff_t slb, rlb, sext, rext; char *tmpsend = NULL, *tmprecv = NULL; size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); if (size % 2) { XBT_DEBUG( "coll:tuned:allgather_intra_neighborexchange WARNING: odd size %d, switching to ring algorithm", size); return smpi_coll_tuned_allgather_ring(sbuf, scount, sdtype, rbuf, rcount, rdtype, comm); } XBT_DEBUG( "coll:tuned:allgather_intra_neighborexchange rank %d", rank); err = smpi_datatype_extent (sdtype, &slb, &sext); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } err = smpi_datatype_extent (rdtype, &rlb, &rext); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } /* Initialization step: - if send buffer is not MPI_IN_PLACE, copy send buffer to appropriate block of receive buffer */ tmprecv = (char*) rbuf + rank * rcount * rext; if (MPI_IN_PLACE != sbuf) { tmpsend = (char*) sbuf; smpi_datatype_copy (tmpsend, scount, sdtype, tmprecv, rcount, rdtype); } /* Determine neighbors, order in which blocks will arrive, etc. */ even_rank = !(rank % 2); if (even_rank) { neighbor[0] = (rank + 1) % size; neighbor[1] = (rank - 1 + size) % size; recv_data_from[0] = rank; recv_data_from[1] = rank; offset_at_step[0] = (+2); offset_at_step[1] = (-2); } else { neighbor[0] = (rank - 1 + size) % size; neighbor[1] = (rank + 1) % size; recv_data_from[0] = neighbor[0]; recv_data_from[1] = neighbor[0]; offset_at_step[0] = (-2); offset_at_step[1] = (+2); } /* Communication loop: - First step is special: exchange a single block with neighbor[0]. - Rest of the steps: update recv_data_from according to offset, and exchange two blocks with appropriate neighbor. the send location becomes previous receve location. */ tmprecv = (char*)rbuf + neighbor[0] * rcount * rext; tmpsend = (char*)rbuf + rank * rcount * rext; /* Sendreceive */ smpi_mpi_sendrecv(tmpsend, rcount, rdtype, neighbor[0], COLL_TAG_ALLGATHER, tmprecv, rcount, rdtype, neighbor[0], COLL_TAG_ALLGATHER, comm, MPI_STATUS_IGNORE); /* Determine initial sending location */ if (even_rank) { send_data_from = rank; } else { send_data_from = recv_data_from[0]; } for (i = 1; i < (size / 2); i++) { const int i_parity = i % 2; recv_data_from[i_parity] = (recv_data_from[i_parity] + offset_at_step[i_parity] + size) % size; tmprecv = (char*)rbuf + recv_data_from[i_parity] * rcount * rext; tmpsend = (char*)rbuf + send_data_from * rcount * rext; /* Sendreceive */ smpi_mpi_sendrecv(tmpsend, 2 * rcount, rdtype, neighbor[i_parity], COLL_TAG_ALLGATHER, tmprecv, 2 * rcount, rdtype, neighbor[i_parity], COLL_TAG_ALLGATHER, comm, MPI_STATUS_IGNORE); send_data_from = recv_data_from[i_parity]; } return MPI_SUCCESS; err_hndl: XBT_DEBUG( "%s:%4d\tError occurred %d, rank %2d", __FILE__, line, err, rank); return err; } SimGrid-3.14.159/src/smpi/colls/allreduce-rab2.c000644 001750 001750 00000005151 13031031271 022674 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" //#include // this requires that count >= NP int smpi_coll_tuned_allreduce_rab2(void *sbuff, void *rbuff, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { MPI_Aint s_extent; int i, rank, nprocs; int nbytes, send_size, s_offset, r_offset; void *recv, *send, *tmp; /* #ifdef MPICH2_REDUCTION MPI_User_function * uop = MPIR_Op_table[op % 16 - 1]; #else MPI_User_function *uop; struct MPIR_OP *op_ptr; op_ptr = MPIR_ToPointer(op); uop = op_ptr->op; #endif */ rank = smpi_comm_rank(comm); nprocs = smpi_comm_size(comm); s_extent = smpi_datatype_get_extent(dtype); // uneven count if (count % nprocs) { if (count < nprocs) send_size = nprocs; else send_size = (count + nprocs) / nprocs; nbytes = send_size * s_extent; send = (void *) smpi_get_tmp_sendbuffer(s_extent * send_size * nprocs); recv = (void *) smpi_get_tmp_recvbuffer(s_extent * send_size * nprocs); tmp = (void *) smpi_get_tmp_sendbuffer(nbytes); memcpy(send, sbuff, s_extent * count); mpi_coll_alltoall_fun(send, send_size, dtype, recv, send_size, dtype, comm); memcpy(tmp, recv, nbytes); for (i = 1, s_offset = nbytes; i < nprocs; i++, s_offset = i * nbytes) smpi_op_apply(op, (char *) recv + s_offset, tmp, &send_size, &dtype); mpi_coll_allgather_fun(tmp, send_size, dtype, recv, send_size, dtype, comm); memcpy(rbuff, recv, count * s_extent); smpi_free_tmp_buffer(recv); smpi_free_tmp_buffer(tmp); smpi_free_tmp_buffer(send); } else { send = sbuff; send_size = count / nprocs; nbytes = send_size * s_extent; r_offset = rank * nbytes; recv = (void *) smpi_get_tmp_recvbuffer(s_extent * send_size * nprocs); mpi_coll_alltoall_fun(send, send_size, dtype, recv, send_size, dtype, comm); memcpy((char *) rbuff + r_offset, recv, nbytes); for (i = 1, s_offset = nbytes; i < nprocs; i++, s_offset = i * nbytes) smpi_op_apply(op, (char *) recv + s_offset, (char *) rbuff + r_offset, &send_size, &dtype); mpi_coll_allgather_fun((char *) rbuff + r_offset, send_size, dtype, rbuff, send_size, dtype, comm); smpi_free_tmp_buffer(recv); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/alltoall-pair-mpi-barrier.c000644 001750 001750 00000004256 13031031271 025061 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** * Function: alltoall_pair_mpi_barrier * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works when P is power of two. In each phase of P - 1 phases, nodes in pair communicate their data. MPI barriers are inserted between each two phases. * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_alltoall_pair_mpi_barrier(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Status s; MPI_Aint send_chunk, recv_chunk; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALL; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "alltoall pair algorithm can't be used with non power of two number of processes ! "); send_chunk = smpi_datatype_get_extent(send_type); recv_chunk = smpi_datatype_get_extent(recv_type); send_chunk *= send_count; recv_chunk *= recv_count; for (i = 0; i < num_procs; i++) { src = dst = rank ^ i; mpi_coll_barrier_fun(comm); smpi_mpi_sendrecv(send_ptr + dst * send_chunk, send_count, send_type, dst, tag, recv_ptr + src * recv_chunk, recv_count, recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/alltoall-3dmesh.c000644 001750 001750 00000013023 13031031271 023072 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" #include /***************************************************************************** * Function: alltoall_3dmesh_shoot * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function realizes the alltoall operation using the 3dmesh algorithm. It actually performs allgather operation in x dimension, y dimension, then in z dimension. Each node then extracts the needed data. The communication in all dimension is simple. * Auther: Ahmad Faraj ****************************************************************************/ static int alltoall_check_is_3dmesh(int num, int *i, int *j, int *k) { int x, max = num / 3; x = cbrt(num); *i = *j = *k = 0; while (x <= max) { if ((num % (x * x)) == 0) { *i = *j = x; *k = num / (x * x); return 1; } x++; } return 0; } int smpi_coll_tuned_alltoall_3dmesh(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Request *reqs, *req_ptr; MPI_Aint extent; MPI_Status status, *statuses; int i, j, src, dst, rank, num_procs, num_reqs, X, Y, Z, block_size, count; int my_z, two_dsize, my_row_base, my_col_base, my_z_base, src_row_base; int src_z_base, send_offset, recv_offset, tag = COLL_TAG_ALLTOALL; char *tmp_buff1, *tmp_buff2; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); extent = smpi_datatype_get_extent(send_type); if (!alltoall_check_is_3dmesh(num_procs, &X, &Y, &Z)) return MPI_ERR_OTHER; num_reqs = X; if (Y > X) num_reqs = Y; if (Z > Y) num_reqs = Z; two_dsize = X * Y; my_z = rank / two_dsize; my_row_base = (rank / X) * X; my_col_base = (rank % Y) + (my_z * two_dsize); my_z_base = my_z * two_dsize; block_size = extent * send_count; tmp_buff1 = (char *) smpi_get_tmp_sendbuffer(block_size * num_procs * two_dsize); tmp_buff2 = (char *) smpi_get_tmp_recvbuffer(block_size * two_dsize); statuses = (MPI_Status *) xbt_malloc(num_reqs * sizeof(MPI_Status)); reqs = (MPI_Request *) xbt_malloc(num_reqs * sizeof(MPI_Request)); req_ptr = reqs; recv_offset = (rank % two_dsize) * block_size * num_procs; smpi_mpi_sendrecv(send_buff, send_count * num_procs, send_type, rank, tag, tmp_buff1 + recv_offset, num_procs * recv_count, recv_type, rank, tag, comm, &status); count = send_count * num_procs; for (i = 0; i < Y; i++) { src = i + my_row_base; if (src == rank) continue; recv_offset = (src % two_dsize) * block_size * num_procs; *(req_ptr++) = smpi_mpi_irecv(tmp_buff1 + recv_offset, count, recv_type, src, tag, comm); } for (i = 0; i < Y; i++) { dst = i + my_row_base; if (dst == rank) continue; smpi_mpi_send(send_buff, count, send_type, dst, tag, comm); } smpi_mpi_waitall(Y - 1, reqs, statuses); req_ptr = reqs; for (i = 0; i < X; i++) { src = (i * Y + my_col_base); if (src == rank) continue; src_row_base = (src / X) * X; recv_offset = (src_row_base % two_dsize) * block_size * num_procs; *(req_ptr++) = smpi_mpi_irecv(tmp_buff1 + recv_offset, recv_count * num_procs * Y, recv_type, src, tag, comm); } send_offset = (my_row_base % two_dsize) * block_size * num_procs; for (i = 0; i < X; i++) { dst = (i * Y + my_col_base); if (dst == rank) continue; smpi_mpi_send(tmp_buff1 + send_offset, send_count * num_procs * Y, send_type, dst, tag, comm); } smpi_mpi_waitall(X - 1, reqs, statuses); req_ptr = reqs; for (i = 0; i < two_dsize; i++) { send_offset = (rank * block_size) + (i * block_size * num_procs); recv_offset = (my_z_base * block_size) + (i * block_size); smpi_mpi_sendrecv(tmp_buff1 + send_offset, send_count, send_type, rank, tag, (char *) recv_buff + recv_offset, recv_count, recv_type, rank, tag, comm, &status); } for (i = 1; i < Z; i++) { src = (rank + i * two_dsize) % num_procs; src_z_base = (src / two_dsize) * two_dsize; recv_offset = (src_z_base * block_size); *(req_ptr++) = smpi_mpi_irecv((char *) recv_buff + recv_offset, recv_count * two_dsize, recv_type, src, tag, comm); } for (i = 1; i < Z; i++) { dst = (rank + i * two_dsize) % num_procs; recv_offset = 0; for (j = 0; j < two_dsize; j++) { send_offset = (dst + j * num_procs) * block_size; smpi_mpi_sendrecv(tmp_buff1 + send_offset, send_count, send_type, rank, tag, tmp_buff2 + recv_offset, recv_count, recv_type, rank, tag, comm, &status); recv_offset += block_size; } smpi_mpi_send(tmp_buff2, send_count * two_dsize, send_type, dst, tag, comm); } smpi_mpi_waitall(Z - 1, reqs, statuses); free(reqs); free(statuses); smpi_free_tmp_buffer(tmp_buff1); smpi_free_tmp_buffer(tmp_buff2); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allreduce-redbcast.c000644 001750 001750 00000001215 13031031271 023632 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" int smpi_coll_tuned_allreduce_redbcast(void *buf, void *buf2, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { mpi_coll_reduce_fun(buf, buf2, count, datatype, op, 0, comm); mpi_coll_bcast_fun(buf2, count, datatype, 0, comm); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/alltoallv-pair.c000644 001750 001750 00000004032 13031031271 023030 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** * Function: alltoall_pair * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works when P is power of two. In each phase of P - 1 phases, nodes in pair communicate their data. * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_alltoallv_pair(void *send_buff, int *send_counts, int *send_disps, MPI_Datatype send_type, void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint send_chunk, recv_chunk; MPI_Status s; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALLV; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); if((num_procs&(num_procs-1))) THROWF(arg_error,0, "alltoallv pair algorithm can't be used with non power of two number of processes ! "); send_chunk = smpi_datatype_get_extent(send_type); recv_chunk = smpi_datatype_get_extent(recv_type); for (i = 0; i < num_procs; i++) { src = dst = rank ^ i; smpi_mpi_sendrecv(send_ptr + send_disps[dst] * send_chunk, send_counts[dst], send_type, dst, tag, recv_ptr + recv_disps[src] * recv_chunk, recv_counts[src], recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/reduce-scatter-gather.c000644 001750 001750 00000030310 13031031272 024266 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /* reduce Author: MPICH */ int smpi_coll_tuned_reduce_scatter_gather(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) { MPI_Status status; int comm_size, rank, pof2, rem, newrank; int mask, *cnts, *disps, i, j, send_idx = 0; int recv_idx, last_idx = 0, newdst; int dst, send_cnt, recv_cnt, newroot, newdst_tree_root; int newroot_tree_root, new_count; int tag = COLL_TAG_REDUCE,temporary_buffer=0; void *send_ptr, *recv_ptr, *tmp_buf; cnts = NULL; disps = NULL; MPI_Aint extent; if (count == 0) return 0; rank = smpi_comm_rank(comm); comm_size = smpi_comm_size(comm); extent = smpi_datatype_get_extent(datatype); /* If I'm not the root, then my recvbuf may not be valid, therefore I have to allocate a temporary one */ if (rank != root && !recvbuf) { temporary_buffer=1; recvbuf = (void *)smpi_get_tmp_recvbuffer(count * extent); } /* find nearest power-of-two less than or equal to comm_size */ pof2 = 1; while (pof2 <= comm_size) pof2 <<= 1; pof2 >>= 1; if (count < comm_size) { new_count = comm_size; send_ptr = (void *) smpi_get_tmp_sendbuffer(new_count * extent); recv_ptr = (void *) smpi_get_tmp_recvbuffer(new_count * extent); tmp_buf = (void *) smpi_get_tmp_sendbuffer(new_count * extent); memcpy(send_ptr, sendbuf != MPI_IN_PLACE ? sendbuf : recvbuf, extent * count); //if ((rank != root)) smpi_mpi_sendrecv(send_ptr, new_count, datatype, rank, tag, recv_ptr, new_count, datatype, rank, tag, comm, &status); rem = comm_size - pof2; if (rank < 2 * rem) { if (rank % 2 != 0) { /* odd */ smpi_mpi_send(recv_ptr, new_count, datatype, rank - 1, tag, comm); newrank = -1; } else { smpi_mpi_recv(tmp_buf, count, datatype, rank + 1, tag, comm, &status); smpi_op_apply(op, tmp_buf, recv_ptr, &new_count, &datatype); newrank = rank / 2; } } else /* rank >= 2*rem */ newrank = rank - rem; cnts = (int *) xbt_malloc(pof2 * sizeof(int)); disps = (int *) xbt_malloc(pof2 * sizeof(int)); if (newrank != -1) { for (i = 0; i < (pof2 - 1); i++) cnts[i] = new_count / pof2; cnts[pof2 - 1] = new_count - (new_count / pof2) * (pof2 - 1); disps[0] = 0; for (i = 1; i < pof2; i++) disps[i] = disps[i - 1] + cnts[i - 1]; mask = 0x1; send_idx = recv_idx = 0; last_idx = pof2; while (mask < pof2) { newdst = newrank ^ mask; /* find real rank of dest */ dst = (newdst < rem) ? newdst * 2 : newdst + rem; send_cnt = recv_cnt = 0; if (newrank < newdst) { send_idx = recv_idx + pof2 / (mask * 2); for (i = send_idx; i < last_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < send_idx; i++) recv_cnt += cnts[i]; } else { recv_idx = send_idx + pof2 / (mask * 2); for (i = send_idx; i < recv_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < last_idx; i++) recv_cnt += cnts[i]; } /* Send data from recvbuf. Recv into tmp_buf */ smpi_mpi_sendrecv((char *) recv_ptr + disps[send_idx] * extent, send_cnt, datatype, dst, tag, (char *) tmp_buf + disps[recv_idx] * extent, recv_cnt, datatype, dst, tag, comm, &status); /* tmp_buf contains data received in this step. recvbuf contains data accumulated so far */ smpi_op_apply(op, (char *) tmp_buf + disps[recv_idx] * extent, (char *) recv_ptr + disps[recv_idx] * extent, &recv_cnt, &datatype); /* update send_idx for next iteration */ send_idx = recv_idx; mask <<= 1; if (mask < pof2) last_idx = recv_idx + pof2 / mask; } } /* now do the gather to root */ if (root < 2 * rem) { if (root % 2 != 0) { if (rank == root) { /* recv */ for (i = 0; i < (pof2 - 1); i++) cnts[i] = new_count / pof2; cnts[pof2 - 1] = new_count - (new_count / pof2) * (pof2 - 1); disps[0] = 0; for (i = 1; i < pof2; i++) disps[i] = disps[i - 1] + cnts[i - 1]; smpi_mpi_recv(recv_ptr, cnts[0], datatype, 0, tag, comm, &status); newrank = 0; send_idx = 0; last_idx = 2; } else if (newrank == 0) { smpi_mpi_send(recv_ptr, cnts[0], datatype, root, tag, comm); newrank = -1; } newroot = 0; } else newroot = root / 2; } else newroot = root - rem; if (newrank != -1) { j = 0; mask = 0x1; while (mask < pof2) { mask <<= 1; j++; } mask >>= 1; j--; while (mask > 0) { newdst = newrank ^ mask; /* find real rank of dest */ dst = (newdst < rem) ? newdst * 2 : newdst + rem; if ((newdst == 0) && (root < 2 * rem) && (root % 2 != 0)) dst = root; newdst_tree_root = newdst >> j; newdst_tree_root <<= j; newroot_tree_root = newroot >> j; newroot_tree_root <<= j; send_cnt = recv_cnt = 0; if (newrank < newdst) { /* update last_idx except on first iteration */ if (mask != pof2 / 2) last_idx = last_idx + pof2 / (mask * 2); recv_idx = send_idx + pof2 / (mask * 2); for (i = send_idx; i < recv_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < last_idx; i++) recv_cnt += cnts[i]; } else { recv_idx = send_idx - pof2 / (mask * 2); for (i = send_idx; i < last_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < send_idx; i++) recv_cnt += cnts[i]; } if (newdst_tree_root == newroot_tree_root) { smpi_mpi_send((char *) recv_ptr + disps[send_idx] * extent, send_cnt, datatype, dst, tag, comm); break; } else { smpi_mpi_recv((char *) recv_ptr + disps[recv_idx] * extent, recv_cnt, datatype, dst, tag, comm, &status); } if (newrank > newdst) send_idx = recv_idx; mask >>= 1; j--; } } memcpy(recvbuf, recv_ptr, extent * count); smpi_free_tmp_buffer(send_ptr); smpi_free_tmp_buffer(recv_ptr); } else /* (count >= comm_size) */ { tmp_buf = (void *) smpi_get_tmp_sendbuffer(count * extent); //if ((rank != root)) smpi_mpi_sendrecv(sendbuf != MPI_IN_PLACE ? sendbuf : recvbuf, count, datatype, rank, tag, recvbuf, count, datatype, rank, tag, comm, &status); rem = comm_size - pof2; if (rank < 2 * rem) { if (rank % 2 != 0) { /* odd */ smpi_mpi_send(recvbuf, count, datatype, rank - 1, tag, comm); newrank = -1; } else { smpi_mpi_recv(tmp_buf, count, datatype, rank + 1, tag, comm, &status); smpi_op_apply(op, tmp_buf, recvbuf, &count, &datatype); newrank = rank / 2; } } else /* rank >= 2*rem */ newrank = rank - rem; cnts = (int *) xbt_malloc(pof2 * sizeof(int)); disps = (int *) xbt_malloc(pof2 * sizeof(int)); if (newrank != -1) { for (i = 0; i < (pof2 - 1); i++) cnts[i] = count / pof2; cnts[pof2 - 1] = count - (count / pof2) * (pof2 - 1); disps[0] = 0; for (i = 1; i < pof2; i++) disps[i] = disps[i - 1] + cnts[i - 1]; mask = 0x1; send_idx = recv_idx = 0; last_idx = pof2; while (mask < pof2) { newdst = newrank ^ mask; /* find real rank of dest */ dst = (newdst < rem) ? newdst * 2 : newdst + rem; send_cnt = recv_cnt = 0; if (newrank < newdst) { send_idx = recv_idx + pof2 / (mask * 2); for (i = send_idx; i < last_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < send_idx; i++) recv_cnt += cnts[i]; } else { recv_idx = send_idx + pof2 / (mask * 2); for (i = send_idx; i < recv_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < last_idx; i++) recv_cnt += cnts[i]; } /* Send data from recvbuf. Recv into tmp_buf */ smpi_mpi_sendrecv((char *) recvbuf + disps[send_idx] * extent, send_cnt, datatype, dst, tag, (char *) tmp_buf + disps[recv_idx] * extent, recv_cnt, datatype, dst, tag, comm, &status); /* tmp_buf contains data received in this step. recvbuf contains data accumulated so far */ smpi_op_apply(op, (char *) tmp_buf + disps[recv_idx] * extent, (char *) recvbuf + disps[recv_idx] * extent, &recv_cnt, &datatype); /* update send_idx for next iteration */ send_idx = recv_idx; mask <<= 1; if (mask < pof2) last_idx = recv_idx + pof2 / mask; } } /* now do the gather to root */ if (root < 2 * rem) { if (root % 2 != 0) { if (rank == root) { /* recv */ for (i = 0; i < (pof2 - 1); i++) cnts[i] = count / pof2; cnts[pof2 - 1] = count - (count / pof2) * (pof2 - 1); disps[0] = 0; for (i = 1; i < pof2; i++) disps[i] = disps[i - 1] + cnts[i - 1]; smpi_mpi_recv(recvbuf, cnts[0], datatype, 0, tag, comm, &status); newrank = 0; send_idx = 0; last_idx = 2; } else if (newrank == 0) { smpi_mpi_send(recvbuf, cnts[0], datatype, root, tag, comm); newrank = -1; } newroot = 0; } else newroot = root / 2; } else newroot = root - rem; if (newrank != -1) { j = 0; mask = 0x1; while (mask < pof2) { mask <<= 1; j++; } mask >>= 1; j--; while (mask > 0) { newdst = newrank ^ mask; /* find real rank of dest */ dst = (newdst < rem) ? newdst * 2 : newdst + rem; if ((newdst == 0) && (root < 2 * rem) && (root % 2 != 0)) dst = root; newdst_tree_root = newdst >> j; newdst_tree_root <<= j; newroot_tree_root = newroot >> j; newroot_tree_root <<= j; send_cnt = recv_cnt = 0; if (newrank < newdst) { /* update last_idx except on first iteration */ if (mask != pof2 / 2) last_idx = last_idx + pof2 / (mask * 2); recv_idx = send_idx + pof2 / (mask * 2); for (i = send_idx; i < recv_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < last_idx; i++) recv_cnt += cnts[i]; } else { recv_idx = send_idx - pof2 / (mask * 2); for (i = send_idx; i < last_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < send_idx; i++) recv_cnt += cnts[i]; } if (newdst_tree_root == newroot_tree_root) { smpi_mpi_send((char *) recvbuf + disps[send_idx] * extent, send_cnt, datatype, dst, tag, comm); break; } else { smpi_mpi_recv((char *) recvbuf + disps[recv_idx] * extent, recv_cnt, datatype, dst, tag, comm, &status); } if (newrank > newdst) send_idx = recv_idx; mask >>= 1; j--; } } } if (tmp_buf) smpi_free_tmp_buffer(tmp_buf); if(temporary_buffer==1) smpi_free_tmp_buffer(recvbuf); if (cnts) free(cnts); if (disps) free(disps); return 0; } SimGrid-3.14.159/src/smpi/colls/alltoallv-ring-light-barrier.c000644 001750 001750 00000005260 13031031271 025571 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** * Function: alltoall_ring_light_barrier * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works in P - 1 steps. In step i, node j - i -> j -> j + i. Light barriers are inserted between communications in different phases. * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_alltoallv_ring_light_barrier(void *send_buff, int *send_counts, int *send_disps, MPI_Datatype send_type, void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint send_chunk, recv_chunk; MPI_Status s; int i, src, dst, rank, num_procs, next_dst, next_src; int tag = COLL_TAG_ALLTOALLV; char send_sync = 'a', recv_sync = 'b'; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); send_chunk = smpi_datatype_get_extent(send_type); recv_chunk = smpi_datatype_get_extent(recv_type); smpi_mpi_sendrecv(send_ptr + send_disps[rank] * send_chunk, send_counts[rank], send_type, rank, tag, recv_ptr + recv_disps[rank] * recv_chunk, recv_counts[rank], recv_type, rank, tag, comm, &s); for (i = 1; i < num_procs; i++) { src = (rank - i + num_procs) % num_procs; dst = (rank + i) % num_procs; smpi_mpi_sendrecv(send_ptr + send_disps[dst] * send_chunk, send_counts[dst], send_type, dst, tag, recv_ptr + recv_disps[src] * recv_chunk, recv_counts[src], recv_type, src, tag, comm, &s); if ((i + 1) < num_procs) { next_src = (rank - (i + 1) + num_procs) % num_procs; next_dst = (rank + (i + 1) + num_procs) % num_procs; smpi_mpi_sendrecv(&send_sync, 1, MPI_CHAR, next_src, tag, &recv_sync, 1, MPI_CHAR, next_dst, tag, comm, &s); } } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/colls.h000644 001750 001750 00000035157 13031031273 021244 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_COLLS_H #define SMPI_COLLS_H #include #include "smpi/mpi.h" #include "src/smpi/private.h" #include "xbt/ex.h" #include "xbt.h" SG_BEGIN_DECL() #define COLL_DESCRIPTION(cat, ret, args, name) \ {# name,\ # cat " " # name " collective",\ (void*)smpi_coll_tuned_ ## cat ## _ ## name} #define COLL_PROTO(cat, ret, args, name) \ ret smpi_coll_tuned_ ## cat ## _ ## name(COLL_UNPAREN args); #define COLL_UNPAREN(...) __VA_ARGS__ #define COLL_APPLY(action, sig, name) action(sig, name) #define COLL_COMMA , #define COLL_NOsep #define COLL_NOTHING(...) /************* * GATHER * *************/ #define COLL_GATHER_SIG gather, int, \ (void *send_buff, int send_count, MPI_Datatype send_type, \ void *recv_buff, int recv_count, MPI_Datatype recv_type, \ int root, MPI_Comm comm) #define COLL_GATHERS(action, COLL_sep) \ COLL_APPLY(action, COLL_GATHER_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_GATHER_SIG, ompi_basic_linear) COLL_sep \ COLL_APPLY(action, COLL_GATHER_SIG, ompi_binomial) COLL_sep \ COLL_APPLY(action, COLL_GATHER_SIG, ompi_linear_sync) COLL_sep \ COLL_APPLY(action, COLL_GATHER_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_GATHER_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_GATHER_SIG, mvapich2_two_level) COLL_sep \ COLL_APPLY(action, COLL_GATHER_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_GATHER_SIG, automatic) COLL_GATHERS(COLL_PROTO, COLL_NOsep) /************* * ALLGATHER * *************/ #define COLL_ALLGATHER_SIG allgather, int, \ (void *send_buff, int send_count, MPI_Datatype send_type, \ void *recv_buff, int recv_count, MPI_Datatype recv_type, \ MPI_Comm comm) #define COLL_ALLGATHERS(action, COLL_sep) \ COLL_APPLY(action, COLL_ALLGATHER_SIG, 2dmesh) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, 3dmesh) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, bruck) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, GB) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, loosely_lr) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, NTSLR) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, NTSLR_NB) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, pair) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, rdb) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, rhv) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, ring) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, SMP_NTS) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, smp_simple) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, spreading_simple) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, ompi_neighborexchange) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, mvapich2_smp) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHER_SIG, automatic) COLL_ALLGATHERS(COLL_PROTO, COLL_NOsep) /************** * ALLGATHERV * **************/ #define COLL_ALLGATHERV_SIG allgatherv, int, \ (void *send_buff, int send_count, MPI_Datatype send_type, \ void *recv_buff, int *recv_count, int *recv_disps, \ MPI_Datatype recv_type, MPI_Comm comm) #define COLL_ALLGATHERVS(action, COLL_sep) \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, GB) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, pair) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, ring) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, ompi_neighborexchange) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, ompi_bruck) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, mpich_rdb) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, mpich_ring) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_ALLGATHERV_SIG, automatic) COLL_ALLGATHERVS(COLL_PROTO, COLL_NOsep) /************* * ALLREDUCE * *************/ #define COLL_ALLREDUCE_SIG allreduce, int, \ (void *sbuf, void *rbuf, int rcount, \ MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) #define COLL_ALLREDUCES(action, COLL_sep) \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, lr) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, rab1) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, rab2) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, rab_rdb) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, rdb) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, smp_binomial) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, smp_binomial_pipeline) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, smp_rdb) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, smp_rsag) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, smp_rsag_lr) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, smp_rsag_rab) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, redbcast) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, ompi_ring_segmented) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, mvapich2_rs) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, mvapich2_two_level) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, rab) COLL_sep \ COLL_APPLY(action, COLL_ALLREDUCE_SIG, automatic) COLL_ALLREDUCES(COLL_PROTO, COLL_NOsep) /************ * ALLTOALL * ************/ #define COLL_ALLTOALL_SIG alltoall, int, \ (void *send_buff, int send_count, MPI_Datatype send_type, \ void *recv_buff, int recv_count, MPI_Datatype recv_type, \ MPI_Comm comm) #define COLL_ALLTOALLS(action, COLL_sep) \ COLL_APPLY(action, COLL_ALLTOALL_SIG, 2dmesh) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, 3dmesh) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, basic_linear) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, bruck) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, pair) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, pair_rma) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, pair_light_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, pair_mpi_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, pair_one_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, rdb) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, ring) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, ring_light_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, ring_mpi_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, ring_one_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, mvapich2_scatter_dest) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALL_SIG, automatic) COLL_ALLTOALLS(COLL_PROTO, COLL_NOsep) /************* * ALLTOALLV * *************/ #define COLL_ALLTOALLV_SIG alltoallv, int, \ (void *send_buff, int *send_counts, int *send_disps, MPI_Datatype send_type, \ void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, \ MPI_Comm comm) #define COLL_ALLTOALLVS(action, COLL_sep) \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, bruck) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, pair) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, pair_light_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, pair_mpi_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, pair_one_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, ring) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, ring_light_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, ring_mpi_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, ring_one_barrier) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, ompi_basic_linear) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_ALLTOALLV_SIG, automatic) COLL_ALLTOALLVS(COLL_PROTO, COLL_NOsep) /********* * BCAST * *********/ #define COLL_BCAST_SIG bcast, int, \ (void *buf, int count, MPI_Datatype datatype, \ int root, MPI_Comm comm) #define COLL_BCASTS(action, COLL_sep) \ COLL_APPLY(action, COLL_BCAST_SIG, arrival_pattern_aware) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, arrival_pattern_aware_wait) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, arrival_scatter) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, binomial_tree) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, flattree) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, flattree_pipeline) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, NTSB) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, NTSL) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, NTSL_Isend) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, scatter_LR_allgather) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, scatter_rdb_allgather) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, SMP_binary) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, SMP_binomial) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, SMP_linear) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, ompi_split_bintree) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, ompi_pipeline) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, mvapich2_inter_node) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, mvapich2_intra_node) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, mvapich2_knomial_intra_node) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_BCAST_SIG, automatic) COLL_BCASTS(COLL_PROTO, COLL_NOsep) /********** * REDUCE * **********/ #define COLL_REDUCE_SIG reduce, int, \ (void *buf, void *rbuf, int count, MPI_Datatype datatype, \ MPI_Op op, int root, MPI_Comm comm) #define COLL_REDUCES(action, COLL_sep) \ COLL_APPLY(action, COLL_REDUCE_SIG, arrival_pattern_aware) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, binomial) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, flat_tree) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, NTSL) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, scatter_gather) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, ompi_chain) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, ompi_pipeline) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, ompi_basic_linear) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, ompi_in_order_binary) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, ompi_binary) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, ompi_binomial) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, mvapich2_knomial) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, mvapich2_two_level) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, rab) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SIG, automatic) COLL_REDUCES(COLL_PROTO, COLL_NOsep) /************* * REDUCE_SCATTER * *************/ #define COLL_REDUCE_SCATTER_SIG reduce_scatter, int, \ (void *sbuf, void *rbuf, int *rcounts,\ MPI_Datatype dtype,MPI_Op op,MPI_Comm comm) #define COLL_REDUCE_SCATTERS(action, COLL_sep) \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, ompi_basic_recursivehalving) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, ompi_ring) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, mpich_pair) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, mpich_rdb) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, mpich_noncomm) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_REDUCE_SCATTER_SIG, automatic) COLL_REDUCE_SCATTERS(COLL_PROTO, COLL_NOsep) /************* * SCATTER * *************/ #define COLL_SCATTER_SIG scatter, int, \ (void *sendbuf, int sendcount, MPI_Datatype sendtype,\ void *recvbuf, int recvcount, MPI_Datatype recvtype,\ int root, MPI_Comm comm) #define COLL_SCATTERS(action, COLL_sep) \ COLL_APPLY(action, COLL_SCATTER_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_SCATTER_SIG, ompi_basic_linear) COLL_sep \ COLL_APPLY(action, COLL_SCATTER_SIG, ompi_binomial) COLL_sep \ COLL_APPLY(action, COLL_SCATTER_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_SCATTER_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_SCATTER_SIG, mvapich2_two_level_binomial) COLL_sep \ COLL_APPLY(action, COLL_SCATTER_SIG, mvapich2_two_level_direct) COLL_sep \ COLL_APPLY(action, COLL_SCATTER_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_SCATTER_SIG, automatic) COLL_SCATTERS(COLL_PROTO, COLL_NOsep) /************* * SCATTER * *************/ #define COLL_BARRIER_SIG barrier, int, \ (MPI_Comm comm) #define COLL_BARRIERS(action, COLL_sep) \ COLL_APPLY(action, COLL_BARRIER_SIG, ompi) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, ompi_basic_linear) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, ompi_two_procs) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, ompi_tree) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, ompi_bruck) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, ompi_recursivedoubling) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, ompi_doublering) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, mpich) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, mvapich2_pair) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, mvapich2) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, impi) COLL_sep \ COLL_APPLY(action, COLL_BARRIER_SIG, automatic) COLL_BARRIERS(COLL_PROTO, COLL_NOsep) SG_END_DECL() #endif SimGrid-3.14.159/src/smpi/colls/alltoallv-ring-mpi-barrier.c000644 001750 001750 00000004057 13031031271 025252 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** * Function: alltoall_ring_mpi_barrier * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works in P - 1 steps. In step i, node j - i -> j -> j + i. MPI barriers are added between each two phases. * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_alltoallv_ring_mpi_barrier(void *send_buff, int *send_counts, int *send_disps, MPI_Datatype send_type, void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Status s; MPI_Aint send_chunk, recv_chunk; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALLV; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); send_chunk = smpi_datatype_get_extent(send_type); recv_chunk = smpi_datatype_get_extent(recv_type); for (i = 0; i < num_procs; i++) { src = (rank - i + num_procs) % num_procs; dst = (rank + i) % num_procs; smpi_mpi_barrier(comm); smpi_mpi_sendrecv(send_ptr + send_disps[dst] * send_chunk, send_counts[dst], send_type, dst, tag, recv_ptr + recv_disps[src] * recv_chunk, recv_counts[src], recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/scatter-ompi.c000644 001750 001750 00000016575 13031031272 022534 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2006 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * * Additional copyrights may follow */ #include "colls_private.h" #include "coll_tuned_topo.h" int smpi_coll_tuned_scatter_ompi_binomial(void *sbuf, int scount, MPI_Datatype sdtype, void *rbuf, int rcount, MPI_Datatype rdtype, int root, MPI_Comm comm ) { int line = -1; int i; int rank; int vrank; int size; int total_send = 0; char *ptmp = NULL; char *tempbuf = NULL; int err; ompi_coll_tree_t* bmtree; MPI_Status status; MPI_Aint sextent, slb, strue_lb, strue_extent; MPI_Aint rextent, rlb, rtrue_lb, rtrue_extent; size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); XBT_DEBUG( "smpi_coll_tuned_scatter_ompi_binomial rank %d", rank); /* create the binomial tree */ // COLL_TUNED_UPDATE_IN_ORDER_BMTREE( comm, tuned_module, root ); bmtree = ompi_coll_tuned_topo_build_in_order_bmtree( comm, root);//ompi_ data->cached_in_order_bmtree; smpi_datatype_extent(sdtype, &slb, &sextent); smpi_datatype_extent(sdtype, &strue_lb, &strue_extent); smpi_datatype_extent(rdtype, &rlb, &rextent); smpi_datatype_extent(rdtype, &rtrue_lb, &rtrue_extent); vrank = (rank - root + size) % size; if (rank == root) { if (0 == root) { /* root on 0, just use the send buffer */ ptmp = (char *) sbuf; if (rbuf != MPI_IN_PLACE) { /* local copy to rbuf */ err = smpi_datatype_copy(sbuf, scount, sdtype, rbuf, rcount, rdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } } } else { /* root is not on 0, allocate temp buffer for send */ tempbuf = (char *) smpi_get_tmp_sendbuffer(strue_extent + (scount*size - 1) * sextent); if (NULL == tempbuf) { err = MPI_ERR_OTHER; line = __LINE__; goto err_hndl; } ptmp = tempbuf - slb; /* and rotate data so they will eventually in the right place */ err = smpi_datatype_copy((char *) sbuf + sextent*root*scount, scount*(size-root), sdtype, ptmp, scount*(size-root), sdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } err = smpi_datatype_copy((char*)sbuf, scount*root, sdtype, ptmp + sextent*scount*(size - root), scount*root, sdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } if (rbuf != MPI_IN_PLACE) { /* local copy to rbuf */ err = smpi_datatype_copy(ptmp, scount, sdtype, rbuf, rcount, rdtype); if (MPI_SUCCESS != err) { line = __LINE__; goto err_hndl; } } } total_send = scount; } else if (!(vrank % 2)) { /* non-root, non-leaf nodes, allocate temp buffer for recv * the most we need is rcount*size/2 */ tempbuf = (char *) smpi_get_tmp_recvbuffer(rtrue_extent + (rcount*size - 1) * rextent); if (NULL == tempbuf) { err= MPI_ERR_OTHER; line = __LINE__; goto err_hndl; } ptmp = tempbuf - rlb; sdtype = rdtype; scount = rcount; sextent = rextent; total_send = scount; } else { /* leaf nodes, just use rbuf */ ptmp = (char *) rbuf; } if (!(vrank % 2)) { if (rank != root) { /* recv from parent on non-root */ smpi_mpi_recv(ptmp, rcount*size, rdtype, bmtree->tree_prev, COLL_TAG_SCATTER, comm, &status); /* local copy to rbuf */ smpi_datatype_copy(ptmp, scount, sdtype, rbuf, rcount, rdtype); } /* send to children on all non-leaf */ for (i = 0; i < bmtree->tree_nextsize; i++) { int mycount = 0, vkid; /* figure out how much data I have to send to this child */ vkid = (bmtree->tree_next[i] - root + size) % size; mycount = vkid - vrank; if (mycount > (size - vkid)) mycount = size - vkid; mycount *= scount; smpi_mpi_send(ptmp + total_send*sextent, mycount, sdtype, bmtree->tree_next[i], COLL_TAG_SCATTER, comm); total_send += mycount; } } else { /* recv from parent on leaf nodes */ smpi_mpi_recv(ptmp, rcount, rdtype, bmtree->tree_prev, COLL_TAG_SCATTER, comm, &status); } if (NULL != tempbuf) smpi_free_tmp_buffer(tempbuf); //!FIXME : store the tree, as done in ompi, instead of calculating it each time ? xbt_free(bmtree); return MPI_SUCCESS; err_hndl: if (NULL != tempbuf) free(tempbuf); XBT_DEBUG( "%s:%4d\tError occurred %d, rank %2d", __FILE__, line, err, rank); return err; } /* * Linear functions are copied from the BASIC coll module * they do not segment the message and are simple implementations * but for some small number of nodes and/or small data sizes they * are just as fast as tuned/tree based segmenting operations * and as such may be selected by the decision functions * These are copied into this module due to the way we select modules * in V1. i.e. in V2 we will handle this differently and so will not * have to duplicate code. * JPG following the examples from other coll_tuned implementations. Dec06. */ /* copied function (with appropriate renaming) starts here */ /* * scatter_intra * * Function: - basic scatter operation * Accepts: - same arguments as MPI_Scatter() * Returns: - MPI_SUCCESS or error code */ int smpi_coll_tuned_scatter_ompi_basic_linear(void *sbuf, int scount, MPI_Datatype sdtype, void *rbuf, int rcount, MPI_Datatype rdtype, int root, MPI_Comm comm ) { int i, rank, size, err; char *ptmp; ptrdiff_t lb, incr; /* Initialize */ rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); /* If not root, receive data. */ if (rank != root) { smpi_mpi_recv(rbuf, rcount, rdtype, root, COLL_TAG_SCATTER, comm, MPI_STATUS_IGNORE); return MPI_SUCCESS; } /* I am the root, loop sending data. */ err = smpi_datatype_extent(sdtype, &lb, &incr); if (MPI_SUCCESS != err) { return MPI_ERR_OTHER; } incr *= scount; for (i = 0, ptmp = (char *) sbuf; i < size; ++i, ptmp += incr) { /* simple optimization */ if (i == rank) { if (MPI_IN_PLACE != rbuf) { err = smpi_datatype_copy(ptmp, scount, sdtype, rbuf, rcount, rdtype); } } else { smpi_mpi_send(ptmp, scount, sdtype, i, COLL_TAG_SCATTER, comm); } if (MPI_SUCCESS != err) { return err; } } /* All done */ return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allreduce-smp-rsag-lr.c000644 001750 001750 00000017011 13031031271 024210 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" //#include /* This fucntion performs all-reduce operation as follow. 1) binomial_tree reduce inside each SMP node 2) reduce-scatter -inter between root of each SMP node 3) allgather - inter between root of each SMP node 4) binomial_tree bcast inside each SMP node */ int smpi_coll_tuned_allreduce_smp_rsag_lr(void *send_buf, void *recv_buf, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { int comm_size, rank; void *tmp_buf; int tag = COLL_TAG_ALLREDUCE; int mask, src, dst; MPI_Status status; if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ smpi_comm_init_smp(comm); } int num_core=1; if (smpi_comm_is_uniform(comm)){ num_core = smpi_comm_size(smpi_comm_get_intra_comm(comm)); } /* #ifdef MPICH2_REDUCTION MPI_User_function * uop = MPIR_Op_table[op % 16 - 1]; #else MPI_User_function *uop; struct MPIR_OP *op_ptr; op_ptr = MPIR_ToPointer(op); uop = op_ptr->op; #endif */ comm_size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); MPI_Aint extent; extent = smpi_datatype_get_extent(dtype); tmp_buf = (void *) smpi_get_tmp_sendbuffer(count * extent); int intra_rank, inter_rank; intra_rank = rank % num_core; inter_rank = rank / num_core; //printf("node %d intra_rank = %d, inter_rank = %d\n", rank, intra_rank, inter_rank); int inter_comm_size = (comm_size + num_core - 1) / num_core; if (!rank) { //printf("intra com size = %d\n",num_core); //printf("inter com size = %d\n",inter_comm_size); } smpi_mpi_sendrecv(send_buf, count, dtype, rank, tag, recv_buf, count, dtype, rank, tag, comm, &status); // SMP_binomial_reduce mask = 1; while (mask < num_core) { if ((mask & intra_rank) == 0) { src = (inter_rank * num_core) + (intra_rank | mask); // if (src < ((inter_rank + 1) * num_core)) { if (src < comm_size) { smpi_mpi_recv(tmp_buf, count, dtype, src, tag, comm, &status); smpi_op_apply(op, tmp_buf, recv_buf, &count, &dtype); //printf("Node %d recv from node %d when mask is %d\n", rank, src, mask); } } else { dst = (inter_rank * num_core) + (intra_rank & (~mask)); smpi_mpi_send(recv_buf, count, dtype, dst, tag, comm); //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); break; } mask <<= 1; } // INTER: reduce-scatter if (intra_rank == 0) { int send_offset, recv_offset; int send_count, recv_count; int curr_size = count / inter_comm_size; int curr_remainder = count % inter_comm_size; int to = ((inter_rank + 1) % inter_comm_size) * num_core; int from = ((inter_rank + inter_comm_size - 1) % inter_comm_size) * num_core; int i; //printf("node %d to %d from %d\n",rank,to,from); /* last segment may have a larger size since it also include the remainder */ int last_segment_ptr = (inter_comm_size - 1) * (count / inter_comm_size) * extent; for (i = 0; i < (inter_comm_size - 1); i++) { send_offset = ((inter_rank - 1 - i + inter_comm_size) % inter_comm_size) * curr_size * extent; recv_offset = ((inter_rank - 2 - i + inter_comm_size) % inter_comm_size) * curr_size * extent; /* adjust size */ if (send_offset != last_segment_ptr) send_count = curr_size; else send_count = curr_size + curr_remainder; if (recv_offset != last_segment_ptr) recv_count = curr_size; else recv_count = curr_size + curr_remainder; smpi_mpi_sendrecv((char *) recv_buf + send_offset, send_count, dtype, to, tag + i, tmp_buf, recv_count, dtype, from, tag + i, comm, &status); // result is in rbuf smpi_op_apply(op, tmp_buf, (char *) recv_buf + recv_offset, &recv_count, &dtype); } // INTER: allgather for (i = 0; i < (inter_comm_size - 1); i++) { send_offset = ((inter_rank - i + inter_comm_size) % inter_comm_size) * curr_size * extent; recv_offset = ((inter_rank - 1 - i + inter_comm_size) % inter_comm_size) * curr_size * extent; /* adjust size */ if (send_offset != last_segment_ptr) send_count = curr_size; else send_count = curr_size + curr_remainder; if (recv_offset != last_segment_ptr) recv_count = curr_size; else recv_count = curr_size + curr_remainder; smpi_mpi_sendrecv((char *) recv_buf + send_offset, send_count, dtype, to, tag + i, (char *) recv_buf + recv_offset, recv_count, dtype, from, tag + i, comm, &status); } } // INTER_binomial_reduce // only root node for each SMP /* if (intra_rank == 0) { mask = 1; while (mask < inter_comm_size) { if ((mask & inter_rank) == 0) { src = (inter_rank | mask) * num_core; if (src < comm_size) { smpi_mpi_recv(tmp_buf, count, dtype, src, tag, comm, &status); (* uop) (tmp_buf, recv_buf, &count, &dtype); //printf("Node %d recv from node %d when mask is %d\n", rank, src, mask); } } else { dst = (inter_rank & (~mask)) * num_core; smpi_mpi_send(recv_buf, count, dtype, dst, tag, comm); //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); break; } mask <<=1; } } */ // INTER_binomial_bcast // if (intra_rank == 0) { // mask = 1; // while (mask < inter_comm_size) { // if (inter_rank & mask) { // src = (inter_rank - mask) * num_core; //printf("Node %d recv from node %d when mask is %d\n", rank, src, mask); // smpi_mpi_recv(recv_buf, count, dtype, src, tag, comm, &status); // break; // } // mask <<= 1; // } // // mask >>= 1; //printf("My rank = %d my mask = %d\n", rank,mask); // while (mask > 0) { // if (inter_rank < inter_comm_size) { // dst = (inter_rank + mask) * num_core; // if (dst < comm_size) { // //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); // smpi_mpi_send(recv_buf, count, dtype, dst, tag, comm); // } // } // mask >>= 1; // } // } // INTRA_binomial_bcast int num_core_in_current_smp = num_core; if (inter_rank == (inter_comm_size - 1)) { num_core_in_current_smp = comm_size - (inter_rank * num_core); } // printf("Node %d num_core = %d\n",rank, num_core_in_current_smp); mask = 1; while (mask < num_core_in_current_smp) { if (intra_rank & mask) { src = (inter_rank * num_core) + (intra_rank - mask); //printf("Node %d recv from node %d when mask is %d\n", rank, src, mask); smpi_mpi_recv(recv_buf, count, dtype, src, tag, comm, &status); break; } mask <<= 1; } mask >>= 1; //printf("My rank = %d my mask = %d\n", rank,mask); while (mask > 0) { dst = (inter_rank * num_core) + (intra_rank + mask); if (dst < comm_size) { //printf("Node %d send to node %d when mask is %d\n", rank, dst, mask); smpi_mpi_send(recv_buf, count, dtype, dst, tag, comm); } mask >>= 1; } smpi_free_tmp_buffer(tmp_buf); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/alltoall-ring-mpi-barrier.c000644 001750 001750 00000004040 13031031271 025054 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** * Function: alltoall_ring_mpi_barrier * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works in P - 1 steps. In step i, node j - i -> j -> j + i. MPI barriers are added between each two phases. * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_alltoall_ring_mpi_barrier(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Status s; MPI_Aint send_chunk, recv_chunk; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALL; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); send_chunk = smpi_datatype_get_extent(send_type); recv_chunk = smpi_datatype_get_extent(recv_type); send_chunk *= send_count; recv_chunk *= recv_count; for (i = 0; i < num_procs; i++) { src = (rank - i + num_procs) % num_procs; dst = (rank + i) % num_procs; mpi_coll_barrier_fun(comm); smpi_mpi_sendrecv(send_ptr + dst * send_chunk, send_count, send_type, dst, tag, recv_ptr + src * recv_chunk, recv_count, recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/alltoall-rdb.c000644 001750 001750 00000011655 13031031271 022467 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** * Function: alltoall_rdb * Return: int * inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function realizes the allgather operation using the recursive doubling algorithm. * Auther: MPICH / slightly modified by Ahmad Faraj. ****************************************************************************/ int smpi_coll_tuned_alltoall_rdb(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int recv_count, MPI_Datatype recv_type, MPI_Comm comm) { /* MPI variables */ MPI_Status status; MPI_Aint send_increment, recv_increment, extent; int dst_tree_root, rank_tree_root, send_offset, recv_offset; int rank, num_procs, j, k, dst, curr_size, max_size; int last_recv_count = 0, tmp_mask, tree_root, num_procs_completed; int tag = COLL_TAG_ALLTOALL, mask = 1, i = 0; char *tmp_buff; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; num_procs = smpi_comm_size(comm); rank = smpi_comm_rank(comm); send_increment = smpi_datatype_get_extent(send_type); recv_increment = smpi_datatype_get_extent(recv_type); extent = smpi_datatype_get_extent(recv_type); send_increment *= (send_count * num_procs); recv_increment *= (recv_count * num_procs); max_size = num_procs * recv_increment; tmp_buff = (char *) smpi_get_tmp_sendbuffer(max_size); curr_size = send_count * num_procs; smpi_mpi_sendrecv(send_ptr, curr_size, send_type, rank, tag, tmp_buff + (rank * recv_increment), curr_size, recv_type, rank, tag, comm, &status); while (mask < num_procs) { dst = rank ^ mask; dst_tree_root = dst >> i; dst_tree_root <<= i; rank_tree_root = rank >> i; rank_tree_root <<= i; send_offset = rank_tree_root * send_increment; recv_offset = dst_tree_root * recv_increment; if (dst < num_procs) { smpi_mpi_sendrecv(tmp_buff + send_offset, curr_size, send_type, dst, tag, tmp_buff + recv_offset, mask * recv_count * num_procs, recv_type, dst, tag, comm, &status); last_recv_count = smpi_mpi_get_count(&status, recv_type); curr_size += last_recv_count; } if (dst_tree_root + mask > num_procs) { num_procs_completed = num_procs - rank_tree_root - mask; /* num_procs_completed is the number of processes in this subtree that have all the data. Send data to others in a tree fashion. First find root of current tree that is being divided into two. k is the number of least-significant bits in this process's rank that must be zeroed out to find the rank of the root */ j = mask; k = 0; while (j) { j >>= 1; k++; } k--; tmp_mask = mask >> 1; while (tmp_mask) { dst = rank ^ tmp_mask; tree_root = rank >> k; tree_root <<= k; /* send only if this proc has data and destination doesn't have data. at any step, multiple processes can send if they have the data */ if ((dst > rank) && (rank < tree_root + num_procs_completed) && (dst >= tree_root + num_procs_completed)) { smpi_mpi_send(tmp_buff + dst_tree_root * send_increment, last_recv_count, send_type, dst, tag, comm); } /* recv only if this proc. doesn't have data and sender has data */ else if ((dst < rank) && (dst < tree_root + num_procs_completed) && (rank >= tree_root + num_procs_completed)) { smpi_mpi_recv(tmp_buff + dst_tree_root * send_increment, mask * num_procs * send_count, send_type, dst, tag, comm, &status); last_recv_count = smpi_mpi_get_count(&status, send_type); curr_size += last_recv_count; } tmp_mask >>= 1; k--; } } mask <<= 1; i++; } for (i = 0; i < num_procs; i++) smpi_mpi_sendrecv(tmp_buff + (rank + i * num_procs) * send_count * extent, send_count, send_type, rank, tag, recv_ptr + (i * recv_count * extent), recv_count, recv_type, rank, tag, comm, &status); smpi_free_tmp_buffer(tmp_buff); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allreduce-rab-rdb.c000644 001750 001750 00000014475 13031031271 023370 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" int smpi_coll_tuned_allreduce_rab_rdb(void *sbuff, void *rbuff, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { int tag = COLL_TAG_ALLREDUCE; unsigned int mask, pof2; int dst, newrank, rem, newdst, i, send_idx, recv_idx, last_idx, send_cnt, recv_cnt, *cnts, *disps; MPI_Aint extent; MPI_Status status; void *tmp_buf = NULL; unsigned int nprocs = smpi_comm_size(comm); unsigned int rank = smpi_comm_rank(comm); extent = smpi_datatype_get_extent(dtype); tmp_buf = (void *) smpi_get_tmp_sendbuffer(count * extent); smpi_datatype_copy(sbuff, count, dtype, rbuff, count, dtype); // find nearest power-of-two less than or equal to comm_size pof2 = 1; while (pof2 <= nprocs) pof2 <<= 1; pof2 >>= 1; rem = nprocs - pof2; // In the non-power-of-two case, all even-numbered // processes of rank < 2*rem send their data to // (rank+1). These even-numbered processes no longer // participate in the algorithm until the very end. The // remaining processes form a nice power-of-two. if (rank < 2 * rem) { // even if (rank % 2 == 0) { smpi_mpi_send(rbuff, count, dtype, rank + 1, tag, comm); // temporarily set the rank to -1 so that this // process does not pariticipate in recursive // doubling newrank = -1; } else // odd { smpi_mpi_recv(tmp_buf, count, dtype, rank - 1, tag, comm, &status); // do the reduction on received data. since the // ordering is right, it doesn't matter whether // the operation is commutative or not. smpi_op_apply(op, tmp_buf, rbuff, &count, &dtype); // change the rank newrank = rank / 2; } } else // rank >= 2 * rem newrank = rank - rem; // If op is user-defined or count is less than pof2, use // recursive doubling algorithm. Otherwise do a reduce-scatter // followed by allgather. (If op is user-defined, // derived datatypes are allowed and the user could pass basic // datatypes on one process and derived on another as long as // the type maps are the same. Breaking up derived // datatypes to do the reduce-scatter is tricky, therefore // using recursive doubling in that case.) if (newrank != -1) { // do a reduce-scatter followed by allgather. for the // reduce-scatter, calculate the count that each process receives // and the displacement within the buffer cnts = (int *) xbt_malloc(pof2 * sizeof(int)); disps = (int *) xbt_malloc(pof2 * sizeof(int)); for (i = 0; i < (pof2 - 1); i++) cnts[i] = count / pof2; cnts[pof2 - 1] = count - (count / pof2) * (pof2 - 1); disps[0] = 0; for (i = 1; i < pof2; i++) disps[i] = disps[i - 1] + cnts[i - 1]; mask = 0x1; send_idx = recv_idx = 0; last_idx = pof2; while (mask < pof2) { newdst = newrank ^ mask; // find real rank of dest dst = (newdst < rem) ? newdst * 2 + 1 : newdst + rem; send_cnt = recv_cnt = 0; if (newrank < newdst) { send_idx = recv_idx + pof2 / (mask * 2); for (i = send_idx; i < last_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < send_idx; i++) recv_cnt += cnts[i]; } else { recv_idx = send_idx + pof2 / (mask * 2); for (i = send_idx; i < recv_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < last_idx; i++) recv_cnt += cnts[i]; } // Send data from recvbuf. Recv into tmp_buf smpi_mpi_sendrecv((char *) rbuff + disps[send_idx] * extent, send_cnt, dtype, dst, tag, (char *) tmp_buf + disps[recv_idx] * extent, recv_cnt, dtype, dst, tag, comm, &status); // tmp_buf contains data received in this step. // recvbuf contains data accumulated so far // This algorithm is used only for predefined ops // and predefined ops are always commutative. smpi_op_apply(op, (char *) tmp_buf + disps[recv_idx] * extent, (char *) rbuff + disps[recv_idx] * extent, &recv_cnt, &dtype); // update send_idx for next iteration send_idx = recv_idx; mask <<= 1; // update last_idx, but not in last iteration because the value // is needed in the allgather step below. if (mask < pof2) last_idx = recv_idx + pof2 / mask; } // now do the allgather mask >>= 1; while (mask > 0) { newdst = newrank ^ mask; // find real rank of dest dst = (newdst < rem) ? newdst * 2 + 1 : newdst + rem; send_cnt = recv_cnt = 0; if (newrank < newdst) { // update last_idx except on first iteration if (mask != pof2 / 2) last_idx = last_idx + pof2 / (mask * 2); recv_idx = send_idx + pof2 / (mask * 2); for (i = send_idx; i < recv_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < last_idx; i++) recv_cnt += cnts[i]; } else { recv_idx = send_idx - pof2 / (mask * 2); for (i = send_idx; i < last_idx; i++) send_cnt += cnts[i]; for (i = recv_idx; i < send_idx; i++) recv_cnt += cnts[i]; } smpi_mpi_sendrecv((char *) rbuff + disps[send_idx] * extent, send_cnt, dtype, dst, tag, (char *) rbuff + disps[recv_idx] * extent, recv_cnt, dtype, dst, tag, comm, &status); if (newrank > newdst) send_idx = recv_idx; mask >>= 1; } free(cnts); free(disps); } // In the non-power-of-two case, all odd-numbered processes of // rank < 2 * rem send the result to (rank-1), the ranks who didn't // participate above. if (rank < 2 * rem) { if (rank % 2) // odd smpi_mpi_send(rbuff, count, dtype, rank - 1, tag, comm); else // even smpi_mpi_recv(rbuff, count, dtype, rank + 1, tag, comm, &status); } smpi_free_tmp_buffer(tmp_buf); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allgatherv-ring.c000644 001750 001750 00000010612 13031031271 023174 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** Copyright (c) 2006, Ahmad Faraj & Xin Yuan, All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Florida State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ************************************************************************* * Any results obtained from executing this software require the * * acknowledgment and citation of the software and its owners. * * The full citation is given below: * * * * A. Faraj and X. Yuan. "Automatic Generation and Tuning of MPI * * Collective Communication Routines." The 19th ACM International * * Conference on Supercomputing (ICS), Cambridge, Massachusetts, * * June 20-22, 2005. * ************************************************************************* *****************************************************************************/ /***************************************************************************** * Function: allgather_ring * return: int * inputs: * send_buff: send input buffer * send_count: number of elements to send * send_type: data type of elements being sent * recv_buff: receive output buffer * recv_count: number of elements to received * recv_type: data type of elements being received * comm: communication * Descrp: Function works in P - 1 steps. In step i, node j - i -> j -> j+ i. * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_allgatherv_ring(void *send_buff, int send_count, MPI_Datatype send_type, void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Aint extent; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLGATHERV; MPI_Status status; char *sendptr = (char *) send_buff; char *recvptr = (char *) recv_buff; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); extent = smpi_datatype_get_extent(send_type); // local send/recv smpi_mpi_sendrecv(sendptr, send_count, send_type, rank, tag, recvptr + recv_disps[rank] * extent, recv_counts[rank], recv_type, rank, tag, comm, &status); for (i = 1; i < num_procs; i++) { src = (rank - i + num_procs) % num_procs; dst = (rank + i) % num_procs; smpi_mpi_sendrecv(sendptr, send_count, send_type, dst, tag, recvptr + recv_disps[src] * extent, recv_counts[src], recv_type, src, tag, comm, &status); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/alltoallv-ring.c000644 001750 001750 00000004115 13031031271 023036 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** * Function: alltoall_ring * Return: int * Inputs: send_buff: send input buffer send_count: number of elements to send send_type: data type of elements being sent recv_buff: receive output buffer recv_count: number of elements to received recv_type: data type of elements being received comm: communicator * Descrp: Function works in P - 1 steps. In step i, node j - i -> j -> j + i. * Auther: Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_alltoallv_ring(void *send_buff, int *send_counts, int *send_disps, MPI_Datatype send_type, void *recv_buff, int *recv_counts, int *recv_disps, MPI_Datatype recv_type, MPI_Comm comm) { MPI_Status s; MPI_Aint send_chunk, recv_chunk; int i, src, dst, rank, num_procs; int tag = COLL_TAG_ALLTOALLV; char *send_ptr = (char *) send_buff; char *recv_ptr = (char *) recv_buff; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); send_chunk = smpi_datatype_get_extent(send_type); recv_chunk = smpi_datatype_get_extent(recv_type); int pof2 = ((num_procs != 0) && ((num_procs & (~num_procs + 1)) == num_procs)); for (i = 0; i < num_procs; i++) { if (pof2 == 1) { /* use exclusive-or algorithm */ src = dst = rank ^ i; } else { src = (rank - i + num_procs) % num_procs; dst = (rank + i) % num_procs; } smpi_mpi_sendrecv(send_ptr + send_disps[dst] * send_chunk, send_counts[dst], send_type, dst, tag, recv_ptr + recv_disps[src] * recv_chunk, recv_counts[src], recv_type, src, tag, comm, &s); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/alltoall-mvapich-scatter-dest.c000644 001750 001750 00000012275 13031031271 025746 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ /* * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2012 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * Copyright (c) 2008 Sun Microsystems, Inc. All rights reserved. * Copyright (c) 2009 University of Houston. All rights reserved. * * Additional copyrights may follow */ /* * * (C) 2001 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ /* Copyright (c) 2001-2014, The Ohio State University. All rights * reserved. * * This file is part of the MVAPICH2 software package developed by the * team members of The Ohio State University's Network-Based Computing * Laboratory (NBCL), headed by Professor Dhabaleswar K. (DK) Panda. * * For detailed copyright and licensing information, please refer to the * copyright file COPYRIGHT in the top level MVAPICH2 directory. * */ //correct on stampede #define MV2_ALLTOALL_THROTTLE_FACTOR 4 #include "colls_private.h" int smpi_coll_tuned_alltoall_mvapich2_scatter_dest( void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm) { int comm_size, i, j; MPI_Aint sendtype_extent = 0, recvtype_extent = 0; int mpi_errno=MPI_SUCCESS; int dst, rank; MPI_Request *reqarray; MPI_Status *starray; if (recvcount == 0) return MPI_SUCCESS; comm_size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); /* Get extent of send and recv types */ recvtype_extent = smpi_datatype_get_extent(recvtype); sendtype_extent = smpi_datatype_get_extent(sendtype); /* Medium-size message. Use isend/irecv with scattered destinations. Use Tony Ladd's modification to post only a small number of isends/irecvs at a time. */ /* FIXME: This converts the Alltoall to a set of blocking phases. Two alternatives should be considered: 1) the choice of communication pattern could try to avoid contending routes in each phase 2) rather than wait for all communication to finish (waitall), we could maintain constant queue size by using waitsome and posting new isend/irecv as others complete. This avoids synchronization delays at the end of each block (when there are only a few isend/irecvs left) */ int ii, ss, bblock; //Stampede is configured with bblock = MV2_ALLTOALL_THROTTLE_FACTOR;//mv2_coll_param.alltoall_throttle_factor; if (bblock >= comm_size) bblock = comm_size; /* If throttle_factor is n, each process posts n pairs of isend/irecv in each iteration. */ /* FIXME: This should use the memory macros (there are storage leaks here if there is an error, for example) */ reqarray= (MPI_Request*)xbt_malloc(2*bblock*sizeof(MPI_Request)); starray=(MPI_Status *)xbt_malloc(2*bblock*sizeof(MPI_Status)); for (ii=0; iiMAXTREEFANOUT) { XBT_DEBUG("coll:tuned:topo_build_tree invalid fanout %d bigger than max %d", fanout, MAXTREEFANOUT); return NULL; } /* * Get size and rank of the process in this communicator */ size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); tree = (ompi_coll_tree_t*)malloc(sizeof(ompi_coll_tree_t)); if (!tree) { XBT_DEBUG("coll:tuned:topo_build_tree PANIC::out of memory"); return NULL; } tree->tree_root = MPI_UNDEFINED; tree->tree_nextsize = MPI_UNDEFINED; /* * Set root */ tree->tree_root = root; /* * Initialize tree */ tree->tree_fanout = fanout; tree->tree_bmtree = 0; tree->tree_root = root; tree->tree_prev = -1; tree->tree_nextsize = 0; for( i = 0; i < fanout; i++ ) { tree->tree_next[i] = -1; } /* return if we have less than 2 processes */ if( size < 2 ) { return tree; } /* * Shift all ranks by root, so that the algorithm can be * designed as if root would be always 0 * shiftedrank should be used in calculating distances * and position in tree */ shiftedrank = rank - root; if( shiftedrank < 0 ) { shiftedrank += size; } /* calculate my level */ level = calculate_level( fanout, shiftedrank ); delta = pown( fanout, level ); /* find my children */ for( i = 0; i < fanout; i++ ) { schild = shiftedrank + delta * (i+1); if( schild < size ) { tree->tree_next[i] = (schild+root)%size; tree->tree_nextsize = tree->tree_nextsize + 1; } else { break; } } /* find my parent */ slimit = calculate_num_nodes_up_to_level( fanout, level ); sparent = shiftedrank; if( sparent < fanout ) { sparent = 0; } else { while( sparent >= slimit ) { sparent -= delta/fanout; } } tree->tree_prev = (sparent+root)%size; return tree; } /* * Constructs in-order binary tree which can be used for non-commutative reduce * operations. * Root of this tree is always rank (size-1) and fanout is 2. * Here are some of the examples of this tree: * size == 2 size == 3 size == 4 size == 9 * 1 2 3 8 * / / \ / \ / \ * 0 1 0 2 1 7 3 * / / \ / \ * 0 6 5 2 1 * / / * 4 0 */ ompi_coll_tree_t* ompi_coll_tuned_topo_build_in_order_bintree( MPI_Comm comm ) { int rank, size; int myrank, rightsize, delta; int parent, lchild, rchild; ompi_coll_tree_t* tree; /* * Get size and rank of the process in this communicator */ size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); tree = (ompi_coll_tree_t*)malloc(sizeof(ompi_coll_tree_t)); if (!tree) { XBT_DEBUG( "coll:tuned:topo_build_tree PANIC::out of memory"); return NULL; } tree->tree_root = MPI_UNDEFINED; tree->tree_nextsize = MPI_UNDEFINED; /* * Initialize tree */ tree->tree_fanout = 2; tree->tree_bmtree = 0; tree->tree_root = size - 1; tree->tree_prev = -1; tree->tree_nextsize = 0; tree->tree_next[0] = -1; tree->tree_next[1] = -1; XBT_DEBUG( "coll:tuned:topo_build_in_order_tree Building fo %d rt %d", tree->tree_fanout, tree->tree_root); /* * Build the tree */ myrank = rank; parent = size - 1; delta = 0; while ( 1 ) { /* Compute the size of the right subtree */ rightsize = size >> 1; /* Determine the left and right child of this parent */ lchild = -1; rchild = -1; if (size - 1 > 0) { lchild = parent - 1; if (lchild > 0) { rchild = rightsize - 1; } } /* The following cases are possible: myrank can be - a parent, - belong to the left subtree, or - belong to the right subtee Each of the cases need to be handled differently. */ if (myrank == parent) { /* I am the parent: - compute real ranks of my children, and exit the loop. */ if (lchild >= 0) tree->tree_next[0] = lchild + delta; if (rchild >= 0) tree->tree_next[1] = rchild + delta; break; } if (myrank > rchild) { /* I belong to the left subtree: - If I am the left child, compute real rank of my parent - Iterate down through tree: compute new size, shift ranks down, and update delta. */ if (myrank == lchild) { tree->tree_prev = parent + delta; } size = size - rightsize - 1; delta = delta + rightsize; myrank = myrank - rightsize; parent = size - 1; } else { /* I belong to the right subtree: - If I am the right child, compute real rank of my parent - Iterate down through tree: compute new size and parent, but the delta and rank do not need to change. */ if (myrank == rchild) { tree->tree_prev = parent + delta; } size = rightsize; parent = rchild; } } if (tree->tree_next[0] >= 0) { tree->tree_nextsize = 1; } if (tree->tree_next[1] >= 0) { tree->tree_nextsize += 1; } return tree; } int ompi_coll_tuned_topo_destroy_tree( ompi_coll_tree_t** tree ) { ompi_coll_tree_t *ptr; if ((!tree)||(!*tree)) { return MPI_SUCCESS; } ptr = *tree; free (ptr); *tree = NULL; /* mark tree as gone */ return MPI_SUCCESS; } /* * * Here are some of the examples of this tree: * size == 2 size = 4 size = 8 * 0 0 0 * / | \ / | \ * 1 2 1 4 2 1 * | | |\ * 3 6 5 3 * | * 7 */ ompi_coll_tree_t* ompi_coll_tuned_topo_build_bmtree( MPI_Comm comm, int root ) { int childs = 0; int rank; int size; int mask = 1; int index; int remote; ompi_coll_tree_t *bmtree; int i; XBT_DEBUG("coll:tuned:topo:build_bmtree rt %d", root); /* * Get size and rank of the process in this communicator */ size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); index = rank -root; bmtree = (ompi_coll_tree_t*)malloc(sizeof(ompi_coll_tree_t)); if (!bmtree) { XBT_DEBUG("coll:tuned:topo:build_bmtree PANIC out of memory"); return NULL; } bmtree->tree_bmtree = 1; bmtree->tree_root = MPI_UNDEFINED; bmtree->tree_nextsize = MPI_UNDEFINED; for(i=0;itree_next[i] = -1; } if( index < 0 ) index += size; while( mask <= index ) mask <<= 1; /* Now I can compute my father rank */ if( root == rank ) { bmtree->tree_prev = root; } else { remote = (index ^ (mask >> 1)) + root; if( remote >= size ) remote -= size; bmtree->tree_prev = remote; } /* And now let's fill my childs */ while( mask < size ) { remote = (index ^ mask); if( remote >= size ) break; remote += root; if( remote >= size ) remote -= size; if (childs==MAXTREEFANOUT) { XBT_DEBUG("coll:tuned:topo:build_bmtree max fanout incorrect %d needed %d", MAXTREEFANOUT, childs); return NULL; } bmtree->tree_next[childs] = remote; mask <<= 1; childs++; } bmtree->tree_nextsize = childs; bmtree->tree_root = root; return bmtree; } /* * Constructs in-order binomial tree which can be used for gather/scatter * operations. * * Here are some of the examples of this tree: * size == 2 size = 4 size = 8 * 0 0 0 * / / | / | \ * 1 1 2 1 2 4 * | | | \ * 3 3 5 6 * | * 7 */ ompi_coll_tree_t* ompi_coll_tuned_topo_build_in_order_bmtree( MPI_Comm comm, int root ) { int childs = 0; int rank, vrank; int size; int mask = 1; int remote; ompi_coll_tree_t *bmtree; int i; XBT_DEBUG("coll:tuned:topo:build_in_order_bmtree rt %d", root); /* * Get size and rank of the process in this communicator */ size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); vrank = (rank - root + size) % size; bmtree = (ompi_coll_tree_t*)xbt_malloc(sizeof(ompi_coll_tree_t)); if (!bmtree) { XBT_DEBUG("coll:tuned:topo:build_bmtree PANIC out of memory"); return NULL; } bmtree->tree_bmtree = 1; bmtree->tree_root = MPI_UNDEFINED; bmtree->tree_nextsize = MPI_UNDEFINED; for(i=0;itree_next[i] = -1; } if (root == rank) { bmtree->tree_prev = root; } while (mask < size) { remote = vrank ^ mask; if (remote < vrank) { bmtree->tree_prev = (remote + root) % size; break; } else if (remote < size) { bmtree->tree_next[childs] = (remote + root) % size; childs++; if (childs==MAXTREEFANOUT) { XBT_DEBUG( "coll:tuned:topo:build_bmtree max fanout incorrect %d needed %d", MAXTREEFANOUT, childs); return NULL; } } mask <<= 1; } bmtree->tree_nextsize = childs; bmtree->tree_root = root; return bmtree; } ompi_coll_tree_t* ompi_coll_tuned_topo_build_chain( int fanout, MPI_Comm comm, int root ) { int rank, size; int srank; /* shifted rank */ int i,maxchainlen; int mark,head,len; ompi_coll_tree_t *chain; XBT_DEBUG("coll:tuned:topo:build_chain fo %d rt %d", fanout, root); /* * Get size and rank of the process in this communicator */ size = smpi_comm_size(comm); rank = smpi_comm_rank(comm); if( fanout < 1 ) { XBT_DEBUG("coll:tuned:topo:build_chain WARNING invalid fanout of ZERO, forcing to 1 (pipeline)!"); fanout = 1; } if (fanout>MAXTREEFANOUT) { XBT_DEBUG("coll:tuned:topo:build_chain WARNING invalid fanout %d bigger than max %d, forcing to max!", fanout, MAXTREEFANOUT); fanout = MAXTREEFANOUT; } /* * Allocate space for topology arrays if needed */ chain = (ompi_coll_tree_t*)malloc( sizeof(ompi_coll_tree_t) ); if (!chain) { XBT_DEBUG("coll:tuned:topo:build_chain PANIC out of memory"); fflush(stdout); return NULL; } chain->tree_root = MPI_UNDEFINED; chain->tree_nextsize = -1; for(i=0;itree_next[i] = -1; /* * Set root & numchain */ chain->tree_root = root; if( (size - 1) < fanout ) { chain->tree_nextsize = size-1; fanout = size-1; } else { chain->tree_nextsize = fanout; } /* * Shift ranks */ srank = rank - root; if (srank < 0) srank += size; /* * Special case - fanout == 1 */ if( fanout == 1 ) { if( srank == 0 ) chain->tree_prev = -1; else chain->tree_prev = (srank-1+root)%size; if( (srank + 1) >= size) { chain->tree_next[0] = -1; chain->tree_nextsize = 0; } else { chain->tree_next[0] = (srank+1+root)%size; chain->tree_nextsize = 1; } return chain; } /* Let's handle the case where there is just one node in the communicator */ if( size == 1 ) { chain->tree_next[0] = -1; chain->tree_nextsize = 0; chain->tree_prev = -1; return chain; } /* * Calculate maximum chain length */ maxchainlen = (size-1) / fanout; if( (size-1) % fanout != 0 ) { maxchainlen++; mark = (size-1)%fanout; } else { mark = fanout+1; } /* * Find your own place in the list of shifted ranks */ if( srank != 0 ) { int column; if( srank-1 < (mark * maxchainlen) ) { column = (srank-1)/maxchainlen; head = 1+column*maxchainlen; len = maxchainlen; } else { column = mark + (srank-1-mark*maxchainlen)/(maxchainlen-1); head = mark*maxchainlen+1+(column-mark)*(maxchainlen-1); len = maxchainlen-1; } if( srank == head ) { chain->tree_prev = 0; /*root*/ } else { chain->tree_prev = srank-1; /* rank -1 */ } if( srank == (head + len - 1) ) { chain->tree_next[0] = -1; chain->tree_nextsize = 0; } else { if( (srank + 1) < size ) { chain->tree_next[0] = srank+1; chain->tree_nextsize = 1; } else { chain->tree_next[0] = -1; chain->tree_nextsize = 0; } } } /* * Unshift values */ if( rank == root ) { chain->tree_prev = -1; chain->tree_next[0] = (root+1)%size; for( i = 1; i < fanout; i++ ) { chain->tree_next[i] = chain->tree_next[i-1] + maxchainlen; if( i > mark ) { chain->tree_next[i]--; } chain->tree_next[i] %= size; } chain->tree_nextsize = fanout; } else { chain->tree_prev = (chain->tree_prev+root)%size; if( chain->tree_next[0] != -1 ) { chain->tree_next[0] = (chain->tree_next[0]+root)%size; } } return chain; } int ompi_coll_tuned_topo_dump_tree (ompi_coll_tree_t* tree, int rank) { int i; XBT_DEBUG("coll:tuned:topo:topo_dump_tree %1d tree root %d" " fanout %d BM %1d nextsize %d prev %d", rank, tree->tree_root, tree->tree_bmtree, tree->tree_fanout, tree->tree_nextsize, tree->tree_prev); if( tree->tree_nextsize ) { for( i = 0; i < tree->tree_nextsize; i++ ) XBT_DEBUG("[%1d] %d", i, tree->tree_next[i]); } return (0); } SimGrid-3.14.159/src/smpi/colls/bcast-binomial-tree.c000644 001750 001750 00000010120 13031031271 023721 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" /***************************************************************************** Copyright (c) 2006, Ahmad Faraj & Xin Yuan, All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the Florida State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ************************************************************************* * Any results obtained from executing this software require the * * acknowledgment and citation of the software and its owners. * * The full citation is given below: * * * * A. Faraj and X. Yuan. "Automatic Generation and Tuning of MPI * * Collective Communication Routines." The 19th ACM International * * Conference on Supercomputing (ICS), Cambridge, Massachusetts, * * June 20-22, 2005. * ************************************************************************* *****************************************************************************/ /***************************************************************************** * Function: bcast_binomial_tree * Return: int * Inputs: buff: send input buffer count: number of elements to send data_type: data type of elements being sent root: source of data comm: communicator * Descrp: broadcasts using a bionomial tree. * Auther: MPIH / modified by Ahmad Faraj ****************************************************************************/ int smpi_coll_tuned_bcast_binomial_tree(void *buff, int count, MPI_Datatype data_type, int root, MPI_Comm comm) { int src, dst, rank, num_procs, mask, relative_rank; int tag = COLL_TAG_BCAST; rank = smpi_comm_rank(comm); num_procs = smpi_comm_size(comm); relative_rank = (rank >= root) ? rank - root : rank - root + num_procs; mask = 0x1; while (mask < num_procs) { if (relative_rank & mask) { src = rank - mask; if (src < 0) src += num_procs; smpi_mpi_recv(buff, count, data_type, src, tag, comm, MPI_STATUS_IGNORE); break; } mask <<= 1; } mask >>= 1; while (mask > 0) { if (relative_rank + mask < num_procs) { dst = rank + mask; if (dst >= num_procs) dst -= num_procs; smpi_mpi_send(buff, count, data_type, dst, tag, comm); } mask >>= 1; } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/allgather-NTSLR.c000644 001750 001750 00000003562 13031031271 022757 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "colls_private.h" // Allgather-Non-Topoloty-Scecific-Logical-Ring algorithm int smpi_coll_tuned_allgather_NTSLR(void *sbuf, int scount, MPI_Datatype stype, void *rbuf, int rcount, MPI_Datatype rtype, MPI_Comm comm) { MPI_Aint rextent, sextent; MPI_Status status; int i, to, from, rank, size; int send_offset, recv_offset; int tag = COLL_TAG_ALLGATHER; rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); rextent = smpi_datatype_get_extent(rtype); sextent = smpi_datatype_get_extent(stype); // irregular case use default MPI fucntions if (scount * sextent != rcount * rextent) { XBT_WARN("MPI_allgather_NTSLR use default MPI_allgather."); smpi_mpi_allgather(sbuf, scount, stype, rbuf, rcount, rtype, comm); return MPI_SUCCESS; } // topo non-specific to = (rank + 1) % size; from = (rank + size - 1) % size; //copy a single segment from sbuf to rbuf send_offset = rank * scount * sextent; smpi_mpi_sendrecv(sbuf, scount, stype, rank, tag, (char *)rbuf + send_offset, rcount, rtype, rank, tag, comm, &status); //start sending logical ring message int increment = scount * sextent; for (i = 0; i < size - 1; i++) { send_offset = ((rank - i + size) % size) * increment; recv_offset = ((rank - i - 1 + size) % size) * increment; smpi_mpi_sendrecv((char *) rbuf + send_offset, scount, stype, to, tag + i, (char *) rbuf + recv_offset, rcount, rtype, from, tag + i, comm, &status); } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/colls/smpi_mvapich2_selector_stampede.h000644 001750 001750 00000170367 13031031273 026456 0ustar00mquinsonmquinson000000 000000 /* selector for collective algorithms based on mvapich decision logic, with calibration from Stampede cluster at TACC*/ /* Copyright (c) 2009-2010, 2013-2014. The SimGrid Team. * All rights reserved. */ /* This is the tuning used by MVAPICH for Stampede platform based on (MV2_ARCH_INTEL_XEON_E5_2680_16, MV2_HCA_MLX_CX_FDR) */ /************ Alltoall variables and initializers */ #define MV2_MAX_NB_THRESHOLDS 32 typedef struct { int min; int max; int (*MV2_pt_Alltoall_function) (void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm_ptr ); } mv2_alltoall_tuning_element; typedef struct { int numproc; int size_table; mv2_alltoall_tuning_element algo_table[MV2_MAX_NB_THRESHOLDS]; mv2_alltoall_tuning_element in_place_algo_table[MV2_MAX_NB_THRESHOLDS]; } mv2_alltoall_tuning_table; int (*MV2_Alltoall_function) (void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm_ptr)=NULL; /* Indicates number of processes per node */ int *mv2_alltoall_table_ppn_conf = NULL; /* Indicates total number of configurations */ int mv2_alltoall_num_ppn_conf = 1; int *mv2_size_alltoall_tuning_table = NULL; mv2_alltoall_tuning_table **mv2_alltoall_thresholds_table = NULL; #define MPIR_Alltoall_bruck_MV2 smpi_coll_tuned_alltoall_bruck #define MPIR_Alltoall_RD_MV2 smpi_coll_tuned_alltoall_rdb #define MPIR_Alltoall_Scatter_dest_MV2 smpi_coll_tuned_alltoall_mvapich2_scatter_dest #define MPIR_Alltoall_pairwise_MV2 smpi_coll_tuned_alltoall_pair #define MPIR_Alltoall_inplace_MV2 smpi_coll_tuned_alltoall_ring static void init_mv2_alltoall_tables_stampede(){ int i; int agg_table_sum = 0; mv2_alltoall_tuning_table **table_ptrs = NULL; mv2_alltoall_num_ppn_conf = 3; if(smpi_coll_cleanup_callback==NULL) smpi_coll_cleanup_callback=&smpi_coll_cleanup_mvapich2; mv2_alltoall_thresholds_table = xbt_malloc(sizeof(mv2_alltoall_tuning_table *) * mv2_alltoall_num_ppn_conf); table_ptrs = xbt_malloc(sizeof(mv2_alltoall_tuning_table *) * mv2_alltoall_num_ppn_conf); mv2_size_alltoall_tuning_table = xbt_malloc(sizeof(int) * mv2_alltoall_num_ppn_conf); mv2_alltoall_table_ppn_conf = xbt_malloc(mv2_alltoall_num_ppn_conf * sizeof(int)); mv2_alltoall_table_ppn_conf[0] = 1; mv2_size_alltoall_tuning_table[0] = 6; mv2_alltoall_tuning_table mv2_tmp_alltoall_thresholds_table_1ppn[] = { {2, 1, {{0, -1, &MPIR_Alltoall_pairwise_MV2}, }, {{0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, {4, 2, {{0, 262144, &MPIR_Alltoall_Scatter_dest_MV2}, {262144, -1, &MPIR_Alltoall_pairwise_MV2}, }, {{0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, {8, 2, {{0, 8, &MPIR_Alltoall_RD_MV2}, {8, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, {{0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, {16, 3, {{0, 64, &MPIR_Alltoall_RD_MV2}, {64, 512, &MPIR_Alltoall_bruck_MV2}, {512, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, {{0,-1, &MPIR_Alltoall_inplace_MV2}, }, }, {32, 3, {{0, 32, &MPIR_Alltoall_RD_MV2}, {32, 2048, &MPIR_Alltoall_bruck_MV2}, {2048, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, {{0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, {64, 3, {{0, 8, &MPIR_Alltoall_RD_MV2}, {8, 1024, &MPIR_Alltoall_bruck_MV2}, {1024, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, {{0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, }; table_ptrs[0] = mv2_tmp_alltoall_thresholds_table_1ppn; mv2_alltoall_table_ppn_conf[1] = 2; mv2_size_alltoall_tuning_table[1] = 6; mv2_alltoall_tuning_table mv2_tmp_alltoall_thresholds_table_2ppn[] = { {4, 2, {{0, 32, &MPIR_Alltoall_RD_MV2}, {32, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, {{0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, {8, 2, {{0, 64, &MPIR_Alltoall_RD_MV2}, {64, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, {{0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, {16, 3, {{0, 64, &MPIR_Alltoall_RD_MV2}, {64, 2048, &MPIR_Alltoall_bruck_MV2}, {2048, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, {{0,-1, &MPIR_Alltoall_inplace_MV2}, }, }, {32, 3, {{0, 16, &MPIR_Alltoall_RD_MV2}, {16, 2048, &MPIR_Alltoall_bruck_MV2}, {2048, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, {{0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, {64, 3, {{0, 8, &MPIR_Alltoall_RD_MV2}, {8, 1024, &MPIR_Alltoall_bruck_MV2}, {1024, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, {{0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, {128, 3, {{0, 4, &MPIR_Alltoall_RD_MV2}, {4, 2048, &MPIR_Alltoall_bruck_MV2}, {2048, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, {{0, -1, &MPIR_Alltoall_inplace_MV2}, }, }, }; table_ptrs[1] = mv2_tmp_alltoall_thresholds_table_2ppn; mv2_alltoall_table_ppn_conf[2] = 16; mv2_size_alltoall_tuning_table[2] = 7; mv2_alltoall_tuning_table mv2_tmp_alltoall_thresholds_table_16ppn[] = { {16, 2, {{0, 2048, &MPIR_Alltoall_bruck_MV2}, {2048, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, {{32768, -1, &MPIR_Alltoall_inplace_MV2}, }, }, {32, 2, {{0, 2048, &MPIR_Alltoall_bruck_MV2}, {2048, -1, &MPIR_Alltoall_Scatter_dest_MV2}, }, {{16384, -1, &MPIR_Alltoall_inplace_MV2}, }, }, {64, 3, {{0, 2048, &MPIR_Alltoall_bruck_MV2}, {2048, 16384, &MPIR_Alltoall_Scatter_dest_MV2}, {16384, -1, &MPIR_Alltoall_pairwise_MV2}, }, {{32768, 131072, &MPIR_Alltoall_inplace_MV2}, }, }, {128, 2, {{0, 2048, &MPIR_Alltoall_bruck_MV2}, {2048, -1, &MPIR_Alltoall_pairwise_MV2}, }, {{16384,65536, &MPIR_Alltoall_inplace_MV2}, }, }, {256, 2, {{0, 1024, &MPIR_Alltoall_bruck_MV2}, {1024, -1, &MPIR_Alltoall_pairwise_MV2}, }, {{16384, 65536, &MPIR_Alltoall_inplace_MV2}, }, }, {512, 2, {{0, 1024, &MPIR_Alltoall_bruck_MV2}, {1024, -1, &MPIR_Alltoall_pairwise_MV2}, }, {{16384, 65536, &MPIR_Alltoall_inplace_MV2}, }, }, {1024, 2, {{0, 1024, &MPIR_Alltoall_bruck_MV2}, {1024, -1, &MPIR_Alltoall_pairwise_MV2}, }, {{16384, 65536, &MPIR_Alltoall_inplace_MV2}, }, }, }; table_ptrs[2] = mv2_tmp_alltoall_thresholds_table_16ppn; agg_table_sum = 0; for (i = 0; i < mv2_alltoall_num_ppn_conf; i++) { agg_table_sum += mv2_size_alltoall_tuning_table[i]; } mv2_alltoall_thresholds_table[0] = xbt_malloc(agg_table_sum * sizeof (mv2_alltoall_tuning_table)); memcpy(mv2_alltoall_thresholds_table[0], table_ptrs[0], (sizeof(mv2_alltoall_tuning_table) * mv2_size_alltoall_tuning_table[0])); for (i = 1; i < mv2_alltoall_num_ppn_conf; i++) { mv2_alltoall_thresholds_table[i] = mv2_alltoall_thresholds_table[i - 1] + mv2_size_alltoall_tuning_table[i - 1]; memcpy(mv2_alltoall_thresholds_table[i], table_ptrs[i], (sizeof(mv2_alltoall_tuning_table) * mv2_size_alltoall_tuning_table[i])); } xbt_free(table_ptrs); } /************ Allgather variables and initializers */ typedef struct { int min; int max; int (*MV2_pt_Allgather_function)(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm_ptr); } mv2_allgather_tuning_element; typedef struct { int numproc; int two_level[MV2_MAX_NB_THRESHOLDS]; int size_inter_table; mv2_allgather_tuning_element inter_leader[MV2_MAX_NB_THRESHOLDS]; } mv2_allgather_tuning_table; int (*MV2_Allgather_function)(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm); int *mv2_allgather_table_ppn_conf = NULL; int mv2_allgather_num_ppn_conf = 1; int *mv2_size_allgather_tuning_table = NULL; mv2_allgather_tuning_table **mv2_allgather_thresholds_table = NULL; static int MPIR_Allgather_RD_Allgather_Comm_MV2( void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm_ptr) { return 0; } #define MPIR_Allgather_Bruck_MV2 smpi_coll_tuned_allgather_bruck #define MPIR_Allgather_RD_MV2 smpi_coll_tuned_allgather_rdb #define MPIR_Allgather_Ring_MV2 smpi_coll_tuned_allgather_ring #define MPIR_2lvl_Allgather_MV2 smpi_coll_tuned_allgather_mvapich2_smp static void init_mv2_allgather_tables_stampede(){ int i; int agg_table_sum = 0; if(smpi_coll_cleanup_callback==NULL) smpi_coll_cleanup_callback=&smpi_coll_cleanup_mvapich2; mv2_allgather_tuning_table **table_ptrs = NULL; mv2_allgather_num_ppn_conf = 3; mv2_allgather_thresholds_table = xbt_malloc(sizeof(mv2_allgather_tuning_table *) * mv2_allgather_num_ppn_conf); table_ptrs = xbt_malloc(sizeof(mv2_allgather_tuning_table *) * mv2_allgather_num_ppn_conf); mv2_size_allgather_tuning_table = xbt_malloc(sizeof(int) * mv2_allgather_num_ppn_conf); mv2_allgather_table_ppn_conf = xbt_malloc(mv2_allgather_num_ppn_conf * sizeof(int)); mv2_allgather_table_ppn_conf[0] = 1; mv2_size_allgather_tuning_table[0] = 6; mv2_allgather_tuning_table mv2_tmp_allgather_thresholds_table_1ppn[] = { { 2, {0}, 1, { {0, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 4, {0,0}, 2, { {0, 262144, &MPIR_Allgather_RD_MV2}, {262144, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 8, {0,0}, 2, { {0, 131072, &MPIR_Allgather_RD_MV2}, {131072, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 16, {0,0}, 2, { {0, 131072, &MPIR_Allgather_RD_MV2}, {131072, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 32, {0,0}, 2, { {0, 65536, &MPIR_Allgather_RD_MV2}, {65536, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 64, {0,0}, 2, { {0, 32768, &MPIR_Allgather_RD_MV2}, {32768, -1, &MPIR_Allgather_Ring_MV2}, }, }, }; table_ptrs[0] = mv2_tmp_allgather_thresholds_table_1ppn; mv2_allgather_table_ppn_conf[1] = 2; mv2_size_allgather_tuning_table[1] = 6; mv2_allgather_tuning_table mv2_tmp_allgather_thresholds_table_2ppn[] = { { 4, {0,0}, 2, { {0, 524288, &MPIR_Allgather_RD_MV2}, {524288, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 8, {0,1,0}, 2, { {0, 32768, &MPIR_Allgather_RD_MV2}, {32768, 524288, &MPIR_Allgather_Ring_MV2}, {524288, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 16, {0,1,0}, 2, { {0, 16384, &MPIR_Allgather_RD_MV2}, {16384, 524288, &MPIR_Allgather_Ring_MV2}, {524288, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 32, {1,1,0}, 2, { {0, 65536, &MPIR_Allgather_RD_MV2}, {65536, 524288, &MPIR_Allgather_Ring_MV2}, {524288, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 64, {1,1,0}, 2, { {0, 32768, &MPIR_Allgather_RD_MV2}, {32768, 524288, &MPIR_Allgather_Ring_MV2}, {524288, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 128, {1,1,0}, 2, { {0, 65536, &MPIR_Allgather_RD_MV2}, {65536, 524288, &MPIR_Allgather_Ring_MV2}, {524288, -1, &MPIR_Allgather_Ring_MV2}, }, }, }; table_ptrs[1] = mv2_tmp_allgather_thresholds_table_2ppn; mv2_allgather_table_ppn_conf[2] = 16; mv2_size_allgather_tuning_table[2] = 6; mv2_allgather_tuning_table mv2_tmp_allgather_thresholds_table_16ppn[] = { { 16, {0,0}, 2, { {0, 1024, &MPIR_Allgather_RD_MV2}, {1024, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 32, {0,0}, 2, { {0, 1024, &MPIR_Allgather_RD_Allgather_Comm_MV2}, {1024, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 64, {0,0}, 2, { {0, 1024, &MPIR_Allgather_RD_Allgather_Comm_MV2}, {1024, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 128, {0,0}, 2, { {0, 1024, &MPIR_Allgather_RD_Allgather_Comm_MV2}, {1024, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 256, {0,0}, 2, { {0, 1024, &MPIR_Allgather_RD_Allgather_Comm_MV2}, {1024, -1, &MPIR_Allgather_Ring_MV2}, }, }, { 512, {0,0}, 2, { {0, 1024, &MPIR_Allgather_RD_Allgather_Comm_MV2}, {1024, -1, &MPIR_Allgather_Ring_MV2}, }, }, }; table_ptrs[2] = mv2_tmp_allgather_thresholds_table_16ppn; agg_table_sum = 0; for (i = 0; i < mv2_allgather_num_ppn_conf; i++) { agg_table_sum += mv2_size_allgather_tuning_table[i]; } mv2_allgather_thresholds_table[0] = xbt_malloc(agg_table_sum * sizeof (mv2_allgather_tuning_table)); memcpy(mv2_allgather_thresholds_table[0], table_ptrs[0], (sizeof(mv2_allgather_tuning_table) * mv2_size_allgather_tuning_table[0])); for (i = 1; i < mv2_allgather_num_ppn_conf; i++) { mv2_allgather_thresholds_table[i] = mv2_allgather_thresholds_table[i - 1] + mv2_size_allgather_tuning_table[i - 1]; memcpy(mv2_allgather_thresholds_table[i], table_ptrs[i], (sizeof(mv2_allgather_tuning_table) * mv2_size_allgather_tuning_table[i])); } xbt_free(table_ptrs); } /************ Gather variables and initializers */ typedef struct { int min; int max; int (*MV2_pt_Gather_function)(void *sendbuf, int sendcnt, MPI_Datatype sendtype, void *recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm_ptr); } mv2_gather_tuning_element; typedef struct { int numproc; int size_inter_table; mv2_gather_tuning_element inter_leader[MV2_MAX_NB_THRESHOLDS]; int size_intra_table; mv2_gather_tuning_element intra_node[MV2_MAX_NB_THRESHOLDS]; } mv2_gather_tuning_table; int mv2_size_gather_tuning_table=7; mv2_gather_tuning_table * mv2_gather_thresholds_table=NULL; typedef int (*MV2_Gather_function_ptr) (void *sendbuf, int sendcnt, MPI_Datatype sendtype, void *recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm); MV2_Gather_function_ptr MV2_Gather_inter_leader_function = NULL; MV2_Gather_function_ptr MV2_Gather_intra_node_function = NULL; #define MPIR_Gather_MV2_Direct smpi_coll_tuned_gather_ompi_basic_linear #define MPIR_Gather_MV2_two_level_Direct smpi_coll_tuned_gather_mvapich2_two_level #define MPIR_Gather_intra smpi_coll_tuned_gather_mpich static void init_mv2_gather_tables_stampede(){ if(smpi_coll_cleanup_callback==NULL) smpi_coll_cleanup_callback=&smpi_coll_cleanup_mvapich2; mv2_size_gather_tuning_table=7; mv2_gather_thresholds_table = xbt_malloc(mv2_size_gather_tuning_table* sizeof (mv2_gather_tuning_table)); mv2_gather_tuning_table mv2_tmp_gather_thresholds_table[]={ {16, 2,{{0, 524288, &MPIR_Gather_MV2_Direct}, {524288, -1, &MPIR_Gather_intra}}, 1,{{0, -1, &MPIR_Gather_MV2_Direct}}}, {32, 3,{{0, 16384, &MPIR_Gather_MV2_Direct}, {16384, 131072, &MPIR_Gather_intra}, {131072, -1, &MPIR_Gather_MV2_two_level_Direct}}, 1,{{0, -1, &MPIR_Gather_intra}}}, {64, 3,{{0, 256, &MPIR_Gather_MV2_two_level_Direct}, {256, 16384, &MPIR_Gather_MV2_Direct}, {256, -1, &MPIR_Gather_MV2_two_level_Direct}}, 1,{{0, -1, &MPIR_Gather_intra}}}, {128, 3,{{0, 512, &MPIR_Gather_MV2_two_level_Direct}, {512, 16384, &MPIR_Gather_MV2_Direct}, {16384, -1, &MPIR_Gather_MV2_two_level_Direct}}, 1,{{0, -1, &MPIR_Gather_intra}}}, {256, 3,{{0, 512, &MPIR_Gather_MV2_two_level_Direct}, {512, 16384, &MPIR_Gather_MV2_Direct}, {16384, -1, &MPIR_Gather_MV2_two_level_Direct}}, 1,{{0, -1, &MPIR_Gather_intra}}}, {512, 3,{{0, 512, &MPIR_Gather_MV2_two_level_Direct}, {512, 16384, &MPIR_Gather_MV2_Direct}, {8196, -1, &MPIR_Gather_MV2_two_level_Direct}}, 1,{{0, -1, &MPIR_Gather_intra}}}, {1024, 3,{{0, 512, &MPIR_Gather_MV2_two_level_Direct}, {512, 16384, &MPIR_Gather_MV2_Direct}, {8196, -1, &MPIR_Gather_MV2_two_level_Direct}}, 1,{{0, -1, &MPIR_Gather_intra}}}, }; memcpy(mv2_gather_thresholds_table, mv2_tmp_gather_thresholds_table, mv2_size_gather_tuning_table * sizeof (mv2_gather_tuning_table)); } /************ Allgatherv variables and initializers */ typedef struct { int min; int max; int (*MV2_pt_Allgatherv_function)(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvtype, MPI_Comm commg); } mv2_allgatherv_tuning_element; typedef struct { int numproc; int size_inter_table; mv2_allgatherv_tuning_element inter_leader[MV2_MAX_NB_THRESHOLDS]; } mv2_allgatherv_tuning_table; int (*MV2_Allgatherv_function)(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvtype, MPI_Comm comm); int mv2_size_allgatherv_tuning_table = 0; mv2_allgatherv_tuning_table *mv2_allgatherv_thresholds_table = NULL; #define MPIR_Allgatherv_Rec_Doubling_MV2 smpi_coll_tuned_allgatherv_mpich_rdb #define MPIR_Allgatherv_Bruck_MV2 smpi_coll_tuned_allgatherv_ompi_bruck #define MPIR_Allgatherv_Ring_MV2 smpi_coll_tuned_allgatherv_mpich_ring static void init_mv2_allgatherv_tables_stampede(){ if(smpi_coll_cleanup_callback==NULL) smpi_coll_cleanup_callback=&smpi_coll_cleanup_mvapich2; mv2_size_allgatherv_tuning_table = 6; mv2_allgatherv_thresholds_table = xbt_malloc(mv2_size_allgatherv_tuning_table * sizeof (mv2_allgatherv_tuning_table)); mv2_allgatherv_tuning_table mv2_tmp_allgatherv_thresholds_table[] = { { 16, 2, { {0, 512, &MPIR_Allgatherv_Rec_Doubling_MV2}, {512, -1, &MPIR_Allgatherv_Ring_MV2}, }, }, { 32, 2, { {0, 512, &MPIR_Allgatherv_Rec_Doubling_MV2}, {512, -1, &MPIR_Allgatherv_Ring_MV2}, }, }, { 64, 2, { {0, 256, &MPIR_Allgatherv_Rec_Doubling_MV2}, {256, -1, &MPIR_Allgatherv_Ring_MV2}, }, }, { 128, 2, { {0, 256, &MPIR_Allgatherv_Rec_Doubling_MV2}, {256, -1, &MPIR_Allgatherv_Ring_MV2}, }, }, { 256, 2, { {0, 256, &MPIR_Allgatherv_Rec_Doubling_MV2}, {256, -1, &MPIR_Allgatherv_Ring_MV2}, }, }, { 512, 2, { {0, 256, &MPIR_Allgatherv_Rec_Doubling_MV2}, {256, -1, &MPIR_Allgatherv_Ring_MV2}, }, }, }; memcpy(mv2_allgatherv_thresholds_table, mv2_tmp_allgatherv_thresholds_table, mv2_size_allgatherv_tuning_table * sizeof (mv2_allgatherv_tuning_table)); } /************ Allreduce variables and initializers */ typedef struct { int min; int max; int (*MV2_pt_Allreduce_function)(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm); } mv2_allreduce_tuning_element; typedef struct { int numproc; int mcast_enabled; int is_two_level_allreduce[MV2_MAX_NB_THRESHOLDS]; int size_inter_table; mv2_allreduce_tuning_element inter_leader[MV2_MAX_NB_THRESHOLDS]; int size_intra_table; mv2_allreduce_tuning_element intra_node[MV2_MAX_NB_THRESHOLDS]; } mv2_allreduce_tuning_table; int (*MV2_Allreduce_function)(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm)=NULL; int (*MV2_Allreduce_intra_function)( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm)=NULL; int mv2_size_allreduce_tuning_table = 0; mv2_allreduce_tuning_table *mv2_allreduce_thresholds_table = NULL; static int MPIR_Allreduce_mcst_reduce_two_level_helper_MV2( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { return 0; } static int MPIR_Allreduce_mcst_reduce_redscat_gather_MV2( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { return 0; } static int MPIR_Allreduce_reduce_p2p_MV2( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { mpi_coll_reduce_fun(sendbuf,recvbuf,count,datatype,op,0,comm); return MPI_SUCCESS; } static int MPIR_Allreduce_reduce_shmem_MV2( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { mpi_coll_reduce_fun(sendbuf,recvbuf,count,datatype,op,0,comm); return MPI_SUCCESS; } #define MPIR_Allreduce_pt2pt_rd_MV2 smpi_coll_tuned_allreduce_rdb #define MPIR_Allreduce_pt2pt_rs_MV2 smpi_coll_tuned_allreduce_mvapich2_rs #define MPIR_Allreduce_two_level_MV2 smpi_coll_tuned_allreduce_mvapich2_two_level static void init_mv2_allreduce_tables_stampede(){ if(smpi_coll_cleanup_callback==NULL) smpi_coll_cleanup_callback=&smpi_coll_cleanup_mvapich2; mv2_size_allreduce_tuning_table = 8; mv2_allreduce_thresholds_table = xbt_malloc(mv2_size_allreduce_tuning_table * sizeof (mv2_allreduce_tuning_table)); mv2_allreduce_tuning_table mv2_tmp_allreduce_thresholds_table[] = { { 16, 0, {1, 0}, 2, { {0, 1024, &MPIR_Allreduce_pt2pt_rd_MV2}, {1024, -1, &MPIR_Allreduce_pt2pt_rs_MV2}, }, 2, { {0, 1024, &MPIR_Allreduce_reduce_shmem_MV2}, {1024, -1, &MPIR_Allreduce_reduce_p2p_MV2}, }, }, { 32, 0, {1, 1, 0}, 3, { {0, 1024, &MPIR_Allreduce_pt2pt_rd_MV2}, {1024, 16384, &MPIR_Allreduce_pt2pt_rd_MV2}, {16384, -1, &MPIR_Allreduce_pt2pt_rs_MV2}, }, 2, { {0, 1024, &MPIR_Allreduce_reduce_shmem_MV2}, {1024, 16384, &MPIR_Allreduce_reduce_p2p_MV2}, }, }, { 64, 0, {1, 1, 0}, 3, { {0, 512, &MPIR_Allreduce_pt2pt_rd_MV2}, {512, 16384, &MPIR_Allreduce_pt2pt_rd_MV2}, {16384, -1, &MPIR_Allreduce_pt2pt_rs_MV2}, }, 2, { {0, 512, &MPIR_Allreduce_reduce_shmem_MV2}, {512, 16384, &MPIR_Allreduce_reduce_p2p_MV2}, }, }, { 128, 0, {1, 1, 0}, 3, { {0, 512, &MPIR_Allreduce_pt2pt_rd_MV2}, {512, 16384, &MPIR_Allreduce_pt2pt_rd_MV2}, {16384, -1, &MPIR_Allreduce_pt2pt_rs_MV2}, }, 2, { {0, 512, &MPIR_Allreduce_reduce_shmem_MV2}, {512, 16384, &MPIR_Allreduce_reduce_p2p_MV2}, }, }, { 256, 0, {1, 1, 0}, 3, { {0, 512, &MPIR_Allreduce_pt2pt_rd_MV2}, {512, 16384, &MPIR_Allreduce_pt2pt_rd_MV2}, {16384, -1, &MPIR_Allreduce_pt2pt_rs_MV2}, }, 2, { {0, 512, &MPIR_Allreduce_reduce_shmem_MV2}, {512, -1, &MPIR_Allreduce_reduce_p2p_MV2}, }, }, { 512, 0, {1, 1, 0}, 3, { {0, 512, &MPIR_Allreduce_pt2pt_rd_MV2}, {512, 16384, &MPIR_Allreduce_pt2pt_rd_MV2}, {16384, -1, &MPIR_Allreduce_pt2pt_rs_MV2}, }, 2, { {0, 512, &MPIR_Allreduce_reduce_shmem_MV2}, {512, 16384, &MPIR_Allreduce_reduce_p2p_MV2}, }, }, { 1024, 0, {1, 1, 1, 0}, 4, { {0, 512, &MPIR_Allreduce_pt2pt_rd_MV2}, {512, 8192, &MPIR_Allreduce_pt2pt_rd_MV2}, {8192, 65536, &MPIR_Allreduce_pt2pt_rs_MV2}, {65536, -1, &MPIR_Allreduce_pt2pt_rs_MV2}, }, 2, { {0, 512, &MPIR_Allreduce_reduce_shmem_MV2}, {512, -1, &MPIR_Allreduce_reduce_p2p_MV2}, }, }, { 2048, 0, {1, 1, 1, 0}, 4, { {0, 64, &MPIR_Allreduce_pt2pt_rd_MV2}, {64, 512, &MPIR_Allreduce_reduce_p2p_MV2}, {512, 4096, &MPIR_Allreduce_mcst_reduce_two_level_helper_MV2}, {4096, 16384, &MPIR_Allreduce_pt2pt_rs_MV2}, {16384, -1, &MPIR_Allreduce_pt2pt_rs_MV2}, }, 2, { {0, 512, &MPIR_Allreduce_reduce_shmem_MV2}, {512, -1, &MPIR_Allreduce_reduce_p2p_MV2}, }, }, }; memcpy(mv2_allreduce_thresholds_table, mv2_tmp_allreduce_thresholds_table, mv2_size_allreduce_tuning_table * sizeof (mv2_allreduce_tuning_table)); } typedef struct { int min; int max; int (*MV2_pt_Bcast_function) (void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm_ptr); int zcpy_pipelined_knomial_factor; } mv2_bcast_tuning_element; typedef struct { int numproc; int bcast_segment_size; int intra_node_knomial_factor; int inter_node_knomial_factor; int is_two_level_bcast[MV2_MAX_NB_THRESHOLDS]; int size_inter_table; mv2_bcast_tuning_element inter_leader[MV2_MAX_NB_THRESHOLDS]; int size_intra_table; mv2_bcast_tuning_element intra_node[MV2_MAX_NB_THRESHOLDS]; } mv2_bcast_tuning_table; int mv2_size_bcast_tuning_table = 0; mv2_bcast_tuning_table *mv2_bcast_thresholds_table = NULL; int (*MV2_Bcast_function) (void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm_ptr) = NULL; int (*MV2_Bcast_intra_node_function) (void *buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm_ptr) = NULL; int zcpy_knomial_factor = 2; int mv2_pipelined_zcpy_knomial_factor = -1; int bcast_segment_size = 8192; int mv2_inter_node_knomial_factor = 4; int mv2_intra_node_knomial_factor = 4; #define mv2_bcast_two_level_system_size 64 #define mv2_bcast_short_msg 16384 #define mv2_bcast_large_msg 512*1024 #define INTRA_NODE_ROOT 0 #define MPIR_Pipelined_Bcast_Zcpy_MV2 smpi_coll_tuned_bcast_mpich #define MPIR_Pipelined_Bcast_MV2 smpi_coll_tuned_bcast_mpich #define MPIR_Bcast_binomial_MV2 smpi_coll_tuned_bcast_binomial_tree #define MPIR_Bcast_scatter_ring_allgather_shm_MV2 smpi_coll_tuned_bcast_scatter_LR_allgather #define MPIR_Bcast_scatter_doubling_allgather_MV2 smpi_coll_tuned_bcast_scatter_rdb_allgather #define MPIR_Bcast_scatter_ring_allgather_MV2 smpi_coll_tuned_bcast_scatter_LR_allgather #define MPIR_Shmem_Bcast_MV2 smpi_coll_tuned_bcast_mpich #define MPIR_Bcast_tune_inter_node_helper_MV2 smpi_coll_tuned_bcast_mvapich2_inter_node #define MPIR_Bcast_inter_node_helper_MV2 smpi_coll_tuned_bcast_mvapich2_inter_node #define MPIR_Knomial_Bcast_intra_node_MV2 smpi_coll_tuned_bcast_mvapich2_knomial_intra_node #define MPIR_Bcast_intra_MV2 smpi_coll_tuned_bcast_mvapich2_intra_node static void init_mv2_bcast_tables_stampede(){ //Stampede, if(smpi_coll_cleanup_callback==NULL) smpi_coll_cleanup_callback=&smpi_coll_cleanup_mvapich2; mv2_size_bcast_tuning_table=8; mv2_bcast_thresholds_table = xbt_malloc(mv2_size_bcast_tuning_table * sizeof (mv2_bcast_tuning_table)); mv2_bcast_tuning_table mv2_tmp_bcast_thresholds_table[]={ { 16, 8192, 4, 4, {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}, 11, { {0, 8, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {8, 16, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {16, 1024, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {1024, 8192, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {8192, 16384, &MPIR_Bcast_binomial_MV2, -1}, {16384, 32768, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {32768, 65536, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {65536, 131072, &MPIR_Bcast_scatter_ring_allgather_shm_MV2, -1}, {131072, 262144, &MPIR_Bcast_scatter_ring_allgather_MV2, -1}, {262144, 524288, &MPIR_Bcast_scatter_doubling_allgather_MV2, -1}, {524288, -1, &MPIR_Bcast_scatter_ring_allgather_MV2, -1} }, 11, { {0, 8, &MPIR_Shmem_Bcast_MV2, 2}, {8, 16, &MPIR_Shmem_Bcast_MV2, 4}, {16, 1024, &MPIR_Shmem_Bcast_MV2, 2}, {1024, 8192, &MPIR_Shmem_Bcast_MV2, 4}, {8192, 16384, &MPIR_Shmem_Bcast_MV2, -1}, {16384, 32768, &MPIR_Shmem_Bcast_MV2, 4}, {32768, 65536, &MPIR_Shmem_Bcast_MV2, 2}, {65536, 131072, &MPIR_Shmem_Bcast_MV2, -1}, {131072, 262144, &MPIR_Shmem_Bcast_MV2, -1}, {262144, 524288, &MPIR_Shmem_Bcast_MV2, -1}, {524288, -1, &MPIR_Shmem_Bcast_MV2, -1} } }, { 32, 8192, 4, 4, {1, 1, 1, 1, 1, 1, 1, 1}, 8, { {0, 128, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {128, 256, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {256, 32768, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {32768, 65536, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {65536, 131072, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {131072, 262144, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}, {262144, 524288, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {524288, -1, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8} }, 8, { {0, 128, &MPIR_Shmem_Bcast_MV2, 2}, {128, 256, &MPIR_Shmem_Bcast_MV2, 4}, {256, 32768, &MPIR_Shmem_Bcast_MV2, 2}, {32768, 65536, &MPIR_Shmem_Bcast_MV2, 4}, {65536, 131072, &MPIR_Shmem_Bcast_MV2, 2}, {131072, 262144, &MPIR_Shmem_Bcast_MV2, 8}, {262144, 524288, &MPIR_Shmem_Bcast_MV2, 2}, {524288, -1, &MPIR_Shmem_Bcast_MV2, 8} } }, { 64, 8192, 4, 4, {1, 1, 1, 1, 1, 1, 1, 1, 1}, 9, { {0, 2, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {2, 4, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}, {4, 16, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {16, 32, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}, {32, 128, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {128, 256, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}, {256, 4096, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {4096, 32768, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}, {32768, -1, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2} }, 9, { {0, 2, &MPIR_Shmem_Bcast_MV2, 4}, {2, 4, &MPIR_Shmem_Bcast_MV2, 8}, {4, 16, &MPIR_Shmem_Bcast_MV2, 4}, {16, 32, &MPIR_Shmem_Bcast_MV2, 8}, {32, 128, &MPIR_Shmem_Bcast_MV2, 4}, {128, 256, &MPIR_Shmem_Bcast_MV2, 8}, {256, 4096, &MPIR_Shmem_Bcast_MV2, 4}, {4096, 32768, &MPIR_Shmem_Bcast_MV2, 8}, {32768, -1, &MPIR_Shmem_Bcast_MV2, 2} } }, { 128, 8192, 4, 4, {1, 1, 1, 0}, 4, { {0, 8192, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}, {8192, 16384, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {16384, 524288, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {524288, -1, &MPIR_Bcast_scatter_ring_allgather_MV2, -1} }, 4, { {0, 8192, &MPIR_Shmem_Bcast_MV2, 8}, {8192, 16384, &MPIR_Shmem_Bcast_MV2, 4}, {16384, 524288, &MPIR_Shmem_Bcast_MV2, 2}, {524288, -1, NULL, -1} } }, { 256, 8192, 4, 4, {1, 1, 1, 1, 1}, 5, { {0, 16384, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {16384, 131072, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {131072, 262144, &MPIR_Bcast_scatter_ring_allgather_shm_MV2, -1}, {262144, 524288, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {524288, -1, &MPIR_Bcast_scatter_ring_allgather_shm_MV2, -1} }, 5, { {0, 16384, &MPIR_Shmem_Bcast_MV2, 4}, {16384, 131072, &MPIR_Shmem_Bcast_MV2, 2}, {131072, 262144, &MPIR_Shmem_Bcast_MV2, -1}, {262144, 524288, &MPIR_Shmem_Bcast_MV2, 2}, {524288, -1, &MPIR_Shmem_Bcast_MV2, -1} } }, { 512, 8192, 4, 4, {1, 1, 1, 1, 1}, 5, { {0, 4096, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}, {4096, 16384, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {16384, 131072, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {131072, 262144, &MPIR_Pipelined_Bcast_MV2, -1}, {262144, -1, &MPIR_Bcast_scatter_ring_allgather_shm_MV2, -1} }, 5, { {0, 4096, &MPIR_Shmem_Bcast_MV2, 8}, {4096, 16384, &MPIR_Shmem_Bcast_MV2, 4}, {16384, 131072, &MPIR_Shmem_Bcast_MV2, 2}, {131072, 262144, &MPIR_Shmem_Bcast_MV2, -1}, {262144, -1, &MPIR_Shmem_Bcast_MV2, -1} } }, { 1024, 8192, 4, 4, {1, 1, 1, 1, 1}, 5, { {0, 8192, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}, {8192, 16384, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {16384, 65536, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {65536, 524288, &MPIR_Pipelined_Bcast_MV2, -1}, {524288, -1, &MPIR_Bcast_scatter_ring_allgather_shm_MV2, -1} }, 5, { {0, 8192, &MPIR_Shmem_Bcast_MV2, 8}, {8192, 16384, &MPIR_Shmem_Bcast_MV2, 4}, {16384, 65536, &MPIR_Shmem_Bcast_MV2, 2}, {65536, 524288, &MPIR_Shmem_Bcast_MV2, -1}, {524288, -1, &MPIR_Shmem_Bcast_MV2, -1} } }, { 2048, 8192, 4, 4, {1, 1, 1, 1, 1, 1, 1}, 7, { {0, 16, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}, {16, 32, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {32, 4096, &MPIR_Pipelined_Bcast_Zcpy_MV2, 8}, {4096, 16384, &MPIR_Pipelined_Bcast_Zcpy_MV2, 4}, {16384, 32768, &MPIR_Pipelined_Bcast_Zcpy_MV2, 2}, {32768, 524288, &MPIR_Pipelined_Bcast_MV2, -1}, {524288, -1, &MPIR_Bcast_scatter_ring_allgather_shm_MV2, -1} }, 7, { {0, 16, &MPIR_Shmem_Bcast_MV2, 8}, {16, 32, &MPIR_Shmem_Bcast_MV2, 4}, {32, 4096, &MPIR_Shmem_Bcast_MV2, 8}, {4096, 16384, &MPIR_Shmem_Bcast_MV2, 4}, {16384, 32768, &MPIR_Shmem_Bcast_MV2, 2}, {32768, 524288, &MPIR_Shmem_Bcast_MV2, -1}, {524288, -1, &MPIR_Shmem_Bcast_MV2, -1} } } }; memcpy(mv2_bcast_thresholds_table, mv2_tmp_bcast_thresholds_table, mv2_size_bcast_tuning_table * sizeof (mv2_bcast_tuning_table)); } /************ Reduce variables and initializers */ typedef struct { int min; int max; int (*MV2_pt_Reduce_function)(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm_ptr); } mv2_reduce_tuning_element; typedef struct { int numproc; int inter_k_degree; int intra_k_degree; int is_two_level_reduce[MV2_MAX_NB_THRESHOLDS]; int size_inter_table; mv2_reduce_tuning_element inter_leader[MV2_MAX_NB_THRESHOLDS]; int size_intra_table; mv2_reduce_tuning_element intra_node[MV2_MAX_NB_THRESHOLDS]; } mv2_reduce_tuning_table; int mv2_size_reduce_tuning_table = 0; mv2_reduce_tuning_table *mv2_reduce_thresholds_table = NULL; int mv2_reduce_intra_knomial_factor = -1; int mv2_reduce_inter_knomial_factor = -1; int (*MV2_Reduce_function)( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm_ptr)=NULL; int (*MV2_Reduce_intra_function)( void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm_ptr)=NULL; #define MPIR_Reduce_inter_knomial_wrapper_MV2 smpi_coll_tuned_reduce_mvapich2_knomial #define MPIR_Reduce_intra_knomial_wrapper_MV2 smpi_coll_tuned_reduce_mvapich2_knomial #define MPIR_Reduce_binomial_MV2 smpi_coll_tuned_reduce_binomial #define MPIR_Reduce_redscat_gather_MV2 smpi_coll_tuned_reduce_scatter_gather #define MPIR_Reduce_shmem_MV2 smpi_coll_tuned_reduce_ompi_basic_linear #define MPIR_Reduce_two_level_helper_MV2 smpi_coll_tuned_reduce_mvapich2_two_level static void init_mv2_reduce_tables_stampede(){ if(smpi_coll_cleanup_callback==NULL) smpi_coll_cleanup_callback=&smpi_coll_cleanup_mvapich2; /*Stampede*/ mv2_size_reduce_tuning_table = 8; mv2_reduce_thresholds_table = xbt_malloc(mv2_size_reduce_tuning_table * sizeof (mv2_reduce_tuning_table)); mv2_reduce_tuning_table mv2_tmp_reduce_thresholds_table[] = { { 16, 4, 4, {1, 0, 0}, 3, { {0, 262144, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {262144, 1048576, &MPIR_Reduce_binomial_MV2}, {1048576, -1, &MPIR_Reduce_redscat_gather_MV2}, }, 2, { {0, 65536, &MPIR_Reduce_shmem_MV2}, {65536,-1, &MPIR_Reduce_binomial_MV2}, }, }, { 32, 4, 4, {1, 1, 1, 1, 0, 0, 0}, 7, { {0, 8192, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {8192, 16384, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {16384, 32768, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {32768, 65536, &MPIR_Reduce_binomial_MV2}, {65536, 262144, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {262144, 1048576, &MPIR_Reduce_binomial_MV2}, {1048576, -1, &MPIR_Reduce_redscat_gather_MV2}, }, 6, { {0, 8192, &MPIR_Reduce_shmem_MV2}, {8192, 16384, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {16384, 32768, &MPIR_Reduce_shmem_MV2}, {32768, 65536, &MPIR_Reduce_shmem_MV2}, {65536, 262144, &MPIR_Reduce_shmem_MV2}, {262144,-1, &MPIR_Reduce_binomial_MV2}, }, }, { 64, 4, 4, {1, 1, 1, 1, 0}, 5, { {0, 8192, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {8192, 16384, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {16384, 65536, &MPIR_Reduce_binomial_MV2}, {65536, 262144, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {262144, -1, &MPIR_Reduce_redscat_gather_MV2}, }, 5, { {0, 8192, &MPIR_Reduce_shmem_MV2}, {8192, 16384, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {16384, 65536, &MPIR_Reduce_shmem_MV2}, {65536, 262144, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {262144, -1, &MPIR_Reduce_binomial_MV2}, }, }, { 128, 4, 4, {1, 0, 1, 0, 1, 0}, 6, { {0, 8192, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {8192, 16384, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {16384, 65536, &MPIR_Reduce_binomial_MV2}, {65536, 262144, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {262144, 1048576, &MPIR_Reduce_binomial_MV2}, {1048576, -1, &MPIR_Reduce_redscat_gather_MV2}, }, 5, { {0, 8192, &MPIR_Reduce_shmem_MV2}, {8192, 16384, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {16384, 65536, &MPIR_Reduce_shmem_MV2}, {65536, 262144, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {262144, -1, &MPIR_Reduce_binomial_MV2}, }, }, { 256, 4, 4, {1, 1, 1, 0, 1, 1, 0}, 7, { {0, 8192, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {8192, 16384, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {16384, 32768, &MPIR_Reduce_binomial_MV2}, {32768, 65536, &MPIR_Reduce_binomial_MV2}, {65536, 262144, &MPIR_Reduce_binomial_MV2}, {262144, 1048576, &MPIR_Reduce_binomial_MV2}, {1048576, -1, &MPIR_Reduce_redscat_gather_MV2}, }, 6, { {0, 8192, &MPIR_Reduce_shmem_MV2}, {8192, 16384, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {16384, 32768, &MPIR_Reduce_shmem_MV2}, {32768, 65536, &MPIR_Reduce_shmem_MV2}, {65536, 262144, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {262144, -1, &MPIR_Reduce_binomial_MV2}, }, }, { 512, 4, 4, {1, 0, 1, 1, 1, 0}, 6, { {0, 8192, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {8192, 16384, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {16384, 65536, &MPIR_Reduce_binomial_MV2}, {65536, 262144, &MPIR_Reduce_binomial_MV2}, {262144, 1048576, &MPIR_Reduce_binomial_MV2}, {1048576, -1, &MPIR_Reduce_redscat_gather_MV2}, }, 5, { {0, 8192, &MPIR_Reduce_shmem_MV2}, {8192, 16384, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {16384, 65536, &MPIR_Reduce_shmem_MV2}, {65536, 262144, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {262144, -1, &MPIR_Reduce_binomial_MV2}, }, }, { 1024, 4, 4, {1, 0, 1, 1, 1}, 5, { {0, 8192, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {8192, 16384, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {16384, 65536, &MPIR_Reduce_binomial_MV2}, {65536, 262144, &MPIR_Reduce_binomial_MV2}, {262144, -1, &MPIR_Reduce_binomial_MV2}, }, 5, { {0, 8192, &MPIR_Reduce_shmem_MV2}, {8192, 16384, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {16384, 65536, &MPIR_Reduce_shmem_MV2}, {65536, 262144, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {262144, -1, &MPIR_Reduce_binomial_MV2}, }, }, { 2048, 4, 4, {1, 0, 1, 1, 1,1}, 6, { {0, 2048, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {2048, 4096, &MPIR_Reduce_inter_knomial_wrapper_MV2}, {4096, 16384, &MPIR_Reduce_binomial_MV2}, {16384, 65536, &MPIR_Reduce_binomial_MV2}, {65536, 131072, &MPIR_Reduce_binomial_MV2}, {131072, -1, &MPIR_Reduce_binomial_MV2}, }, 6, { {0, 2048, &MPIR_Reduce_shmem_MV2}, {2048, 4096, &MPIR_Reduce_shmem_MV2}, {4096, 16384, &MPIR_Reduce_shmem_MV2}, {16384, 65536, &MPIR_Reduce_intra_knomial_wrapper_MV2}, {65536, 131072, &MPIR_Reduce_binomial_MV2}, {131072, -1, &MPIR_Reduce_shmem_MV2}, }, }, }; memcpy(mv2_reduce_thresholds_table, mv2_tmp_reduce_thresholds_table, mv2_size_reduce_tuning_table * sizeof (mv2_reduce_tuning_table)); } /************ Reduce scatter variables and initializers */ typedef struct { int min; int max; int (*MV2_pt_Red_scat_function)(void *sendbuf, void *recvbuf, int *recvcnts, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm_ptr); } mv2_red_scat_tuning_element; typedef struct { int numproc; int size_inter_table; mv2_red_scat_tuning_element inter_leader[MV2_MAX_NB_THRESHOLDS]; } mv2_red_scat_tuning_table; int mv2_size_red_scat_tuning_table = 0; mv2_red_scat_tuning_table *mv2_red_scat_thresholds_table = NULL; int (*MV2_Red_scat_function)(void *sendbuf, void *recvbuf, int *recvcnts, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm_ptr); static int MPIR_Reduce_Scatter_Basic_MV2(void *sendbuf, void *recvbuf, int *recvcnts, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) { smpi_mpi_reduce_scatter(sendbuf,recvbuf,recvcnts,datatype,op,comm); return MPI_SUCCESS; } #define MPIR_Reduce_scatter_non_comm_MV2 smpi_coll_tuned_reduce_scatter_mpich_noncomm #define MPIR_Reduce_scatter_Rec_Halving_MV2 smpi_coll_tuned_reduce_scatter_ompi_basic_recursivehalving #define MPIR_Reduce_scatter_Pair_Wise_MV2 smpi_coll_tuned_reduce_scatter_mpich_pair static void init_mv2_reduce_scatter_tables_stampede(){ if(smpi_coll_cleanup_callback==NULL) smpi_coll_cleanup_callback=&smpi_coll_cleanup_mvapich2; mv2_size_red_scat_tuning_table = 6; mv2_red_scat_thresholds_table = xbt_malloc(mv2_size_red_scat_tuning_table * sizeof (mv2_red_scat_tuning_table)); mv2_red_scat_tuning_table mv2_tmp_red_scat_thresholds_table[] = { { 16, 3, { {0, 64, &MPIR_Reduce_Scatter_Basic_MV2}, {64, 65536, &MPIR_Reduce_scatter_Rec_Halving_MV2}, {65536, -1, &MPIR_Reduce_scatter_Pair_Wise_MV2}, }, }, { 32, 3, { {0, 64, &MPIR_Reduce_Scatter_Basic_MV2}, {64, 131072, &MPIR_Reduce_scatter_Rec_Halving_MV2}, {131072, -1, &MPIR_Reduce_scatter_Pair_Wise_MV2}, }, }, { 64, 3, { {0, 1024, &MPIR_Reduce_Scatter_Basic_MV2}, {1024, 262144, &MPIR_Reduce_scatter_Rec_Halving_MV2}, {262144, -1, &MPIR_Reduce_scatter_Pair_Wise_MV2}, }, }, { 128, 2, { {0, 128, &MPIR_Reduce_Scatter_Basic_MV2}, {128, -1, &MPIR_Reduce_scatter_Rec_Halving_MV2}, }, }, { 256, 2, { {0, 128, &MPIR_Reduce_Scatter_Basic_MV2}, {128, -1, &MPIR_Reduce_scatter_Rec_Halving_MV2}, }, }, { 512, 2, { {0, 256, &MPIR_Reduce_Scatter_Basic_MV2}, {256, -1, &MPIR_Reduce_scatter_Rec_Halving_MV2}, }, }, }; memcpy(mv2_red_scat_thresholds_table, mv2_tmp_red_scat_thresholds_table, mv2_size_red_scat_tuning_table * sizeof (mv2_red_scat_tuning_table)); } /************ Scatter variables and initializers */ typedef struct { int min; int max; int (*MV2_pt_Scatter_function)(void *sendbuf, int sendcnt, MPI_Datatype sendtype, void *recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm); } mv2_scatter_tuning_element; typedef struct { int numproc; int size_inter_table; mv2_scatter_tuning_element inter_leader[MV2_MAX_NB_THRESHOLDS]; int size_intra_table; mv2_scatter_tuning_element intra_node[MV2_MAX_NB_THRESHOLDS]; } mv2_scatter_tuning_table; int *mv2_scatter_table_ppn_conf = NULL; int mv2_scatter_num_ppn_conf = 1; int *mv2_size_scatter_tuning_table = NULL; mv2_scatter_tuning_table **mv2_scatter_thresholds_table = NULL; int (*MV2_Scatter_function) (void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm)=NULL; int (*MV2_Scatter_intra_function) (void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm)=NULL; int MPIR_Scatter_mcst_wrap_MV2(void *sendbuf, int sendcnt, MPI_Datatype sendtype, void *recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm_ptr); int MPIR_Scatter_mcst_wrap_MV2(void *sendbuf, int sendcnt, MPI_Datatype sendtype, void *recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm_ptr) { return 0; } #define MPIR_Scatter_MV2_Binomial smpi_coll_tuned_scatter_ompi_binomial #define MPIR_Scatter_MV2_Direct smpi_coll_tuned_scatter_ompi_basic_linear #define MPIR_Scatter_MV2_two_level_Binomial smpi_coll_tuned_scatter_mvapich2_two_level_binomial #define MPIR_Scatter_MV2_two_level_Direct smpi_coll_tuned_scatter_mvapich2_two_level_direct static void init_mv2_scatter_tables_stampede(){ if(smpi_coll_cleanup_callback==NULL) smpi_coll_cleanup_callback=&smpi_coll_cleanup_mvapich2; int agg_table_sum = 0; int i; mv2_scatter_tuning_table **table_ptrs = NULL; mv2_scatter_num_ppn_conf = 3; mv2_scatter_thresholds_table = xbt_malloc(sizeof(mv2_scatter_tuning_table *) * mv2_scatter_num_ppn_conf); table_ptrs = xbt_malloc(sizeof(mv2_scatter_tuning_table *) * mv2_scatter_num_ppn_conf); mv2_size_scatter_tuning_table = xbt_malloc(sizeof(int) * mv2_scatter_num_ppn_conf); mv2_scatter_table_ppn_conf = xbt_malloc(mv2_scatter_num_ppn_conf * sizeof(int)); mv2_scatter_table_ppn_conf[0] = 1; mv2_size_scatter_tuning_table[0] = 6; mv2_scatter_tuning_table mv2_tmp_scatter_thresholds_table_1ppn[] = { {2, 1, { {0, -1, &MPIR_Scatter_MV2_Binomial}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Binomial}, }, }, {4, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, }, {8, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, }, {16, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, }, {32, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, }, {64, 2, { {0, 32, &MPIR_Scatter_MV2_Binomial}, {32, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Binomial}, }, }, }; table_ptrs[0] = mv2_tmp_scatter_thresholds_table_1ppn; mv2_scatter_table_ppn_conf[1] = 2; mv2_size_scatter_tuning_table[1] = 6; mv2_scatter_tuning_table mv2_tmp_scatter_thresholds_table_2ppn[] = { {4, 2, { {0, 4096, &MPIR_Scatter_MV2_Binomial}, {4096, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Direct}, }, }, {8, 2, { {0, 512, &MPIR_Scatter_MV2_two_level_Direct}, {512, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Binomial}, }, }, {16, 2, { {0, 2048, &MPIR_Scatter_MV2_two_level_Direct}, {2048, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Binomial}, }, }, {32, 2, { {0, 2048, &MPIR_Scatter_MV2_two_level_Direct}, {2048, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Binomial}, }, }, {64, 2, { {0, 8192, &MPIR_Scatter_MV2_two_level_Direct}, {8192, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, -1, &MPIR_Scatter_MV2_Binomial}, }, }, {128, 4, { {0, 16, &MPIR_Scatter_MV2_Binomial}, {16, 128, &MPIR_Scatter_MV2_two_level_Binomial}, {128, 16384, &MPIR_Scatter_MV2_two_level_Direct}, {16384, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { {0, 128, &MPIR_Scatter_MV2_Direct}, {128, -1, &MPIR_Scatter_MV2_Binomial}, }, }, }; table_ptrs[1] = mv2_tmp_scatter_thresholds_table_2ppn; mv2_scatter_table_ppn_conf[2] = 16; mv2_size_scatter_tuning_table[2] = 8; mv2_scatter_tuning_table mv2_tmp_scatter_thresholds_table_16ppn[] = { { 16, 2, { {0, 256, &MPIR_Scatter_MV2_Binomial}, {256, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { { 0, -1, &MPIR_Scatter_MV2_Direct}, }, }, { 32, 2, { {0, 512, &MPIR_Scatter_MV2_Binomial}, {512, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { { 0, -1, &MPIR_Scatter_MV2_Direct}, }, }, { 64, 2, { {0, 1024, &MPIR_Scatter_MV2_two_level_Direct}, {1024, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { { 0, -1, &MPIR_Scatter_MV2_Direct}, }, }, { 128, 4, { {0, 16, &MPIR_Scatter_mcst_wrap_MV2}, {0, 16, &MPIR_Scatter_MV2_two_level_Direct}, {16, 2048, &MPIR_Scatter_MV2_two_level_Direct}, {2048, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { { 0, -1, &MPIR_Scatter_MV2_Direct}, }, }, { 256, 4, { {0, 16, &MPIR_Scatter_mcst_wrap_MV2}, {0, 16, &MPIR_Scatter_MV2_two_level_Direct}, {16, 2048, &MPIR_Scatter_MV2_two_level_Direct}, {2048, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { { 0, -1, &MPIR_Scatter_MV2_Direct}, }, }, { 512, 4, { {0, 16, &MPIR_Scatter_mcst_wrap_MV2}, {16, 16, &MPIR_Scatter_MV2_two_level_Direct}, {16, 4096, &MPIR_Scatter_MV2_two_level_Direct}, {4096, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { { 0, -1, &MPIR_Scatter_MV2_Binomial}, }, }, { 1024, 5, { {0, 16, &MPIR_Scatter_mcst_wrap_MV2}, {0, 16, &MPIR_Scatter_MV2_Binomial}, {16, 32, &MPIR_Scatter_MV2_Binomial}, {32, 4096, &MPIR_Scatter_MV2_two_level_Direct}, {4096, -1, &MPIR_Scatter_MV2_Direct}, }, 1, { { 0, -1, &MPIR_Scatter_MV2_Binomial}, }, }, { 2048, 7, { {0, 16, &MPIR_Scatter_mcst_wrap_MV2}, {0, 16, &MPIR_Scatter_MV2_two_level_Binomial}, {16, 128, &MPIR_Scatter_MV2_two_level_Binomial}, {128, 1024, &MPIR_Scatter_MV2_two_level_Direct}, {1024, 16384, &MPIR_Scatter_MV2_two_level_Direct}, {16384, 65536, &MPIR_Scatter_MV2_Direct}, {65536, -1, &MPIR_Scatter_MV2_two_level_Direct}, }, 6, { {0, 16, &MPIR_Scatter_MV2_Binomial}, {16, 128, &MPIR_Scatter_MV2_Binomial}, {128, 1024, &MPIR_Scatter_MV2_Binomial}, {1024, 16384, &MPIR_Scatter_MV2_Direct}, {16384, 65536, &MPIR_Scatter_MV2_Direct}, {65536, -1, &MPIR_Scatter_MV2_Direct}, }, }, }; table_ptrs[2] = mv2_tmp_scatter_thresholds_table_16ppn; agg_table_sum = 0; for (i = 0; i < mv2_scatter_num_ppn_conf; i++) { agg_table_sum += mv2_size_scatter_tuning_table[i]; } mv2_scatter_thresholds_table[0] = xbt_malloc(agg_table_sum * sizeof (mv2_scatter_tuning_table)); memcpy(mv2_scatter_thresholds_table[0], table_ptrs[0], (sizeof(mv2_scatter_tuning_table) * mv2_size_scatter_tuning_table[0])); for (i = 1; i < mv2_scatter_num_ppn_conf; i++) { mv2_scatter_thresholds_table[i] = mv2_scatter_thresholds_table[i - 1] + mv2_size_scatter_tuning_table[i - 1]; memcpy(mv2_scatter_thresholds_table[i], table_ptrs[i], (sizeof(mv2_scatter_tuning_table) * mv2_size_scatter_tuning_table[i])); } xbt_free(table_ptrs); } SimGrid-3.14.159/src/smpi/colls/smpi_automatic_selector.cpp000644 001750 001750 00000007247 13031031272 025377 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include "colls_private.h" //attempt to do a quick autotuning version of the collective, #define TRACE_AUTO_COLL(cat) if (TRACE_is_enabled()){\ type_t type = PJ_type_get_or_null (#cat, PJ_type_get_root());\ if (!type){\ type=PJ_type_event_new(#cat, PJ_type_get_root());\ }\ char cont_name[25];\ snprintf(cont_name,25, "rank-%d", smpi_process_index());\ val_t value = PJ_value_get_or_new(mpi_coll_##cat##_description[i].name,"1.0 1.0 1.0", type);\ new_pajeNewEvent (SIMIX_get_clock(), PJ_container_get(cont_name), type, value);\ } #define AUTOMATIC_COLL_BENCH(cat, ret, args, args2)\ ret smpi_coll_tuned_ ## cat ## _ ## automatic(COLL_UNPAREN args)\ {\ double time1, time2, time_min=DBL_MAX;\ int min_coll=-1, global_coll=-1;\ int i;\ double buf_in, buf_out, max_min=DBL_MAX;\ for (i = 0; mpi_coll_##cat##_description[i].name; i++){\ if(!strcmp(mpi_coll_##cat##_description[i].name, "automatic"))continue;\ if(!strcmp(mpi_coll_##cat##_description[i].name, "default"))continue;\ smpi_mpi_barrier(comm);\ TRACE_AUTO_COLL(cat)\ time1 = SIMIX_get_clock();\ try {\ ((int (*) args)\ mpi_coll_##cat##_description[i].coll) args2 ;\ }\ catch (std::exception& ex) {\ continue;\ }\ time2 = SIMIX_get_clock();\ buf_out=time2-time1;\ smpi_mpi_reduce((void*)&buf_out,(void*)&buf_in, 1, MPI_DOUBLE, MPI_MAX, 0,comm );\ if(time2-time1= (TYPELNG)) && \ ((SEGSIZE) < ((TYPELNG) * (SEGCOUNT))) ) { \ size_t residual; \ (SEGCOUNT) = (int)((SEGSIZE) / (TYPELNG)); \ residual = (SEGSIZE) - (SEGCOUNT) * (TYPELNG); \ if( residual > ((TYPELNG) >> 1) ) \ (SEGCOUNT)++; \ } \ #define COLL_TUNED_COMPUTE_BLOCKCOUNT( COUNT, NUM_BLOCKS, SPLIT_INDEX, \ EARLY_BLOCK_COUNT, LATE_BLOCK_COUNT ) \ EARLY_BLOCK_COUNT = LATE_BLOCK_COUNT = COUNT / NUM_BLOCKS; \ SPLIT_INDEX = COUNT % NUM_BLOCKS; \ if (0 != SPLIT_INDEX) { \ EARLY_BLOCK_COUNT = EARLY_BLOCK_COUNT + 1; \ } \ #include "colls_private.h" int smpi_coll_tuned_allreduce_ompi_ring_segmented(void *sbuf, void *rbuf, int count, MPI_Datatype dtype, MPI_Op op, MPI_Comm comm) { int ret = MPI_SUCCESS; int line; int k, recv_from, send_to; int early_blockcount, late_blockcount, split_rank; int segcount, max_segcount; int num_phases, phase; int block_count; unsigned int inbi; size_t typelng; char *tmpsend = NULL, *tmprecv = NULL; char *inbuf[2] = {NULL, NULL}; ptrdiff_t true_extent, extent; ptrdiff_t block_offset, max_real_segsize; MPI_Request reqs[2] = {NULL, NULL}; const size_t segsize = 1 << 20; /* 1 MB */ unsigned int size = smpi_comm_size(comm); unsigned int rank = smpi_comm_rank(comm); XBT_DEBUG("coll:tuned:allreduce_intra_ring_segmented rank %d, count %d", rank, count); /* Special case for size == 1 */ if (1 == size) { if (MPI_IN_PLACE != sbuf) { ret= smpi_datatype_copy(sbuf, count, dtype,rbuf, count, dtype); if (ret < 0) { line = __LINE__; goto error_hndl; } } return MPI_SUCCESS; } /* Determine segment count based on the suggested segment size */ extent = smpi_datatype_get_extent(dtype); if (MPI_SUCCESS != ret) { line = __LINE__; goto error_hndl; } true_extent = smpi_datatype_get_extent(dtype); if (MPI_SUCCESS != ret) { line = __LINE__; goto error_hndl; } typelng = smpi_datatype_size(dtype); if (MPI_SUCCESS != ret) { line = __LINE__; goto error_hndl; } segcount = count; COLL_TUNED_COMPUTED_SEGCOUNT(segsize, typelng, segcount) /* Special case for count less than size * segcount - use regular ring */ if (count < size * segcount) { XBT_DEBUG( "coll:tuned:allreduce_ring_segmented rank %d/%d, count %d, switching to regular ring", rank, size, count); return (smpi_coll_tuned_allreduce_lr(sbuf, rbuf, count, dtype, op, comm)); } /* Determine the number of phases of the algorithm */ num_phases = count / (size * segcount); if ((count % (size * segcount) >= size) && (count % (size * segcount) > ((size * segcount) / 2))) { num_phases++; } /* Determine the number of elements per block and corresponding block sizes. The blocks are divided into "early" and "late" ones: blocks 0 .. (split_rank - 1) are "early" and blocks (split_rank) .. (size - 1) are "late". Early blocks are at most 1 element larger than the late ones. Note, these blocks will be split into num_phases segments, out of the largest one will have max_segcount elements. */ COLL_TUNED_COMPUTE_BLOCKCOUNT( count, size, split_rank, early_blockcount, late_blockcount ) COLL_TUNED_COMPUTE_BLOCKCOUNT( early_blockcount, num_phases, inbi, max_segcount, k) max_real_segsize = true_extent + (max_segcount - 1) * extent; /* Allocate and initialize temporary buffers */ inbuf[0] = (char*)smpi_get_tmp_sendbuffer(max_real_segsize); if (NULL == inbuf[0]) { ret = -1; line = __LINE__; goto error_hndl; } if (size > 2) { inbuf[1] = (char*)smpi_get_tmp_recvbuffer(max_real_segsize); if (NULL == inbuf[1]) { ret = -1; line = __LINE__; goto error_hndl; } } /* Handle MPI_IN_PLACE */ if (MPI_IN_PLACE != sbuf) { ret= smpi_datatype_copy(sbuf, count, dtype,rbuf, count, dtype); if (ret < 0) { line = __LINE__; goto error_hndl; } } /* Computation loop: for each phase, repeat ring allreduce computation loop */ for (phase = 0; phase < num_phases; phase ++) { ptrdiff_t phase_offset; int early_phase_segcount, late_phase_segcount, split_phase, phase_count; /* For each of the remote nodes: - post irecv for block (r-1) - send block (r) To do this, first compute block offset and count, and use block offset to compute phase offset. - in loop for every step k = 2 .. n - post irecv for block (r + n - k) % n - wait on block (r + n - k + 1) % n to arrive - compute on block (r + n - k + 1) % n - send block (r + n - k + 1) % n - wait on block (r + 1) - compute on block (r + 1) - send block (r + 1) to rank (r + 1) Note that we must be careful when computing the begining of buffers and for send operations and computation we must compute the exact block size. */ send_to = (rank + 1) % size; recv_from = (rank + size - 1) % size; inbi = 0; /* Initialize first receive from the neighbor on the left */ reqs[inbi] = smpi_mpi_irecv(inbuf[inbi], max_segcount, dtype, recv_from, 666, comm); /* Send first block (my block) to the neighbor on the right: - compute my block and phase offset - send data */ block_offset = ((rank < split_rank)? (rank * early_blockcount) : (rank * late_blockcount + split_rank)); block_count = ((rank < split_rank)? early_blockcount : late_blockcount); COLL_TUNED_COMPUTE_BLOCKCOUNT(block_count, num_phases, split_phase, early_phase_segcount, late_phase_segcount) phase_count = ((phase < split_phase)? (early_phase_segcount) : (late_phase_segcount)); phase_offset = ((phase < split_phase)? (phase * early_phase_segcount) : (phase * late_phase_segcount + split_phase)); tmpsend = ((char*)rbuf) + (block_offset + phase_offset) * extent; smpi_mpi_send(tmpsend, phase_count, dtype, send_to, 666, comm); for (k = 2; k < size; k++) { const int prevblock = (rank + size - k + 1) % size; inbi = inbi ^ 0x1; /* Post irecv for the current block */ reqs[inbi] = smpi_mpi_irecv(inbuf[inbi], max_segcount, dtype, recv_from, 666, comm); if (MPI_SUCCESS != ret) { line = __LINE__; goto error_hndl; } /* Wait on previous block to arrive */ smpi_mpi_wait(&reqs[inbi ^ 0x1], MPI_STATUS_IGNORE); /* Apply operation on previous block: result goes to rbuf rbuf[prevblock] = inbuf[inbi ^ 0x1] (op) rbuf[prevblock] */ block_offset = ((prevblock < split_rank)? (prevblock * early_blockcount) : (prevblock * late_blockcount + split_rank)); block_count = ((prevblock < split_rank)? early_blockcount : late_blockcount); COLL_TUNED_COMPUTE_BLOCKCOUNT(block_count, num_phases, split_phase, early_phase_segcount, late_phase_segcount) phase_count = ((phase < split_phase)? (early_phase_segcount) : (late_phase_segcount)); phase_offset = ((phase < split_phase)? (phase * early_phase_segcount) : (phase * late_phase_segcount + split_phase)); tmprecv = ((char*)rbuf) + (block_offset + phase_offset) * extent; smpi_op_apply(op, inbuf[inbi ^ 0x1], tmprecv, &phase_count, &dtype); /* send previous block to send_to */ smpi_mpi_send(tmprecv, phase_count, dtype, send_to, 666, comm); } /* Wait on the last block to arrive */ smpi_mpi_wait(&reqs[inbi], MPI_STATUS_IGNORE); /* Apply operation on the last block (from neighbor (rank + 1) rbuf[rank+1] = inbuf[inbi] (op) rbuf[rank + 1] */ recv_from = (rank + 1) % size; block_offset = ((recv_from < split_rank)? (recv_from * early_blockcount) : (recv_from * late_blockcount + split_rank)); block_count = ((recv_from < split_rank)? early_blockcount : late_blockcount); COLL_TUNED_COMPUTE_BLOCKCOUNT(block_count, num_phases, split_phase, early_phase_segcount, late_phase_segcount) phase_count = ((phase < split_phase)? (early_phase_segcount) : (late_phase_segcount)); phase_offset = ((phase < split_phase)? (phase * early_phase_segcount) : (phase * late_phase_segcount + split_phase)); tmprecv = ((char*)rbuf) + (block_offset + phase_offset) * extent; smpi_op_apply(op, inbuf[inbi], tmprecv, &phase_count, &dtype); } /* Distribution loop - variation of ring allgather */ send_to = (rank + 1) % size; recv_from = (rank + size - 1) % size; for (k = 0; k < size - 1; k++) { const int recv_data_from = (rank + size - k) % size; const int send_data_from = (rank + 1 + size - k) % size; const int send_block_offset = ((send_data_from < split_rank)? (send_data_from * early_blockcount) : (send_data_from * late_blockcount + split_rank)); const int recv_block_offset = ((recv_data_from < split_rank)? (recv_data_from * early_blockcount) : (recv_data_from * late_blockcount + split_rank)); block_count = ((send_data_from < split_rank)? early_blockcount : late_blockcount); tmprecv = (char*)rbuf + recv_block_offset * extent; tmpsend = (char*)rbuf + send_block_offset * extent; smpi_mpi_sendrecv(tmpsend, block_count, dtype, send_to, 666, tmprecv, early_blockcount, dtype, recv_from, 666, comm, MPI_STATUS_IGNORE); } if (NULL != inbuf[0]) smpi_free_tmp_buffer(inbuf[0]); if (NULL != inbuf[1]) smpi_free_tmp_buffer(inbuf[1]); return MPI_SUCCESS; error_hndl: XBT_DEBUG("%s:%4d\tRank %d Error occurred %d\n", __FILE__, line, rank, ret); if (NULL != inbuf[0]) smpi_free_tmp_buffer(inbuf[0]); if (NULL != inbuf[1]) smpi_free_tmp_buffer(inbuf[1]); return ret; } SimGrid-3.14.159/src/smpi/smpi_topo.cpp000644 001750 001750 00000034616 13031031272 021356 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "xbt/sysdep.h" #include "smpi/smpi.h" #include "private.h" #include typedef struct s_smpi_mpi_cart_topology { int nnodes; int ndims; int *dims; int *periodic; int *position; } s_smpi_mpi_cart_topology_t; typedef struct s_smpi_mpi_graph_topology { int nnodes; int nedges; int *index; int *edges; } s_smpi_mpi_graph_topology_t; typedef struct s_smpi_dist_graph_topology { int indegree; int *in; int *in_weights; int outdegree; int *out; int *out_weights; int is_weighted; } s_smpi_mpi_dist_graph_topology_t; typedef struct s_smpi_mpi_topology { MPIR_Topo_type kind; union topo { MPIR_Graph_Topology graph; MPIR_Cart_Topology cart; MPIR_Dist_Graph_Topology dist_graph; } topo; } s_smpi_mpi_topology_t; void smpi_topo_destroy(MPI_Topology topo) { if(topo == nullptr) { return; } switch (topo->kind) { case MPI_GRAPH: // Not implemented // smpi_graph_topo_destroy(topo->topo.graph); break; case MPI_CART: smpi_cart_topo_destroy(topo->topo.cart); break; case MPI_DIST_GRAPH: // Not implemented // smpi_dist_graph_topo_destroy(topo->topo.dist_graph); break; default: return; break; } xbt_free(topo); } MPI_Topology smpi_topo_create(MPIR_Topo_type kind) { MPI_Topology newTopo = static_cast(xbt_malloc(sizeof(*newTopo))); newTopo->kind = kind; // Allocate and initialize the right topo should be done by the caller return newTopo; } /******************************************************************************* * Cartesian topologies ******************************************************************************/ void smpi_cart_topo_destroy(MPIR_Cart_Topology cart) { if (cart) { if(cart->dims) { xbt_free(cart->dims); } if(cart->periodic) { xbt_free(cart->periodic); } if(cart->position) { xbt_free(cart->position); } xbt_free(cart); } } MPI_Topology smpi_cart_topo_create(int ndims) { MPI_Topology newTopo = smpi_topo_create(MPI_CART); MPIR_Cart_Topology newCart = static_cast(xbt_malloc(sizeof(*newCart))); newCart->nnodes = 0; newCart->ndims = ndims; newCart->dims = xbt_new(int, ndims); newCart->periodic = xbt_new(int, ndims); newCart->position = xbt_new(int, ndims); newTopo->topo.cart = newCart; return newTopo; } /* reorder is ignored, don't know what would be the consequences of a dumb * reordering but neither do I see the point of reordering*/ int smpi_mpi_cart_create(MPI_Comm comm_old, int ndims, int dims[], int periods[], int reorder, MPI_Comm *comm_cart) { int retval = MPI_SUCCESS; int i; MPI_Topology newCart; MPI_Group newGroup, oldGroup; int rank, nranks, newSize; rank = smpi_comm_rank(comm_old); newSize = 1; if(ndims != 0) { for (i = 0 ; i < ndims ; i++) { newSize *= dims[i]; } if(rank >= newSize) { *comm_cart = MPI_COMM_NULL; return retval; } newCart = smpi_cart_topo_create(ndims); oldGroup = smpi_comm_group(comm_old); newGroup = smpi_group_new(newSize); for (i = 0 ; i < newSize ; i++) { smpi_group_set_mapping(newGroup, smpi_group_index(oldGroup, i), i); } newCart->topo.cart->nnodes = newSize; // FIXME : code duplication... See smpi_mpi_cart_coords nranks = newSize; for (i=0; itopo.cart->dims[i] = dims[i]; newCart->topo.cart->periodic[i] = periods[i]; nranks = nranks / dims[i]; /* FIXME: nranks could be zero (?) */ newCart->topo.cart->position[i] = rank / nranks; rank = rank % nranks; } *comm_cart = smpi_comm_new(newGroup, newCart); } else { if (rank == 0) { newCart = smpi_cart_topo_create(ndims); *comm_cart = smpi_comm_new(smpi_group_copy(smpi_comm_group(MPI_COMM_SELF)), newCart); } else { *comm_cart = MPI_COMM_NULL; } } return retval; } int smpi_mpi_cart_sub(MPI_Comm comm, const int remain_dims[], MPI_Comm *newcomm) { MPI_Topology oldTopo = smpi_comm_topo(comm); int oldNDims = oldTopo->topo.cart->ndims; int i, j = 0, newNDims, *newDims = nullptr, *newPeriodic = nullptr; if (remain_dims == nullptr && oldNDims != 0) { return MPI_ERR_ARG; } newNDims = 0; for (i = 0 ; i < oldNDims ; i++) { if (remain_dims[i]) newNDims++; } if (newNDims > 0) { newDims = xbt_new(int, newNDims); newPeriodic = xbt_new(int, newNDims); // that should not segfault for (i = 0 ; j < newNDims ; i++) { if(remain_dims[i]) { newDims[j] = oldTopo->topo.cart->dims[i]; newPeriodic[j] = oldTopo->topo.cart->periodic[i]; j++; } } } return smpi_mpi_cart_create(comm, newNDims, newDims, newPeriodic, 0, newcomm); } int smpi_mpi_cart_coords(MPI_Comm comm, int rank, int maxdims, int coords[]) { int nnodes; MPI_Topology topo = smpi_comm_topo(comm); nnodes = topo->topo.cart->nnodes; for (int i=0; i < topo->topo.cart->ndims; i++ ) { nnodes = nnodes / topo->topo.cart->dims[i]; coords[i] = rank / nnodes; rank = rank % nnodes; } return MPI_SUCCESS; } int smpi_mpi_cart_get(MPI_Comm comm, int maxdims, int* dims, int* periods, int* coords) { MPI_Topology topo = smpi_comm_topo(comm); int ndims=topo->topo.cart->ndims < maxdims ? topo->topo.cart->ndims : maxdims; for(int i = 0 ; i < ndims ; i++) { dims[i] = topo->topo.cart->dims[i]; periods[i] = topo->topo.cart->periodic[i]; coords[i] = topo->topo.cart->position[i]; } return MPI_SUCCESS; } int smpi_mpi_cart_rank(MPI_Comm comm, int* coords, int* rank) { MPI_Topology topo = smpi_comm_topo(comm); int ndims = topo->topo.cart->ndims; int multiplier, coord,i; *rank = 0; multiplier = 1; for ( i=ndims-1; i >=0; i-- ) { coord = coords[i]; /* The user can give us whatever coordinates he wants. If one of them is out of range, either this dimension is * periodic, and we consider the equivalent coordinate inside the bounds, or it's not and then it's an error */ if (coord >= topo->topo.cart->dims[i]) { if ( topo->topo.cart->periodic[i] ) { coord = coord % topo->topo.cart->dims[i]; } else { // Should I do that ? *rank = -1; return MPI_ERR_ARG; } } else if (coord < 0) { if(topo->topo.cart->periodic[i]) { coord = coord % topo->topo.cart->dims[i]; if (coord) coord = topo->topo.cart->dims[i] + coord; } else { *rank = -1; return MPI_ERR_ARG; } } *rank += multiplier * coord; multiplier *= topo->topo.cart->dims[i]; } return MPI_SUCCESS; } int smpi_mpi_cart_shift(MPI_Comm comm, int direction, int disp, int *rank_source, int *rank_dest) { MPI_Topology topo = smpi_comm_topo(comm); int position[topo->topo.cart->ndims]; if(topo->topo.cart->ndims == 0) { return MPI_ERR_ARG; } if (topo->topo.cart->ndims < direction) { return MPI_ERR_DIMS; } smpi_mpi_cart_coords(comm, smpi_comm_rank(comm), topo->topo.cart->ndims, position); position[direction] += disp; if(position[direction] < 0 || position[direction] >= topo->topo.cart->dims[direction]) { if(topo->topo.cart->periodic[direction]) { position[direction] %= topo->topo.cart->dims[direction]; smpi_mpi_cart_rank(comm, position, rank_dest); } else { *rank_dest = MPI_PROC_NULL; } } else { smpi_mpi_cart_rank(comm, position, rank_dest); } position[direction] = topo->topo.cart->position[direction] - disp; if(position[direction] < 0 || position[direction] >= topo->topo.cart->dims[direction]) { if(topo->topo.cart->periodic[direction]) { position[direction] %= topo->topo.cart->dims[direction]; smpi_mpi_cart_rank(comm, position, rank_source); } else { *rank_source = MPI_PROC_NULL; } } else { smpi_mpi_cart_rank(comm, position, rank_source); } return MPI_SUCCESS; } int smpi_mpi_cartdim_get(MPI_Comm comm, int *ndims) { MPI_Topology topo = smpi_comm_topo(comm); *ndims = topo->topo.cart->ndims; return MPI_SUCCESS; } // Everything below has been taken from ompi, but could be easily rewritten. /* * Copyright (c) 2004-2007 The Trustees of Indiana University and Indiana * University Research and Technology * Corporation. All rights reserved. * Copyright (c) 2004-2005 The University of Tennessee and The University * of Tennessee Research Foundation. All rights * reserved. * Copyright (c) 2004-2014 High Performance Computing Center Stuttgart, * University of Stuttgart. All rights reserved. * Copyright (c) 2004-2005 The Regents of the University of California. * All rights reserved. * Copyright (c) 2012 Los Alamos National Security, LLC. All rights * reserved. * Copyright (c) 2014 Intel, Inc. All rights reserved * $COPYRIGHT$ * * Additional copyrights may follow * * $HEADER$ */ /* static functions */ static int assignnodes(int ndim, int nfactor, int *pfacts,int **pdims); static int getfactors(int num, int *nfators, int **factors); /* * This is a utility function, no need to have anything in the lower * layer for this at all */ int smpi_mpi_dims_create(int nnodes, int ndims, int dims[]) { int i; int freeprocs; int freedims; int nfactors; int *factors; int *procs; int *p; int err; /* Get # of free-to-be-assigned processes and # of free dimensions */ freeprocs = nnodes; freedims = 0; for (i = 0, p = dims; i < ndims; ++i,++p) { if (*p == 0) { ++freedims; } else if ((*p < 0) || ((nnodes % *p) != 0)) { return MPI_ERR_DIMS; } else { freeprocs /= *p; } } if (freedims == 0) { if (freeprocs == 1) { return MPI_SUCCESS; } return MPI_ERR_DIMS; } if (freeprocs == 1) { for (i = 0; i < ndims; ++i, ++dims) { if (*dims == 0) { *dims = 1; } } return MPI_SUCCESS; } /* Factor the number of free processes */ if (MPI_SUCCESS != (err = getfactors(freeprocs, &nfactors, &factors))) { return err; } /* Assign free processes to free dimensions */ if (MPI_SUCCESS != (err = assignnodes(freedims, nfactors, factors, &procs))) { return err; } /* Return assignment results */ p = procs; for (i = 0; i < ndims; ++i, ++dims) { if (*dims == 0) { *dims = *p++; } } free((char *) factors); free((char *) procs); /* all done */ return MPI_SUCCESS; } /* * assignnodes * * Function: - assign processes to dimensions * - get "best-balanced" grid * - greedy bin-packing algorithm used * - sort dimensions in decreasing order * - dimensions array dynamically allocated * Accepts: - # of dimensions * - # of prime factors * - array of prime factors * - ptr to array of dimensions (returned value) * Returns: - 0 or ERROR */ static int assignnodes(int ndim, int nfactor, int *pfacts, int **pdims) { int *bins; int i, j; int n; int f; int *p; int *pmin; if (0 >= ndim) { return MPI_ERR_DIMS; } /* Allocate and initialize the bins */ bins = (int *) malloc((unsigned) ndim * sizeof(int)); if (nullptr == bins) { return MPI_ERR_NO_MEM; } *pdims = bins; for (i = 0, p = bins; i < ndim; ++i, ++p) { *p = 1; } /* Loop assigning factors from the highest to the lowest */ for (j = nfactor - 1; j >= 0; --j) { f = pfacts[j]; /* Assign a factor to the smallest bin */ pmin = bins; for (i = 1, p = pmin + 1; i < ndim; ++i, ++p) { if (*p < *pmin) { pmin = p; } } *pmin *= f; } /* Sort dimensions in decreasing order (O(n^2) for now) */ for (i = 0, pmin = bins; i < ndim - 1; ++i, ++pmin) { for (j = i + 1, p = pmin + 1; j < ndim; ++j, ++p) { if (*p > *pmin) { n = *p; *p = *pmin; *pmin = n; } } } return MPI_SUCCESS; } /* * getfactors * * Function: - factorize a number * Accepts: - number * - # prime factors * - array of prime factors * Returns: - MPI_SUCCESS or ERROR */ static int getfactors(int num, int *nfactors, int **factors) { int size; int d; int i; int sqrtnum; if(num < 2) { (*nfactors) = 0; (*factors) = nullptr; return MPI_SUCCESS; } /* Allocate the array of prime factors which cannot exceed log_2(num) entries */ sqrtnum = ceil(sqrt(num)); size = ceil(log(num) / log(2)); *factors = (int *) malloc((unsigned) size * sizeof(int)); i = 0; /* determine all occurences of factor 2 */ while((num % 2) == 0) { num /= 2; (*factors)[i++] = 2; } /* determine all occurences of uneven prime numbers up to sqrt(num) */ for(d = 3; (num > 1) && (d < sqrtnum); d += 2) { while((num % d) == 0) { num /= d; (*factors)[i++] = d; } } /* as we looped only up to sqrt(num) one factor > sqrt(num) may be left over */ if(num != 1) { (*factors)[i++] = num; } (*nfactors) = i; return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/smpi_comm.cpp000644 001750 001750 00000045073 13031031272 021327 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include #include #include "private.h" #include "smpi_mpi_dt_private.h" #include "src/simix/smx_private.h" #include "colls/colls.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_comm, smpi, "Logging specific to SMPI (comm)"); xbt_dict_t smpi_comm_keyvals = nullptr; int comm_keyval_id = 0;//avoid collisions /* Support for cartesian topology was added, but there are 2 other types of topology, graph et dist graph. In order to * support them, we have to add a field MPIR_Topo_type, and replace the MPI_Topology field by an union. */ typedef struct s_smpi_mpi_communicator { MPI_Group group; MPIR_Topo_type topoType; MPI_Topology topo; // to be replaced by an union int refcount; MPI_Comm leaders_comm;//inter-node communicator MPI_Comm intra_comm;//intra-node communicator . For MPI_COMM_WORLD this can't be used, as var is global. //use an intracomm stored in the process data instead int* leaders_map; //who is the leader of each process int is_uniform; int* non_uniform_map; //set if smp nodes have a different number of processes allocated int is_blocked;// are ranks allocated on the same smp node contiguous ? xbt_dict_t attributes; } s_smpi_mpi_communicator_t; static int smpi_compare_rankmap(const void *a, const void *b) { const int* x = static_cast(a); const int* y = static_cast(b); if (x[1] < y[1]) { return -1; } if (x[1] == y[1]) { if (x[0] < y[0]) { return -1; } if (x[0] == y[0]) { return 0; } return 1; } return 1; } MPI_Comm smpi_comm_new(MPI_Group group, MPI_Topology topo) { MPI_Comm comm; comm = xbt_new(s_smpi_mpi_communicator_t, 1); comm->group = group; comm->refcount=1; comm->topoType = MPI_INVALID_TOPO; comm->topo = topo; comm->intra_comm = MPI_COMM_NULL; comm->leaders_comm = MPI_COMM_NULL; comm->is_uniform=1; comm->non_uniform_map = nullptr; comm->leaders_map = nullptr; comm->is_blocked=0; comm->attributes=nullptr; return comm; } void smpi_comm_destroy(MPI_Comm comm) { if (comm == MPI_COMM_UNINITIALIZED) comm = smpi_process_comm_world(); smpi_topo_destroy(comm->topo); // there's no use count on topos smpi_comm_unuse(comm); } int smpi_comm_dup(MPI_Comm comm, MPI_Comm* newcomm){ if(smpi_privatize_global_variables){ //we need to switch as the called function may silently touch global variables smpi_switch_data_segment(smpi_process_index()); } MPI_Group cp=smpi_group_copy(smpi_comm_group(comm)); (*newcomm) = smpi_comm_new(cp, smpi_comm_topo(comm)); int ret = MPI_SUCCESS; if(comm->attributes !=nullptr){ (*newcomm)->attributes = xbt_dict_new_homogeneous(nullptr); xbt_dict_cursor_t cursor = nullptr; int* key; int flag; void* value_in; void* value_out; xbt_dict_foreach (comm->attributes, cursor, key, value_in) { smpi_comm_key_elem elem = static_cast( xbt_dict_get_or_null_ext(smpi_comm_keyvals, reinterpret_cast(key), sizeof(int))); if (elem != nullptr && elem->copy_fn != MPI_NULL_COPY_FN) { ret = elem->copy_fn(comm, *key, nullptr, value_in, &value_out, &flag); if (ret != MPI_SUCCESS) { smpi_comm_destroy(*newcomm); *newcomm = MPI_COMM_NULL; xbt_dict_cursor_free(&cursor); return ret; } if (flag) xbt_dict_set_ext((*newcomm)->attributes, reinterpret_cast(key), sizeof(int), value_out, nullptr); } } } return ret; } MPI_Group smpi_comm_group(MPI_Comm comm) { if (comm == MPI_COMM_UNINITIALIZED) comm = smpi_process_comm_world(); return comm->group; } MPI_Topology smpi_comm_topo(MPI_Comm comm) { if (comm != MPI_COMM_NULL) return comm->topo; return nullptr; } int smpi_comm_size(MPI_Comm comm) { if (comm == MPI_COMM_UNINITIALIZED) comm = smpi_process_comm_world(); return smpi_group_size(smpi_comm_group(comm)); } int smpi_comm_rank(MPI_Comm comm) { if (comm == MPI_COMM_UNINITIALIZED) comm = smpi_process_comm_world(); return smpi_group_rank(smpi_comm_group(comm), smpi_process_index()); } void smpi_comm_get_name (MPI_Comm comm, char* name, int* len) { if (comm == MPI_COMM_UNINITIALIZED) comm = smpi_process_comm_world(); if(comm == MPI_COMM_WORLD) { strncpy(name, "WORLD",5); *len = 5; } else { *len = snprintf(name, MPI_MAX_NAME_STRING, "%p", comm); } } void smpi_comm_set_leaders_comm(MPI_Comm comm, MPI_Comm leaders){ if (comm == MPI_COMM_UNINITIALIZED) comm = smpi_process_comm_world(); comm->leaders_comm=leaders; } void smpi_comm_set_intra_comm(MPI_Comm comm, MPI_Comm leaders){ comm->intra_comm=leaders; } int* smpi_comm_get_non_uniform_map(MPI_Comm comm){ if (comm == MPI_COMM_UNINITIALIZED) comm = smpi_process_comm_world(); return comm->non_uniform_map; } int* smpi_comm_get_leaders_map(MPI_Comm comm){ if (comm == MPI_COMM_UNINITIALIZED) comm = smpi_process_comm_world(); return comm->leaders_map; } MPI_Comm smpi_comm_get_leaders_comm(MPI_Comm comm){ if (comm == MPI_COMM_UNINITIALIZED) comm = smpi_process_comm_world(); return comm->leaders_comm; } MPI_Comm smpi_comm_get_intra_comm(MPI_Comm comm){ if (comm == MPI_COMM_UNINITIALIZED || comm==MPI_COMM_WORLD) return smpi_process_get_comm_intra(); else return comm->intra_comm; } int smpi_comm_is_uniform(MPI_Comm comm){ if (comm == MPI_COMM_UNINITIALIZED) comm = smpi_process_comm_world(); return comm->is_uniform; } int smpi_comm_is_blocked(MPI_Comm comm){ if (comm == MPI_COMM_UNINITIALIZED) comm = smpi_process_comm_world(); return comm->is_blocked; } MPI_Comm smpi_comm_split(MPI_Comm comm, int color, int key) { if (comm == MPI_COMM_UNINITIALIZED) comm = smpi_process_comm_world(); int system_tag = 123; int index, rank, size, i, j, count, reqs; int* sendbuf; int* recvbuf; int* rankmap; MPI_Group group, group_root, group_out; MPI_Group* group_snd; MPI_Request* requests; group_root = nullptr; group_out = nullptr; group = smpi_comm_group(comm); rank = smpi_comm_rank(comm); size = smpi_comm_size(comm); /* Gather all colors and keys on rank 0 */ sendbuf = xbt_new(int, 2); sendbuf[0] = color; sendbuf[1] = key; if(rank == 0) { recvbuf = xbt_new(int, 2 * size); } else { recvbuf = nullptr; } smpi_mpi_gather(sendbuf, 2, MPI_INT, recvbuf, 2, MPI_INT, 0, comm); xbt_free(sendbuf); /* Do the actual job */ if(rank == 0) { group_snd = xbt_new(MPI_Group, size); rankmap = xbt_new(int, 2 * size); for(i = 0; i < size; i++) { if(recvbuf[2 * i] != MPI_UNDEFINED) { count = 0; for(j = i + 1; j < size; j++) { if(recvbuf[2 * i] == recvbuf[2 * j]) { recvbuf[2 * j] = MPI_UNDEFINED; rankmap[2 * count] = j; rankmap[2 * count + 1] = recvbuf[2 * j + 1]; count++; } } /* Add self in the group */ recvbuf[2 * i] = MPI_UNDEFINED; rankmap[2 * count] = i; rankmap[2 * count + 1] = recvbuf[2 * i + 1]; count++; qsort(rankmap, count, 2 * sizeof(int), &smpi_compare_rankmap); group_out = smpi_group_new(count); if(i == 0) { group_root = group_out; /* Save root's group */ } for(j = 0; j < count; j++) { index = smpi_group_index(group, rankmap[2 * j]); smpi_group_set_mapping(group_out, index, j); } requests = xbt_new(MPI_Request, count); reqs = 0; for(j = 0; j < count; j++) { if(rankmap[2 * j] != 0) { group_snd[reqs]=smpi_group_copy(group_out); requests[reqs] = smpi_mpi_isend(&(group_snd[reqs]), 1, MPI_PTR, rankmap[2 * j], system_tag, comm); reqs++; } } if(i != 0) { smpi_group_destroy(group_out); } smpi_mpi_waitall(reqs, requests, MPI_STATUS_IGNORE); xbt_free(requests); } } xbt_free(recvbuf); xbt_free(rankmap); xbt_free(group_snd); group_out = group_root; /* exit with root's group */ } else { if(color != MPI_UNDEFINED) { smpi_mpi_recv(&group_out, 1, MPI_PTR, 0, system_tag, comm, MPI_STATUS_IGNORE); } /* otherwise, exit with group_out == nullptr */ } return group_out!=nullptr ? smpi_comm_new(group_out, nullptr) : MPI_COMM_NULL; } void smpi_comm_use(MPI_Comm comm){ if (comm == MPI_COMM_UNINITIALIZED) comm = smpi_process_comm_world(); smpi_group_use(comm->group); comm->refcount++; } void smpi_comm_cleanup_attributes(MPI_Comm comm){ if(comm->attributes !=nullptr){ xbt_dict_cursor_t cursor = nullptr; int* key; void * value; int flag; xbt_dict_foreach(comm->attributes, cursor, key, value){ smpi_comm_key_elem elem = static_cast(xbt_dict_get_or_null(smpi_comm_keyvals, reinterpret_cast(key))); if(elem!=nullptr && elem->delete_fn!=nullptr) elem->delete_fn(comm, *key, value, &flag); } xbt_dict_free(&comm->attributes); } } void smpi_comm_cleanup_smp(MPI_Comm comm){ if(comm->intra_comm != MPI_COMM_NULL) smpi_comm_unuse(comm->intra_comm); if(comm->leaders_comm != MPI_COMM_NULL) smpi_comm_unuse(comm->leaders_comm); if(comm->non_uniform_map !=nullptr) xbt_free(comm->non_uniform_map); if(comm->leaders_map !=nullptr) xbt_free(comm->leaders_map); } void smpi_comm_unuse(MPI_Comm comm){ if (comm == MPI_COMM_UNINITIALIZED) comm = smpi_process_comm_world(); comm->refcount--; smpi_group_unuse(comm->group); if(comm->refcount==0){ smpi_comm_cleanup_smp(comm); smpi_comm_cleanup_attributes(comm); xbt_free(comm); } } static int compare_ints (const void *a, const void *b) { const int *da = static_cast(a); const int *db = static_cast(b); return static_cast(*da > *db) - static_cast(*da < *db); } void smpi_comm_init_smp(MPI_Comm comm){ int leader = -1; if (comm == MPI_COMM_UNINITIALIZED) comm = smpi_process_comm_world(); int comm_size =smpi_comm_size(comm); // If we are in replay - perform an ugly hack // tell SimGrid we are not in replay for a while, because we need the buffers to be copied for the following calls bool replaying = false; //cache data to set it back again after if(smpi_process_get_replaying()){ replaying=true; smpi_process_set_replaying(false); } if(smpi_privatize_global_variables){ //we need to switch as the called function may silently touch global variables smpi_switch_data_segment(smpi_process_index()); } //identify neighbours in comm //get the indexes of all processes sharing the same simix host xbt_swag_t process_list = SIMIX_host_self()->processes(); int intra_comm_size = 0; int i =0; int min_index=INT_MAX;//the minimum index will be the leader smx_actor_t process = nullptr; xbt_swag_foreach(process, process_list) { int index = process->pid -1; if(smpi_group_rank(smpi_comm_group(comm), index)!=MPI_UNDEFINED){ intra_comm_size++; //the process is in the comm if(index < min_index) min_index=index; i++; } } XBT_DEBUG("number of processes deployed on my node : %d", intra_comm_size); MPI_Group group_intra = smpi_group_new(intra_comm_size); i=0; process = nullptr; xbt_swag_foreach(process, process_list) { int index = process->pid -1; if(smpi_group_rank(smpi_comm_group(comm), index)!=MPI_UNDEFINED){ smpi_group_set_mapping(group_intra, index, i); i++; } } MPI_Comm comm_intra = smpi_comm_new(group_intra, nullptr); leader=min_index; int * leaders_map= static_cast(xbt_malloc0(sizeof(int)*comm_size)); int * leader_list= static_cast(xbt_malloc0(sizeof(int)*comm_size)); for(i=0; ileaders_map==nullptr){ comm->leaders_map= leaders_map; }else{ xbt_free(leaders_map); } int j=0; int leader_group_size = 0; for(i=0; ileaders_map[i]==leader_list[j]){ already_done=1; } } if(already_done==0){ leader_list[leader_group_size]=comm->leaders_map[i]; leader_group_size++; } } qsort(leader_list, leader_group_size, sizeof(int),compare_ints); MPI_Group leaders_group = smpi_group_new(leader_group_size); MPI_Comm leader_comm = MPI_COMM_NULL; if(MPI_COMM_WORLD!=MPI_COMM_UNINITIALIZED && comm!=MPI_COMM_WORLD){ //create leader_communicator for (i=0; i< leader_group_size;i++) smpi_group_set_mapping(leaders_group, leader_list[i], i); leader_comm = smpi_comm_new(leaders_group, nullptr); smpi_comm_set_leaders_comm(comm, leader_comm); smpi_comm_set_intra_comm(comm, comm_intra); //create intracommunicator }else{ for (i=0; i< leader_group_size;i++) smpi_group_set_mapping(leaders_group, leader_list[i], i); if(smpi_comm_get_leaders_comm(comm)==MPI_COMM_NULL){ leader_comm = smpi_comm_new(leaders_group, nullptr); smpi_comm_set_leaders_comm(comm, leader_comm); }else{ leader_comm=smpi_comm_get_leaders_comm(comm); smpi_group_unuse(leaders_group); } smpi_process_set_comm_intra(comm_intra); } int is_uniform = 1; // Are the nodes uniform ? = same number of process/node int my_local_size=smpi_comm_size(comm_intra); if(smpi_comm_rank(comm_intra)==0) { int* non_uniform_map = xbt_new0(int,leader_group_size); smpi_coll_tuned_allgather_mpich(&my_local_size, 1, MPI_INT, non_uniform_map, 1, MPI_INT, leader_comm); for(i=0; i < leader_group_size; i++) { if(non_uniform_map[0] != non_uniform_map[i]) { is_uniform = 0; break; } } if(is_uniform==0 && smpi_comm_is_uniform(comm)!=0){ comm->non_uniform_map= non_uniform_map; }else{ xbt_free(non_uniform_map); } comm->is_uniform=is_uniform; } smpi_coll_tuned_bcast_mpich(&(comm->is_uniform),1, MPI_INT, 0, comm_intra ); if(smpi_privatize_global_variables){ //we need to switch as the called function may silently touch global variables smpi_switch_data_segment(smpi_process_index()); } // Are the ranks blocked ? = allocated contiguously on the SMP nodes int is_blocked=1; int prev=smpi_group_rank(smpi_comm_group(comm), smpi_group_index(smpi_comm_group(comm_intra), 0)); for (i=1; iis_blocked=global_blocked; } }else{ comm->is_blocked=global_blocked; } xbt_free(leader_list); if(replaying) smpi_process_set_replaying(true); } int smpi_comm_attr_delete(MPI_Comm comm, int keyval){ smpi_comm_key_elem elem = static_cast(xbt_dict_get_or_null_ext(smpi_comm_keyvals, reinterpret_cast(&keyval), sizeof(int))); if(elem==nullptr) return MPI_ERR_ARG; if(elem->delete_fn!=MPI_NULL_DELETE_FN){ void* value = nullptr; int flag; if(smpi_comm_attr_get(comm, keyval, &value, &flag)==MPI_SUCCESS){ int ret = elem->delete_fn(comm, keyval, value, &flag); if(ret!=MPI_SUCCESS) return ret; } } if(comm->attributes==nullptr) return MPI_ERR_ARG; xbt_dict_remove_ext(comm->attributes, reinterpret_cast(&keyval), sizeof(int)); return MPI_SUCCESS; } int smpi_comm_attr_get(MPI_Comm comm, int keyval, void* attr_value, int* flag){ smpi_comm_key_elem elem = static_cast(xbt_dict_get_or_null_ext(smpi_comm_keyvals, reinterpret_cast(&keyval), sizeof(int))); if(elem==nullptr) return MPI_ERR_ARG; if(comm->attributes==nullptr){ *flag=0; return MPI_SUCCESS; } try { *static_cast(attr_value) = xbt_dict_get_ext(comm->attributes, reinterpret_cast(&keyval), sizeof(int)); *flag=1; } catch (xbt_ex& ex) { *flag=0; } return MPI_SUCCESS; } int smpi_comm_attr_put(MPI_Comm comm, int keyval, void* attr_value){ if(smpi_comm_keyvals==nullptr) smpi_comm_keyvals = xbt_dict_new_homogeneous(nullptr); smpi_comm_key_elem elem = static_cast(xbt_dict_get_or_null_ext(smpi_comm_keyvals, reinterpret_cast(&keyval), sizeof(int))); if(elem==nullptr) return MPI_ERR_ARG; int flag; void* value = nullptr; smpi_comm_attr_get(comm, keyval, &value, &flag); if(flag!=0 && elem->delete_fn!=MPI_NULL_DELETE_FN){ int ret = elem->delete_fn(comm, keyval, value, &flag); if(ret!=MPI_SUCCESS) return ret; } if(comm->attributes==nullptr) comm->attributes = xbt_dict_new_homogeneous(nullptr); xbt_dict_set_ext(comm->attributes, reinterpret_cast(&keyval), sizeof(int), attr_value, nullptr); return MPI_SUCCESS; } int smpi_comm_keyval_create(MPI_Comm_copy_attr_function* copy_fn, MPI_Comm_delete_attr_function* delete_fn, int* keyval, void* extra_state){ if(smpi_comm_keyvals==nullptr) smpi_comm_keyvals = xbt_dict_new_homogeneous(nullptr); smpi_comm_key_elem value = static_cast(xbt_new0(s_smpi_mpi_comm_key_elem_t,1)); value->copy_fn=copy_fn; value->delete_fn=delete_fn; *keyval = comm_keyval_id; xbt_dict_set_ext(smpi_comm_keyvals, reinterpret_cast(keyval), sizeof(int),static_cast(value), nullptr); comm_keyval_id++; return MPI_SUCCESS; } int smpi_comm_keyval_free(int* keyval){ smpi_comm_key_elem elem = static_cast(xbt_dict_get_or_null_ext(smpi_comm_keyvals, reinterpret_cast(keyval), sizeof(int))); if(elem==nullptr) return MPI_ERR_ARG; xbt_dict_remove_ext(smpi_comm_keyvals, reinterpret_cast(keyval), sizeof(int)); xbt_free(elem); return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/smpitools.sh000644 001750 001750 00000004050 13031031276 021217 0ustar00mquinsonmquinson000000 000000 #!/bin/sh # Copyright (c) 2013-2014. The SimGrid Team. # All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. SAVEIFS="$IFS" LISTSEP="$(printf '\b')" # Create a temporary file, with its name of the form $1_XXX$2, where XXX is replaced by an unique string. # $1: prefix, $2: suffix mymktemp () { tmp=$(mktemp --suffix="$2" "$1_XXXXXXXXXX" 2> /dev/null) if [ -z "$tmp" ]; then # mktemp failed (unsupported --suffix ?), try unsafe mode tmp=$(mktemp -u "$1_XXXXXXXXXX" 2> /dev/null) if [ -z "$tmp" ]; then # mktemp failed again (doesn't exist ?), try very unsafe mode if [ -z "${mymktemp_seq}" ]; then mymktemp_seq=$(date +%d%H%M%S) fi tmp="$1_$$x${mymktemp_seq}" mymktemp_seq=$((mymktemp_seq + 1)) fi tmp="${tmp}$2" # create temp file, and exit if it existed before sh -C -c "true > \"${tmp}\"" || exit 1 fi echo "${tmp}" } # Add a word to the end of a list (words separated by LISTSEP) # $1: list, $2...: words to add list_add () { local list content newcontent list="$1" shift if [ $# -gt 0 ]; then eval content=\"\${$list}\" IFS="$LISTSEP" newcontent="$*" IFS="$SAVEIFS" if [ -z "$content" ]; then content="$newcontent" else content="$content${LISTSEP}$newcontent" fi eval $list=\"\${content}\" fi } # Like list_add, but only if first word to add ($2) is not empty list_add_not_empty () { if [ -n "$2" ]; then list_add "$@" fi } # Set contents of a list (words separated by LISTSEP) # $1: list, $2...: words to set list_set () { eval $1="" list_add "$@" } # Get the content of a list: positional parameters ($1, $2, ...) are set to the content of the list # $1: list # usage: eval $(list_get list) list_get () { printf 'IFS="$LISTSEP"; eval set -- \\$%s; IFS="$SAVEIFS"' "$1" } SimGrid-3.14.159/src/smpi/smpi_dvfs.cpp000644 001750 001750 00000006331 13031031272 021330 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2013-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include // FIXME: this plugin should be separated from the core #include #include #include #include #include "src/internal_config.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_dvfs, smpi, "Logging specific to SMPI (experimental DVFS support)"); /** * \brief Return the speed of the processor (in flop/s) at a given pstate * * \param pstate_index pstate to test * \return Returns the processor speed associated with pstate_index */ double smpi_get_host_power_peak_at(int pstate_index) { return SIMIX_host_self()->getPstateSpeed(pstate_index); } /** * \brief Return the current speed of the processor (in flop/s) * * \return Returns the current processor speed */ double smpi_get_host_current_power_peak() { return SIMIX_host_self()->getPstateSpeedCurrent(); } /** * \brief Return the number of pstates defined for the current host */ int smpi_get_host_nb_pstates() { return sg_host_get_nb_pstates(SIMIX_host_self()); } /** * \brief Sets the pstate at which the processor should run * * \param pstate_index pstate to switch to */ void smpi_set_host_pstate(int pstate_index) { sg_host_set_pstate(SIMIX_host_self(), pstate_index); } /** @brief Gets the pstate at which the processor currently running */ int smpi_get_host_pstate() { return sg_host_get_pstate(SIMIX_host_self()); } /** * \brief Return the total energy consumed by a host (in Joules) * * \return Returns the consumed energy */ double smpi_get_host_consumed_energy() { return sg_host_get_consumed_energy(SIMIX_host_self()); } #if SMPI_FORTRAN #if defined(__alpha__) || defined(__sparc64__) || defined(__x86_64__) || defined(__ia64__) typedef int integer; typedef unsigned int uinteger; #else typedef long int integer; typedef unsigned long int uinteger; #endif typedef char *address; typedef short int shortint; typedef float real; typedef double doublereal; typedef struct { real r; real i; } complex; typedef struct { doublereal r; doublereal i; } doublecomplex; extern "C" XBT_PUBLIC(doublereal) smpi_get_host_power_peak_at_(integer *pstate_index); doublereal smpi_get_host_power_peak_at_(integer *pstate_index) { return static_cast(smpi_get_host_power_peak_at((int)*pstate_index)); } extern "C" XBT_PUBLIC(doublereal) smpi_get_host_current_power_peak_(); doublereal smpi_get_host_current_power_peak_() { return smpi_get_host_current_power_peak(); } extern "C" XBT_PUBLIC(integer) smpi_get_host_nb_pstates_(); integer smpi_get_host_nb_pstates_() { return static_cast(smpi_get_host_nb_pstates()); } extern "C" XBT_PUBLIC(void) smpi_set_host_pstate_(integer *pstate_index); void smpi_set_host_pstate_(integer *pstate_index) { smpi_set_host_pstate(static_cast(*pstate_index)); } extern "C" XBT_PUBLIC(doublereal) smpi_get_host_consumed_energy_(); doublereal smpi_get_host_consumed_energy_() { return static_cast(smpi_get_host_consumed_energy()); } #endif SimGrid-3.14.159/src/smpi/private.h000644 001750 001750 00000130524 13031031273 020460 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007, 2009-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_PRIVATE_H #define SMPI_PRIVATE_H #include "simgrid/simix.h" #include "smpi/smpi.h" #include "src/include/smpi/smpi_interface.h" #include "src/instr/instr_private.h" #include "src/internal_config.h" #include "xbt.h" #include "xbt/base.h" #include "xbt/synchro.h" #include "xbt/xbt_os_time.h" SG_BEGIN_DECL() struct s_smpi_process_data; typedef struct s_smpi_process_data *smpi_process_data_t; #define PERSISTENT 0x1 #define NON_PERSISTENT 0x2 #define SEND 0x4 #define RECV 0x8 #define RECV_DELETE 0x10 #define ISEND 0x20 #define SSEND 0x40 #define PREPARED 0x80 #define FINISHED 0x100 #define RMA 0x200 #define ACCUMULATE 0x400 enum smpi_process_state{ SMPI_UNINITIALIZED, SMPI_INITIALIZED, SMPI_FINALIZED }; // this struct is here to handle the problem of non-contignous data // for each such structure these function should be implemented (vector // index hvector hindex struct) typedef struct s_smpi_subtype{ void (*serialize)(const void * input, void *output, int count, void* subtype); void (*unserialize)(const void * input, void *output, int count, void* subtype, MPI_Op op); void (*subtype_free)(MPI_Datatype* type); void (*subtype_use)(MPI_Datatype* type); } s_smpi_subtype_t; typedef struct s_smpi_mpi_datatype{ char* name; size_t size; /* this let us know if a serialization is required*/ size_t sizeof_substruct; MPI_Aint lb; MPI_Aint ub; int flags; xbt_dict_t attributes; /* this let us know how to serialize and unserialize*/ void *substruct; int in_use; } s_smpi_mpi_datatype_t; #define COLL_TAG_REDUCE -112 #define COLL_TAG_SCATTER -223 #define COLL_TAG_SCATTERV -334 #define COLL_TAG_GATHER -445 #define COLL_TAG_ALLGATHER -556 #define COLL_TAG_ALLGATHERV -667 #define COLL_TAG_BARRIER -778 #define COLL_TAG_REDUCE_SCATTER -889 #define COLL_TAG_ALLTOALLV -1000 #define COLL_TAG_ALLTOALL -1112 #define COLL_TAG_GATHERV -2223 #define COLL_TAG_BCAST -3334 #define COLL_TAG_ALLREDUCE -4445 #define SMPI_RMA_TAG -1234 #define MPI_COMM_UNINITIALIZED ((MPI_Comm)-1) typedef struct s_smpi_mpi_request { void *buf; /* in the case of non-contiguous memory the user address should be keep * to unserialize the data inside the user memory*/ void *old_buf; /* this let us know how to unserialize at the end of * the communication*/ MPI_Datatype old_type; size_t size; int src; int dst; int tag; //to handle cases where we have an unknown sender //We can't override src, tag, and size, because the request may be reused later int real_src; int real_tag; int truncated; size_t real_size; MPI_Comm comm; smx_activity_t action; unsigned flags; int detached; MPI_Request detached_sender; int refcount; MPI_Op op; int send; int recv; } s_smpi_mpi_request_t; typedef struct s_smpi_mpi_comm_key_elem { MPI_Comm_copy_attr_function* copy_fn; MPI_Comm_delete_attr_function* delete_fn; } s_smpi_mpi_comm_key_elem_t; typedef struct s_smpi_mpi_comm_key_elem *smpi_comm_key_elem; typedef struct s_smpi_mpi_type_key_elem { MPI_Type_copy_attr_function* copy_fn; MPI_Type_delete_attr_function* delete_fn; } s_smpi_mpi_type_key_elem_t; typedef struct s_smpi_mpi_type_key_elem *smpi_type_key_elem; typedef struct s_smpi_mpi_info { xbt_dict_t info_dict; int refcount; } s_smpi_mpi_info_t; XBT_PRIVATE void smpi_process_destroy(); XBT_PRIVATE void smpi_process_finalize(); XBT_PRIVATE int smpi_process_finalized(); XBT_PRIVATE int smpi_process_initialized(); XBT_PRIVATE void smpi_process_mark_as_initialized(); struct s_smpi_mpi_cart_topology; typedef struct s_smpi_mpi_cart_topology *MPIR_Cart_Topology; struct s_smpi_mpi_graph_topology; typedef struct s_smpi_mpi_graph_topology *MPIR_Graph_Topology; struct s_smpi_dist_graph_topology; typedef struct s_smpi_dist_graph_topology *MPIR_Dist_Graph_Topology; // MPI_Topology defined in smpi.h, as it is public XBT_PRIVATE void smpi_topo_destroy(MPI_Topology topo); XBT_PRIVATE MPI_Topology smpi_topo_create(MPIR_Topo_type kind); XBT_PRIVATE void smpi_cart_topo_destroy(MPIR_Cart_Topology cart); XBT_PRIVATE MPI_Topology smpi_cart_topo_create(int ndims); XBT_PRIVATE int smpi_mpi_cart_create(MPI_Comm comm_old, int ndims, int dims[], int periods[], int reorder, MPI_Comm *comm_cart); XBT_PRIVATE int smpi_mpi_cart_sub(MPI_Comm comm, const int remain_dims[], MPI_Comm *newcomm); XBT_PRIVATE int smpi_mpi_cart_coords(MPI_Comm comm, int rank, int maxdims, int coords[]); XBT_PRIVATE int smpi_mpi_cart_get(MPI_Comm comm, int maxdims, int* dims, int* periods, int* coords); XBT_PRIVATE int smpi_mpi_cart_rank(MPI_Comm comm, int* coords, int* rank); XBT_PRIVATE int smpi_mpi_cart_shift(MPI_Comm comm, int direction, int disp, int *rank_source, int *rank_dest); XBT_PRIVATE int smpi_mpi_cartdim_get(MPI_Comm comm, int *ndims); XBT_PRIVATE int smpi_mpi_dims_create(int nnodes, int ndims, int dims[]); XBT_PRIVATE smpi_process_data_t smpi_process_data(); XBT_PRIVATE smpi_process_data_t smpi_process_remote_data(int index); // smpi_process_[set/get]_user_data must be public /* XBT_PRIVATE void smpi_process_set_user_data(void *); */ /* XBT_PRIVATE void* smpi_process_get_user_data(void); */ XBT_PRIVATE int smpi_process_count(); XBT_PRIVATE MPI_Comm smpi_process_comm_world(); XBT_PRIVATE MPI_Comm smpi_process_get_comm_intra(); XBT_PRIVATE void smpi_process_set_comm_intra(MPI_Comm comm); XBT_PRIVATE smx_mailbox_t smpi_process_mailbox(); XBT_PRIVATE smx_mailbox_t smpi_process_remote_mailbox(int index); XBT_PRIVATE smx_mailbox_t smpi_process_mailbox_small(); XBT_PRIVATE smx_mailbox_t smpi_process_remote_mailbox_small(int index); XBT_PRIVATE xbt_mutex_t smpi_process_mailboxes_mutex(); XBT_PRIVATE xbt_mutex_t smpi_process_remote_mailboxes_mutex(int index); XBT_PRIVATE xbt_os_timer_t smpi_process_timer(); XBT_PRIVATE void smpi_process_simulated_start(); XBT_PRIVATE double smpi_process_simulated_elapsed(); XBT_PRIVATE void smpi_process_set_sampling(int s); XBT_PRIVATE int smpi_process_get_sampling(); XBT_PRIVATE void smpi_process_set_replaying(bool s); XBT_PRIVATE bool smpi_process_get_replaying(); XBT_PRIVATE void smpi_deployment_register_process(const char* instance_id, int rank, int index, MPI_Comm** comm, msg_bar_t* bar); XBT_PRIVATE void smpi_deployment_cleanup_instances(); XBT_PRIVATE void smpi_comm_copy_buffer_callback(smx_activity_t comm, void *buff, size_t buff_size); XBT_PRIVATE void smpi_comm_null_copy_buffer_callback(smx_activity_t comm, void *buff, size_t buff_size); XBT_PRIVATE void print_request(const char *message, MPI_Request request); XBT_PRIVATE int smpi_enabled(); XBT_PRIVATE void smpi_global_init(); XBT_PRIVATE void smpi_global_destroy(); XBT_PRIVATE double smpi_mpi_wtime(); XBT_PRIVATE void smpi_mpi_init(); XBT_PRIVATE bool is_datatype_valid(MPI_Datatype datatype); XBT_PRIVATE size_t smpi_datatype_size(MPI_Datatype datatype); XBT_PRIVATE MPI_Aint smpi_datatype_lb(MPI_Datatype datatype); XBT_PRIVATE MPI_Aint smpi_datatype_ub(MPI_Datatype datatype); XBT_PRIVATE int smpi_datatype_dup(MPI_Datatype datatype, MPI_Datatype* new_t); XBT_PRIVATE int smpi_datatype_extent(MPI_Datatype datatype, MPI_Aint * lb, MPI_Aint * extent); XBT_PRIVATE MPI_Aint smpi_datatype_get_extent(MPI_Datatype datatype); XBT_PRIVATE void smpi_datatype_get_name(MPI_Datatype datatype, char* name, int* length); XBT_PRIVATE void smpi_datatype_set_name(MPI_Datatype datatype, char* name); XBT_PRIVATE int smpi_datatype_copy(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype); XBT_PRIVATE void smpi_datatype_use(MPI_Datatype type); XBT_PRIVATE void smpi_datatype_unuse(MPI_Datatype type); XBT_PRIVATE int smpi_datatype_contiguous(int count, MPI_Datatype old_type, MPI_Datatype* new_type, MPI_Aint lb); XBT_PRIVATE int smpi_datatype_vector(int count, int blocklen, int stride, MPI_Datatype old_type, MPI_Datatype* new_type); XBT_PRIVATE int smpi_datatype_hvector(int count, int blocklen, MPI_Aint stride,MPI_Datatype old_type, MPI_Datatype* new_type); XBT_PRIVATE int smpi_datatype_indexed(int count, int* blocklens, int* indices, MPI_Datatype old_type, MPI_Datatype* new_type); XBT_PRIVATE int smpi_datatype_hindexed(int count, int* blocklens, MPI_Aint* indices, MPI_Datatype old_type, MPI_Datatype* new_type); XBT_PRIVATE int smpi_datatype_struct(int count, int* blocklens, MPI_Aint* indices, MPI_Datatype* old_types, MPI_Datatype* new_type); XBT_PRIVATE void smpi_datatype_create(MPI_Datatype* new_type, int size,int lb, int ub, int sizeof_substruct, void *struct_type, int flags); XBT_PRIVATE void smpi_datatype_free(MPI_Datatype* type); XBT_PRIVATE void smpi_datatype_commit(MPI_Datatype* datatype); XBT_PRIVATE int smpi_mpi_unpack(void* inbuf, int insize, int* position, void* outbuf, int outcount, MPI_Datatype type, MPI_Comm comm); XBT_PRIVATE int smpi_mpi_pack(void* inbuf, int incount, MPI_Datatype type, void* outbuf, int outcount, int* position, MPI_Comm comm); XBT_PRIVATE void smpi_empty_status(MPI_Status * status); XBT_PRIVATE MPI_Op smpi_op_new(MPI_User_function * function, bool commute); XBT_PRIVATE bool smpi_op_is_commute(MPI_Op op); XBT_PRIVATE void smpi_op_destroy(MPI_Op op); XBT_PRIVATE void smpi_op_apply(MPI_Op op, const void *invec, void *inoutvec, int *len, MPI_Datatype * datatype); XBT_PRIVATE MPI_Group smpi_group_new(int size); XBT_PRIVATE MPI_Group smpi_group_copy(MPI_Group origin); XBT_PRIVATE void smpi_group_destroy(MPI_Group group); XBT_PRIVATE void smpi_group_set_mapping(MPI_Group group, int index, int rank); XBT_PRIVATE int smpi_group_index(MPI_Group group, int rank); XBT_PRIVATE int smpi_group_rank(MPI_Group group, int index); XBT_PRIVATE int smpi_group_use(MPI_Group group); XBT_PRIVATE int smpi_group_unuse(MPI_Group group); XBT_PRIVATE int smpi_group_size(MPI_Group group); XBT_PRIVATE int smpi_group_compare(MPI_Group group1, MPI_Group group2); XBT_PRIVATE int smpi_group_incl(MPI_Group group, int n, int* ranks, MPI_Group* newgroup); XBT_PRIVATE MPI_Topology smpi_comm_topo(MPI_Comm comm); XBT_PRIVATE MPI_Comm smpi_comm_new(MPI_Group group, MPI_Topology topo); XBT_PRIVATE void smpi_comm_destroy(MPI_Comm comm); XBT_PRIVATE MPI_Group smpi_comm_group(MPI_Comm comm); XBT_PRIVATE int smpi_comm_size(MPI_Comm comm); XBT_PRIVATE void smpi_comm_get_name(MPI_Comm comm, char* name, int* len); XBT_PRIVATE int smpi_comm_rank(MPI_Comm comm); XBT_PRIVATE MPI_Comm smpi_comm_split(MPI_Comm comm, int color, int key); XBT_PRIVATE int smpi_comm_dup(MPI_Comm comm, MPI_Comm* newcomm); XBT_PRIVATE void smpi_comm_use(MPI_Comm comm); XBT_PRIVATE void smpi_comm_unuse(MPI_Comm comm); XBT_PRIVATE void smpi_comm_cleanup_attributes(MPI_Comm comm); XBT_PRIVATE void smpi_comm_cleanup_smp(MPI_Comm comm); XBT_PRIVATE void smpi_comm_set_leaders_comm(MPI_Comm comm, MPI_Comm leaders); XBT_PRIVATE void smpi_comm_set_intra_comm(MPI_Comm comm, MPI_Comm leaders); XBT_PRIVATE int* smpi_comm_get_non_uniform_map(MPI_Comm comm); XBT_PRIVATE int* smpi_comm_get_leaders_map(MPI_Comm comm); XBT_PRIVATE MPI_Comm smpi_comm_get_leaders_comm(MPI_Comm comm); XBT_PRIVATE MPI_Comm smpi_comm_get_intra_comm(MPI_Comm comm); XBT_PRIVATE int smpi_comm_is_uniform(MPI_Comm comm); XBT_PRIVATE int smpi_comm_is_blocked(MPI_Comm comm); XBT_PRIVATE void smpi_comm_init_smp(MPI_Comm comm); XBT_PRIVATE int smpi_comm_c2f(MPI_Comm comm); XBT_PRIVATE MPI_Comm smpi_comm_f2c(int comm); XBT_PRIVATE int smpi_group_c2f(MPI_Group group); XBT_PRIVATE MPI_Group smpi_group_f2c(int group); XBT_PRIVATE int smpi_request_c2f(MPI_Request req); XBT_PRIVATE MPI_Request smpi_request_f2c(int req); XBT_PRIVATE int smpi_type_c2f(MPI_Datatype datatype); XBT_PRIVATE MPI_Datatype smpi_type_f2c(int datatype); XBT_PRIVATE int smpi_op_c2f(MPI_Op op); XBT_PRIVATE MPI_Op smpi_op_f2c(int op); XBT_PRIVATE int smpi_win_c2f(MPI_Win win); XBT_PRIVATE MPI_Win smpi_win_f2c(int win); XBT_PRIVATE int smpi_info_c2f(MPI_Info info); XBT_PRIVATE MPI_Info smpi_info_f2c(int info); XBT_PRIVATE MPI_Request smpi_mpi_send_init(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm); XBT_PRIVATE MPI_Request smpi_mpi_recv_init(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm); XBT_PRIVATE MPI_Request smpi_mpi_ssend_init(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm); XBT_PRIVATE void smpi_mpi_start(MPI_Request request); XBT_PRIVATE void smpi_mpi_startall(int count, MPI_Request * requests); XBT_PRIVATE void smpi_mpi_request_free(MPI_Request * request); XBT_PRIVATE MPI_Request smpi_isend_init(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm); XBT_PRIVATE MPI_Request smpi_mpi_isend(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm); XBT_PRIVATE MPI_Request smpi_issend_init(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm); XBT_PRIVATE MPI_Request smpi_mpi_issend(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm); XBT_PRIVATE MPI_Request smpi_irecv_init(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm); XBT_PRIVATE MPI_Request smpi_mpi_irecv(void *buf, int count, MPI_Datatype datatype, int src, int tag, MPI_Comm comm); XBT_PRIVATE MPI_Request smpi_rma_send_init(void *buf, int count, MPI_Datatype datatype, int src, int dst, int tag, MPI_Comm comm, MPI_Op op); XBT_PRIVATE MPI_Request smpi_rma_recv_init(void *buf, int count, MPI_Datatype datatype, int src, int dst, int tag, MPI_Comm comm, MPI_Op op); XBT_PRIVATE void smpi_mpi_recv(void *buf, int count, MPI_Datatype datatype, int src,int tag, MPI_Comm comm, MPI_Status * status); XBT_PRIVATE void smpi_mpi_send(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm); XBT_PRIVATE void smpi_mpi_ssend(void *buf, int count, MPI_Datatype datatype, int dst, int tag, MPI_Comm comm); XBT_PRIVATE void smpi_mpi_sendrecv(void *sendbuf, int sendcount, MPI_Datatype sendtype, int dst, int sendtag, void *recvbuf, int recvcount, MPI_Datatype recvtype, int src, int recvtag, MPI_Comm comm, MPI_Status * status); XBT_PRIVATE int smpi_mpi_test(MPI_Request * request, MPI_Status * status); XBT_PRIVATE int smpi_mpi_testany(int count, MPI_Request requests[], int *index, MPI_Status * status); XBT_PRIVATE int smpi_mpi_testall(int count, MPI_Request requests[], MPI_Status status[]); XBT_PRIVATE void smpi_mpi_probe(int source, int tag, MPI_Comm comm, MPI_Status* status); XBT_PRIVATE void smpi_mpi_iprobe(int source, int tag, MPI_Comm comm, int* flag, MPI_Status* status); XBT_PRIVATE int smpi_mpi_get_count(MPI_Status * status, MPI_Datatype datatype); XBT_PRIVATE void smpi_mpi_wait(MPI_Request * request, MPI_Status * status); XBT_PRIVATE int smpi_mpi_waitany(int count, MPI_Request requests[], MPI_Status * status); XBT_PRIVATE int smpi_mpi_waitall(int count, MPI_Request requests[], MPI_Status status[]); XBT_PRIVATE int smpi_mpi_waitsome(int incount, MPI_Request requests[], int *indices, MPI_Status status[]); XBT_PRIVATE int smpi_mpi_testsome(int incount, MPI_Request requests[], int *indices, MPI_Status status[]); XBT_PRIVATE void smpi_mpi_bcast(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm); XBT_PRIVATE void smpi_mpi_barrier(MPI_Comm comm); XBT_PRIVATE void smpi_mpi_gather(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm); XBT_PRIVATE void smpi_mpi_reduce_scatter(void *sendbuf, void *recvbuf, int *recvcounts, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm); XBT_PRIVATE void smpi_mpi_gatherv(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvtype, int root, MPI_Comm comm); XBT_PRIVATE void smpi_mpi_allgather(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm); XBT_PRIVATE void smpi_mpi_allgatherv(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *displs, MPI_Datatype recvtype, MPI_Comm comm); XBT_PRIVATE void smpi_mpi_scatter(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm); XBT_PRIVATE void smpi_mpi_scatterv(void *sendbuf, int *sendcounts, int *displs, MPI_Datatype sendtype, void *recvbuf, int recvcount,MPI_Datatype recvtype, int root, MPI_Comm comm); XBT_PRIVATE void smpi_mpi_reduce(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm); XBT_PRIVATE void smpi_mpi_allreduce(void *sendbuf, void *recvbuf, int count,MPI_Datatype datatype, MPI_Op op, MPI_Comm comm); XBT_PRIVATE void smpi_mpi_scan(void *sendbuf, void *recvbuf, int count,MPI_Datatype datatype, MPI_Op op, MPI_Comm comm); XBT_PRIVATE void smpi_mpi_exscan(void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm); XBT_PRIVATE int smpi_mpi_win_free( MPI_Win* win); XBT_PRIVATE MPI_Win smpi_mpi_win_create( void *base, MPI_Aint size, int disp_unit, MPI_Info info, MPI_Comm comm); XBT_PRIVATE void smpi_mpi_win_get_name(MPI_Win win, char* name, int* length); XBT_PRIVATE void smpi_mpi_win_get_group(MPI_Win win, MPI_Group* group); XBT_PRIVATE void smpi_mpi_win_set_name(MPI_Win win, char* name); XBT_PRIVATE int smpi_mpi_win_fence( int assert, MPI_Win win); XBT_PRIVATE int smpi_mpi_win_post(MPI_Group group, int assert, MPI_Win win); XBT_PRIVATE int smpi_mpi_win_start(MPI_Group group, int assert, MPI_Win win); XBT_PRIVATE int smpi_mpi_win_complete(MPI_Win win); XBT_PRIVATE int smpi_mpi_win_wait(MPI_Win win); XBT_PRIVATE int smpi_mpi_get( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Win win); XBT_PRIVATE int smpi_mpi_put( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Win win); XBT_PRIVATE int smpi_mpi_accumulate( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Op op, MPI_Win win); XBT_PRIVATE void nary_tree_bcast(void *buf, int count, MPI_Datatype datatype, int root, MPI_Comm comm, int arity); XBT_PRIVATE void nary_tree_barrier(MPI_Comm comm, int arity); XBT_PRIVATE int smpi_coll_tuned_alltoall_ompi2(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm); XBT_PRIVATE int smpi_coll_tuned_alltoall_bruck(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm); XBT_PRIVATE int smpi_coll_tuned_alltoall_basic_linear(void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm); XBT_PRIVATE int smpi_coll_basic_alltoallv(void *sendbuf, int *sendcounts, int *senddisps, MPI_Datatype sendtype, void *recvbuf, int *recvcounts, int *recvdisps, MPI_Datatype recvtype, MPI_Comm comm); XBT_PRIVATE int smpi_comm_keyval_create(MPI_Comm_copy_attr_function* copy_fn, MPI_Comm_delete_attr_function* delete_fn, int* keyval, void* extra_state); XBT_PRIVATE int smpi_comm_keyval_free(int* keyval); XBT_PRIVATE int smpi_comm_attr_get(MPI_Comm comm, int keyval, void* attr_value, int* flag); XBT_PRIVATE int smpi_comm_attr_delete(MPI_Comm comm, int keyval); XBT_PRIVATE int smpi_comm_attr_put(MPI_Comm comm, int keyval, void* attr_value); XBT_PRIVATE int smpi_type_attr_delete(MPI_Datatype type, int keyval); XBT_PRIVATE int smpi_type_attr_get(MPI_Datatype type, int keyval, void* attr_value, int* flag); XBT_PRIVATE int smpi_type_attr_put(MPI_Datatype type, int keyval, void* attr_value); XBT_PRIVATE int smpi_type_keyval_create(MPI_Type_copy_attr_function* copy_fn, MPI_Type_delete_attr_function* delete_fn, int* keyval, void* extra_state); XBT_PRIVATE int smpi_type_keyval_free(int* keyval); // utilities extern XBT_PRIVATE double smpi_cpu_threshold; extern XBT_PRIVATE double smpi_host_speed; extern XBT_PRIVATE bool smpi_privatize_global_variables; extern XBT_PRIVATE char* smpi_start_data_exe; //start of the data+bss segment of the executable extern XBT_PRIVATE int smpi_size_data_exe; //size of the data+bss segment of the executable XBT_PRIVATE void smpi_switch_data_segment(int dest); XBT_PRIVATE void smpi_really_switch_data_segment(int dest); XBT_PRIVATE int smpi_is_privatisation_file(char* file); XBT_PRIVATE void smpi_get_executable_global_size(); XBT_PRIVATE void smpi_initialize_global_memory_segments(); XBT_PRIVATE void smpi_destroy_global_memory_segments(); XBT_PRIVATE void smpi_bench_destroy(); XBT_PRIVATE void smpi_bench_begin(); XBT_PRIVATE void smpi_bench_end(); XBT_PRIVATE void* smpi_get_tmp_sendbuffer(int size); XBT_PRIVATE void* smpi_get_tmp_recvbuffer(int size); XBT_PRIVATE void smpi_free_tmp_buffer(void* buf); XBT_PRIVATE int smpi_comm_attr_get(MPI_Comm comm, int keyval, void* attr_value, int* flag); XBT_PRIVATE XBT_PRIVATE int smpi_comm_attr_delete(MPI_Comm comm, int keyval); XBT_PRIVATE int smpi_comm_attr_put(MPI_Comm comm, int keyval, void* attr_value); // f77 wrappers void mpi_init_(int* ierr); void mpi_finalize_(int* ierr); void mpi_abort_(int* comm, int* errorcode, int* ierr); void mpi_comm_rank_(int* comm, int* rank, int* ierr); void mpi_comm_size_(int* comm, int* size, int* ierr); double mpi_wtime_(); double mpi_wtick_(); void mpi_initialized_(int* flag, int* ierr); void mpi_comm_dup_(int* comm, int* newcomm, int* ierr); void mpi_comm_create_(int* comm, int* group, int* newcomm, int* ierr); void mpi_comm_free_(int* comm, int* ierr); void mpi_comm_split_(int* comm, int* color, int* key, int* comm_out, int* ierr); void mpi_group_incl_(int* group, int* n, int* key, int* group_out, int* ierr) ; void mpi_comm_group_(int* comm, int* group_out, int* ierr); void mpi_send_init_(void *buf, int* count, int* datatype, int* dst, int* tag, int* comm, int* request, int* ierr); void mpi_isend_(void *buf, int* count, int* datatype, int* dst, int* tag, int* comm, int* request, int* ierr); void mpi_irsend_(void *buf, int* count, int* datatype, int* dst, int* tag, int* comm, int* request, int* ierr); void mpi_send_(void* buf, int* count, int* datatype, int* dst, int* tag, int* comm, int* ierr); void mpi_rsend_(void* buf, int* count, int* datatype, int* dst, int* tag, int* comm, int* ierr); void mpi_recv_init_(void *buf, int* count, int* datatype, int* src, int* tag, int* comm, int* request, int* ierr); void mpi_irecv_(void *buf, int* count, int* datatype, int* src, int* tag, int* comm, int* request, int* ierr); void mpi_recv_(void* buf, int* count, int* datatype, int* src, int* tag, int* comm, MPI_Status* status, int* ierr); void mpi_start_(int* request, int* ierr); void mpi_startall_(int* count, int* requests, int* ierr); void mpi_wait_(int* request, MPI_Status* status, int* ierr); void mpi_waitany_(int* count, int* requests, int* index, MPI_Status* status, int* ierr); void mpi_waitall_(int* count, int* requests, MPI_Status* status, int* ierr); void mpi_barrier_(int* comm, int* ierr); void mpi_bcast_(void* buf, int* count, int* datatype, int* root, int* comm, int* ierr); void mpi_reduce_(void* sendbuf, void* recvbuf, int* count, int* datatype, int* op, int* root, int* comm, int* ierr); void mpi_allreduce_(void* sendbuf, void* recvbuf, int* count, int* datatype, int* op, int* comm, int* ierr); void mpi_reduce_scatter_(void* sendbuf, void* recvbuf, int* recvcounts, int* datatype, int* op, int* comm, int* ierr) ; void mpi_scatter_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcount, int* recvtype, int* root, int* comm, int* ierr); void mpi_scatterv_(void* sendbuf, int* sendcounts, int* displs, int* sendtype, void* recvbuf, int* recvcount, int* recvtype, int* root, int* comm, int* ierr); void mpi_gather_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcount, int* recvtype, int* root, int* comm, int* ierr); void mpi_gatherv_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcounts, int* displs, int* recvtype, int* root, int* comm, int* ierr); void mpi_allgather_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcount, int* recvtype, int* comm, int* ierr); void mpi_allgatherv_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcount,int* displs, int* recvtype, int* comm, int* ierr) ; void mpi_type_size_(int* datatype, int *size, int* ierr); void mpi_scan_(void* sendbuf, void* recvbuf, int* count, int* datatype, int* op, int* comm, int* ierr); void mpi_alltoall_(void* sendbuf, int* sendcount, int* sendtype, void* recvbuf, int* recvcount, int* recvtype, int* comm, int* ierr); void mpi_alltoallv_(void* sendbuf, int* sendcounts, int* senddisps, int* sendtype, void* recvbuf, int* recvcounts, int* recvdisps, int* recvtype, int* comm, int* ierr); void mpi_get_processor_name_(char *name, int *resultlen, int* ierr); void mpi_test_ (int * request, int *flag, MPI_Status * status, int* ierr); void mpi_testall_ (int* count, int * requests, int *flag, MPI_Status * statuses, int* ierr); void mpi_get_count_(MPI_Status * status, int* datatype, int *count, int* ierr); void mpi_type_extent_(int* datatype, MPI_Aint * extent, int* ierr); void mpi_attr_get_(int* comm, int* keyval, void* attr_value, int* flag, int* ierr ); void mpi_type_commit_(int* datatype, int* ierr); void mpi_type_vector_(int* count, int* blocklen, int* stride, int* old_type, int* newtype, int* ierr); void mpi_type_create_vector_(int* count, int* blocklen, int* stride, int* old_type, int* newtype, int* ierr); void mpi_type_hvector_(int* count, int* blocklen, MPI_Aint* stride, int* old_type, int* newtype, int* ierr); void mpi_type_create_hvector_(int* count, int* blocklen, MPI_Aint* stride, int* old_type, int* newtype, int* ierr); void mpi_type_free_(int* datatype, int* ierr); void mpi_type_lb_(int* datatype, MPI_Aint * extent, int* ierr); void mpi_type_ub_(int* datatype, MPI_Aint * extent, int* ierr); void mpi_win_fence_( int* assert, int* win, int* ierr); void mpi_win_free_( int* win, int* ierr); void mpi_win_create_( int *base, MPI_Aint* size, int* disp_unit, int* info, int* comm, int *win, int* ierr); void mpi_win_set_name_ (int* win, char * name, int* ierr, int size); void mpi_win_get_name_ (int* win, char * name, int* len, int* ierr); void mpi_win_post_(int* group, int assert, int* win, int* ierr); void mpi_win_start_(int* group, int assert, int* win, int* ierr); void mpi_win_complete_(int* win, int* ierr); void mpi_win_wait_(int* win, int* ierr); void mpi_info_create_( int *info, int* ierr); void mpi_info_set_( int *info, char *key, char *value, int* ierr, unsigned int keylen, unsigned int valuelen); void mpi_info_free_(int* info, int* ierr); void mpi_get_( int *origin_addr, int* origin_count, int* origin_datatype, int* target_rank, MPI_Aint* target_disp, int* target_count, int* target_datatype, int* win, int* ierr); void mpi_put_( int *origin_addr, int* origin_count, int* origin_datatype, int* target_rank, MPI_Aint* target_disp, int* target_count, int* target_datatype, int* win, int* ierr); void mpi_accumulate_( int *origin_addr, int* origin_count, int* origin_datatype, int* target_rank, MPI_Aint* target_disp, int* target_count, int* target_datatype, int* op, int* win, int* ierr); void mpi_error_string_(int* errorcode, char* string, int* resultlen, int* ierr); void mpi_sendrecv_(void* sendbuf, int* sendcount, int* sendtype, int* dst, int* sendtag, void *recvbuf, int* recvcount, int* recvtype, int* src, int* recvtag, int* comm, MPI_Status* status, int* ierr); void mpi_finalized_ (int * flag, int* ierr); void mpi_init_thread_ (int *required, int *provided, int* ierr); void mpi_query_thread_ (int *provided, int* ierr); void mpi_is_thread_main_ (int *flag, int* ierr); void mpi_address_ (void *location, MPI_Aint * address, int* ierr); void mpi_get_address_ (void *location, MPI_Aint * address, int* ierr); void mpi_type_dup_ (int* datatype, int* newdatatype, int* ierr); void mpi_type_set_name_ (int* datatype, char * name, int* ierr, int size); void mpi_type_get_name_ (int* datatype, char * name, int* len, int* ierr); void mpi_type_get_attr_ (int* type, int* type_keyval, void *attribute_val, int* flag, int* ierr); void mpi_type_set_attr_ (int* type, int* type_keyval, void *attribute_val, int* ierr); void mpi_type_delete_attr_ (int* type, int* type_keyval, int* ierr); void mpi_type_create_keyval_ (void* copy_fn, void* delete_fn, int* keyval, void* extra_state, int* ierr); void mpi_type_free_keyval_ (int* keyval, int* ierr) ; void mpi_pcontrol_ (int* level , int* ierr); void mpi_type_get_extent_ (int* datatype, MPI_Aint * lb, MPI_Aint * extent, int* ierr); void mpi_type_get_true_extent_ (int* datatype, MPI_Aint * lb, MPI_Aint * extent, int* ierr); void mpi_op_create_ (void * function, int* commute, int* op, int* ierr); void mpi_op_free_ (int* op, int* ierr); void mpi_group_free_ (int* group, int* ierr); void mpi_group_size_ (int* group, int *size, int* ierr); void mpi_group_rank_ (int* group, int *rank, int* ierr); void mpi_group_translate_ranks_ (int* group1, int* n, int *ranks1, int* group2, int *ranks2, int* ierr); void mpi_group_compare_ (int* group1, int* group2, int *result, int* ierr); void mpi_group_union_ (int* group1, int* group2, int* newgroup, int* ierr); void mpi_group_intersection_ (int* group1, int* group2, int* newgroup, int* ierr); void mpi_group_difference_ (int* group1, int* group2, int* newgroup, int* ierr); void mpi_group_excl_ (int* group, int* n, int *ranks, int* newgroup, int* ierr); void mpi_group_range_incl_ (int* group, int* n, int ranges[][3], int* newgroup, int* ierr); void mpi_group_range_excl_ (int* group, int* n, int ranges[][3], int* newgroup, int* ierr); void mpi_comm_get_attr_ (int* comm, int* comm_keyval, void *attribute_val, int *flag, int* ierr); void mpi_comm_set_attr_ (int* comm, int* comm_keyval, void *attribute_val, int* ierr); void mpi_comm_delete_attr_ (int* comm, int* comm_keyval, int* ierr); void mpi_comm_create_keyval_ (void* copy_fn, void* delete_fn, int* keyval, void* extra_state, int* ierr); void mpi_comm_free_keyval_ (int* keyval, int* ierr) ; void mpi_comm_get_name_ (int* comm, char* name, int* len, int* ierr); void mpi_comm_compare_ (int* comm1, int* comm2, int *result, int* ierr); void mpi_comm_disconnect_ (int* comm, int* ierr); void mpi_request_free_ (int* request, int* ierr); void mpi_sendrecv_replace_ (void *buf, int* count, int* datatype, int* dst, int* sendtag, int* src, int* recvtag, int* comm, MPI_Status* status, int* ierr); void mpi_testany_ (int* count, int* requests, int *index, int *flag, MPI_Status* status, int* ierr); void mpi_waitsome_ (int* incount, int* requests, int *outcount, int *indices, MPI_Status* status, int* ierr); void mpi_reduce_local_ (void *inbuf, void *inoutbuf, int* count, int* datatype, int* op, int* ierr); void mpi_reduce_scatter_block_ (void *sendbuf, void *recvbuf, int* recvcount, int* datatype, int* op, int* comm, int* ierr); void mpi_pack_size_ (int* incount, int* datatype, int* comm, int* size, int* ierr) ; void mpi_cart_coords_ (int* comm, int* rank, int* maxdims, int* coords, int* ierr) ; void mpi_cart_create_ (int* comm_old, int* ndims, int* dims, int* periods, int* reorder, int* comm_cart, int* ierr) ; void mpi_cart_get_ (int* comm, int* maxdims, int* dims, int* periods, int* coords, int* ierr) ; void mpi_cart_map_ (int* comm_old, int* ndims, int* dims, int* periods, int* newrank, int* ierr) ; void mpi_cart_rank_ (int* comm, int* coords, int* rank, int* ierr) ; void mpi_cart_shift_ (int* comm, int* direction, int* displ, int* source, int* dest, int* ierr) ; void mpi_cart_sub_ (int* comm, int* remain_dims, int* comm_new, int* ierr) ; void mpi_cartdim_get_ (int* comm, int* ndims, int* ierr) ; void mpi_graph_create_ (int* comm_old, int* nnodes, int* index, int* edges, int* reorder, int* comm_graph, int* ierr) ; void mpi_graph_get_ (int* comm, int* maxindex, int* maxedges, int* index, int* edges, int* ierr) ; void mpi_graph_map_ (int* comm_old, int* nnodes, int* index, int* edges, int* newrank, int* ierr) ; void mpi_graph_neighbors_ (int* comm, int* rank, int* maxneighbors, int* neighbors, int* ierr) ; void mpi_graph_neighbors_count_ (int* comm, int* rank, int* nneighbors, int* ierr) ; void mpi_graphdims_get_ (int* comm, int* nnodes, int* nedges, int* ierr) ; void mpi_topo_test_ (int* comm, int* top_type, int* ierr) ; void mpi_error_class_ (int* errorcode, int* errorclass, int* ierr) ; void mpi_errhandler_create_ (void* function, void* errhandler, int* ierr) ; void mpi_errhandler_free_ (void* errhandler, int* ierr) ; void mpi_errhandler_get_ (int* comm, void* errhandler, int* ierr) ; void mpi_errhandler_set_ (int* comm, void* errhandler, int* ierr) ; void mpi_comm_set_errhandler_ (int* comm, void* errhandler, int* ierr) ; void mpi_comm_get_errhandler_ (int* comm, void* errhandler, int* ierr) ; void mpi_type_contiguous_ (int* count, int* old_type, int* newtype, int* ierr) ; void mpi_cancel_ (int* request, int* ierr) ; void mpi_buffer_attach_ (void* buffer, int* size, int* ierr) ; void mpi_buffer_detach_ (void* buffer, int* size, int* ierr) ; void mpi_testsome_ (int* incount, int* requests, int* outcount, int* indices, MPI_Status* statuses, int* ierr) ; void mpi_comm_test_inter_ (int* comm, int* flag, int* ierr) ; void mpi_unpack_ (void* inbuf, int* insize, int* position, void* outbuf, int* outcount, int* type, int* comm, int* ierr) ; void mpi_pack_external_size_ (char *datarep, int* incount, int* datatype, MPI_Aint *size, int* ierr); void mpi_pack_external_ (char *datarep, void *inbuf, int* incount, int* datatype, void *outbuf, MPI_Aint* outcount, MPI_Aint *position, int* ierr); void mpi_unpack_external_ (char *datarep, void *inbuf, MPI_Aint* insize, MPI_Aint *position, void *outbuf, int* outcount, int* datatype, int* ierr); void mpi_type_hindexed_ (int* count, int* blocklens, MPI_Aint* indices, int* old_type, int* newtype, int* ierr) ; void mpi_type_create_hindexed_ (int* count, int* blocklens, MPI_Aint* indices, int* old_type, int* newtype, int* ierr); void mpi_type_create_hindexed_block_ (int* count, int* blocklength, MPI_Aint* indices, int* old_type, int* newtype, int* ierr) ; void mpi_type_indexed_ (int* count, int* blocklens, int* indices, int* old_type, int* newtype, int* ierr) ; void mpi_type_create_indexed_block_ (int* count, int* blocklength, int* indices, int* old_type, int*newtype, int* ierr); void mpi_type_struct_ (int* count, int* blocklens, MPI_Aint* indices, int* old_types, int* newtype, int* ierr) ; void mpi_type_create_struct_ (int* count, int* blocklens, MPI_Aint* indices, int* old_types, int* newtype, int* ierr); void mpi_ssend_ (void* buf, int* count, int* datatype, int* dest, int* tag, int* comm, int* ierr) ; void mpi_ssend_init_ (void* buf, int* count, int* datatype, int* dest, int* tag, int* comm, int* request, int* ierr) ; void mpi_intercomm_create_ (int* local_comm, int* local_leader, int* peer_comm, int* remote_leader, int* tag, int* comm_out, int* ierr) ; void mpi_intercomm_merge_ (int* comm, int* high, int* comm_out, int* ierr) ; void mpi_bsend_ (void* buf, int* count, int* datatype, int* dest, int* tag, int* comm, int* ierr) ; void mpi_bsend_init_ (void* buf, int* count, int* datatype, int* dest, int* tag, int* comm, int* request, int* ierr) ; void mpi_ibsend_ (void* buf, int* count, int* datatype, int* dest, int* tag, int* comm, int* request, int* ierr) ; void mpi_comm_remote_group_ (int* comm, int* group, int* ierr) ; void mpi_comm_remote_size_ (int* comm, int* size, int* ierr) ; void mpi_issend_ (void* buf, int* count, int* datatype, int* dest, int* tag, int* comm, int* request, int* ierr) ; void mpi_probe_ (int* source, int* tag, int* comm, MPI_Status* status, int* ierr) ; void mpi_attr_delete_ (int* comm, int* keyval, int* ierr) ; void mpi_attr_put_ (int* comm, int* keyval, void* attr_value, int* ierr) ; void mpi_rsend_init_ (void* buf, int* count, int* datatype, int* dest, int* tag, int* comm, int* request, int* ierr) ; void mpi_keyval_create_ (void* copy_fn, void* delete_fn, int* keyval, void* extra_state, int* ierr) ; void mpi_keyval_free_ (int* keyval, int* ierr) ; void mpi_test_cancelled_ (MPI_Status* status, int* flag, int* ierr) ; void mpi_pack_ (void* inbuf, int* incount, int* type, void* outbuf, int* outcount, int* position, int* comm, int* ierr); void mpi_get_elements_ (MPI_Status* status, int* datatype, int* elements, int* ierr) ; void mpi_dims_create_ (int* nnodes, int* ndims, int* dims, int* ierr) ; void mpi_iprobe_ (int* source, int* tag, int* comm, int* flag, MPI_Status* status, int* ierr) ; void mpi_type_get_envelope_ ( int* datatype, int *num_integers, int *num_addresses, int *num_datatypes, int *combiner, int* ierr); void mpi_type_get_contents_ (int* datatype, int* max_integers, int* max_addresses, int* max_datatypes, int* array_of_integers, MPI_Aint* array_of_addresses, int*array_of_datatypes, int* ierr); void mpi_type_create_darray_ (int* size, int* rank, int* ndims, int* array_of_gsizes, int* array_of_distribs, int* array_of_dargs, int* array_of_psizes, int* order, int* oldtype, int*newtype, int* ierr) ; void mpi_type_create_resized_ (int* oldtype,MPI_Aint* lb, MPI_Aint* extent, int*newtype, int* ierr); void mpi_type_create_subarray_ (int* ndims,int *array_of_sizes, int *array_of_subsizes, int *array_of_starts, int* order, int* oldtype, int*newtype, int* ierr); void mpi_type_match_size_ (int* typeclass,int* size,int*datatype, int* ierr); void mpi_alltoallw_ ( void *sendbuf, int *sendcnts, int *sdispls, int*sendtypes, void *recvbuf, int *recvcnts, int *rdispls, int*recvtypes, int* comm, int* ierr); void mpi_exscan_ (void *sendbuf, void *recvbuf, int* count, int* datatype, int* op, int* comm, int* ierr); void mpi_comm_set_name_ (int* comm, char* name, int* ierr, int size); void mpi_comm_dup_with_info_ (int* comm, int* info, int* newcomm, int* ierr); void mpi_comm_split_type_ (int* comm, int* split_type, int* key, int* info, int*newcomm, int* ierr); void mpi_comm_set_info_ (int* comm, int* info, int* ierr); void mpi_comm_get_info_ (int* comm, int* info, int* ierr); void mpi_info_get_ (int* info,char *key,int* valuelen, char *value, int *flag, int* ierr, unsigned int keylen); void mpi_comm_create_errhandler_ ( void *function, void *errhandler, int* ierr); void mpi_add_error_class_ ( int *errorclass, int* ierr); void mpi_add_error_code_ ( int* errorclass, int *errorcode, int* ierr); void mpi_add_error_string_ ( int* errorcode, char *string, int* ierr); void mpi_comm_call_errhandler_ (int* comm,int* errorcode, int* ierr); void mpi_info_dup_ (int* info, int* newinfo, int* ierr); void mpi_info_get_valuelen_ ( int* info, char *key, int *valuelen, int *flag, int* ierr, unsigned int keylen); void mpi_info_delete_ (int* info, char *key, int* ierr, unsigned int keylen); void mpi_info_get_nkeys_ ( int* info, int *nkeys, int* ierr); void mpi_info_get_nthkey_ ( int* info, int* n, char *key, int* ierr, unsigned int keylen); void mpi_get_version_ (int *version,int *subversion, int* ierr); void mpi_get_library_version_ (char *version,int *len, int* ierr); void mpi_request_get_status_ ( int* request, int *flag, MPI_Status* status, int* ierr); void mpi_grequest_start_ ( void *query_fn, void *free_fn, void *cancel_fn, void *extra_state, int*request, int* ierr); void mpi_grequest_complete_ ( int* request, int* ierr); void mpi_status_set_cancelled_ (MPI_Status* status,int* flag, int* ierr); void mpi_status_set_elements_ ( MPI_Status* status, int* datatype, int* count, int* ierr); void mpi_comm_connect_ ( char *port_name, int* info, int* root, int* comm, int* newcomm, int* ierr); void mpi_publish_name_ ( char *service_name, int* info, char *port_name, int* ierr); void mpi_unpublish_name_ ( char *service_name, int* info, char *port_name, int* ierr); void mpi_lookup_name_ ( char *service_name, int* info, char *port_name, int* ierr); void mpi_comm_join_ ( int* fd, int*intercomm, int* ierr); void mpi_open_port_ ( int* info, char *port_name, int* ierr); void mpi_close_port_ ( char *port_name, int* ierr); void mpi_comm_accept_ ( char *port_name, int* info, int* root, int* comm, int* newcomm, int* ierr); void mpi_comm_spawn_ ( char *command, char *argv, int* maxprocs, int* info, int* root, int* comm, int*intercomm, int* array_of_errcodes, int* ierr); void mpi_comm_spawn_multiple_ ( int* count, char *array_of_commands, char** array_of_argv, int* array_of_maxprocs, int* array_of_info, int* root, int* comm, int*intercomm, int* array_of_errcodes, int* ierr); void mpi_comm_get_parent_ ( int*parent, int* ierr); void mpi_file_close_ ( int* file, int* ierr); void mpi_file_delete_ ( char* filename, int* info, int* ierr); void mpi_file_open_ ( int* comm, char* filename, int* amode, int* info, int* fh, int* ierr); void mpi_file_set_view_ ( int* fh, long long int* offset, int* etype, int* filetype, char* datarep, int* info, int* ierr); void mpi_file_read_ ( int* fh, void* buf, int* count, int* datatype, MPI_Status* status, int* ierr); void mpi_file_write_ ( int* fh, void* buf, int* count, int* datatype, MPI_Status* status, int* ierr); /********** Tracing **********/ /* from instr_smpi.c */ XBT_PRIVATE void TRACE_internal_smpi_set_category (const char *category); XBT_PRIVATE const char *TRACE_internal_smpi_get_category (); XBT_PRIVATE void TRACE_smpi_collective_in(int rank, int root, const char *operation, instr_extra_data extra); XBT_PRIVATE void TRACE_smpi_collective_out(int rank, int root, const char *operation); XBT_PRIVATE void TRACE_smpi_computing_init(int rank); XBT_PRIVATE void TRACE_smpi_computing_out(int rank); XBT_PRIVATE void TRACE_smpi_computing_in(int rank, instr_extra_data extra); XBT_PRIVATE void TRACE_smpi_sleeping_init(int rank); XBT_PRIVATE void TRACE_smpi_sleeping_out(int rank); XBT_PRIVATE void TRACE_smpi_sleeping_in(int rank, instr_extra_data extra); XBT_PRIVATE void TRACE_smpi_testing_out(int rank); XBT_PRIVATE void TRACE_smpi_testing_in(int rank, instr_extra_data extra); XBT_PRIVATE void TRACE_smpi_alloc(); XBT_PRIVATE void TRACE_smpi_release(); XBT_PRIVATE void TRACE_smpi_ptp_in(int rank, int src, int dst, const char *operation, instr_extra_data extra); XBT_PRIVATE void TRACE_smpi_ptp_out(int rank, int src, int dst, const char *operation); XBT_PRIVATE void TRACE_smpi_send(int rank, int src, int dst, int tag, int size); XBT_PRIVATE void TRACE_smpi_recv(int rank, int src, int dst, int tag); XBT_PRIVATE void TRACE_smpi_init(int rank); XBT_PRIVATE void TRACE_smpi_finalize(int rank); XBT_PRIVATE char *smpi_container(int rank, char *container, int n); XBT_PRIVATE const char* encode_datatype(MPI_Datatype datatype, int* known); // TODO, make this static and expose it more cleanly typedef struct s_smpi_privatisation_region { void* address; int file_descriptor; } s_smpi_privatisation_region_t, *smpi_privatisation_region_t; extern XBT_PRIVATE smpi_privatisation_region_t smpi_privatisation_regions; extern XBT_PRIVATE int smpi_loaded_page; extern XBT_PRIVATE int smpi_universe_size; XBT_PRIVATE int SIMIX_process_get_PID(smx_actor_t self); static inline __attribute__ ((always_inline)) int smpi_process_index_of_smx_process(smx_actor_t process) { return SIMIX_process_get_PID(process) -1; } SG_END_DECL() #endif SimGrid-3.14.159/src/smpi/instr_smpi.cpp000644 001750 001750 00000032465 13031031272 021534 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010, 2012-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.h" #include "private.hpp" #include #include #include #include XBT_LOG_NEW_DEFAULT_SUBCATEGORY(instr_smpi, instr, "Tracing SMPI"); static xbt_dict_t keys; static const char *smpi_colors[] ={ "recv", "1 0 0", "irecv", "1 0.52 0.52", "send", "0 0 1", "isend", "0.52 0.52 1", "sendrecv", "0 1 1", "wait", "1 1 0", "waitall", "0.78 0.78 0", "waitany", "0.78 0.78 0.58", "test", "0.52 0.52 0", "allgather", "1 0 0", "allgatherv", "1 0.52 0.52", "allreduce", "1 0 1", "alltoall", "0.52 0 1", "alltoallv", "0.78 0.52 1", "barrier", "0 0.78 0.78", "bcast", "0 0.78 0.39", "gather", "1 1 0", "gatherv", "1 1 0.52", "reduce", "0 1 0", "reducescatter", "0.52 1 0.52", "scan", "1 0.58 0.23", "exscan", "1 0.54 0.25", "scatterv", "0.52 0 0.52", "scatter", "1 0.74 0.54", "computing", "0 1 1", "sleeping", "0 0.5 0.5", "init", "0 1 0", "finalize", "0 1 0", "put", "0.3 1 0", "get", "0 1 0.3", "accumulate", "1 0.3 0", "win_fence", "1 0 0.3", "win_post", "1 0 0.8", "win_wait", "1 0.8 0", "win_start", "0.8 0 1", "win_complete", "0.8 1 0", nullptr, nullptr, }; static char *str_tolower (const char *str) { char *ret = xbt_strdup (str); int i, n = strlen (ret); for (i = 0; i < n; i++) ret[i] = tolower (str[i]); return ret; } static const char *instr_find_color (const char *state) { char *target = str_tolower (state); const char *ret = nullptr; unsigned int i = 0; const char *current = smpi_colors[i]; while ((current != nullptr)){ if (strcmp (state, current) == 0 //exact match || strstr(target, current) != 0 ){//as substring ret = smpi_colors[i+1]; break; } i+=2; current = smpi_colors[i]; } xbt_free(target); return ret; } XBT_PRIVATE char *smpi_container(int rank, char *container, int n) { snprintf(container, n, "rank-%d", rank); return container; } static char *TRACE_smpi_get_key(int src, int dst, int tag, char *key, int n, int send); static char *TRACE_smpi_put_key(int src, int dst, int tag, char *key, int n, int send) { //get the dynar for src#dst char aux[INSTR_DEFAULT_STR_SIZE]; snprintf(aux, INSTR_DEFAULT_STR_SIZE, "%d#%d#%d#%d", src, dst, tag, send); xbt_dynar_t d = static_cast(xbt_dict_get_or_null(keys, aux)); if (d == nullptr) { d = xbt_dynar_new(sizeof(char *), &xbt_free_ref); xbt_dict_set(keys, aux, d, nullptr); } //generate the key static unsigned long long counter = 0; counter++; snprintf(key, n, "%d_%d_%d_%llu", src, dst, tag, counter); //push it char *a = static_cast (xbt_strdup(key)); xbt_dynar_push_as(d, char *, a); return key; } static char *TRACE_smpi_get_key(int src, int dst, int tag, char *key, int n, int send) { char aux[INSTR_DEFAULT_STR_SIZE]; snprintf(aux, INSTR_DEFAULT_STR_SIZE, "%d#%d#%d#%d", src, dst, tag, send==1?0:1); xbt_dynar_t d = static_cast(xbt_dict_get_or_null(keys, aux)); // first posted if(xbt_dynar_is_empty(d)){ TRACE_smpi_put_key(src, dst, tag, key, n, send); return key; } char *s = xbt_dynar_get_as (d, 0, char *); snprintf (key, n, "%s", s); xbt_dynar_remove_at (d, 0, nullptr); return key; } static xbt_dict_t process_category; static void cleanup_extra_data (instr_extra_data extra){ if(extra!=nullptr){ if(extra->sendcounts!=nullptr) xbt_free(extra->sendcounts); if(extra->recvcounts!=nullptr) xbt_free(extra->recvcounts); xbt_free(extra); } } void TRACE_internal_smpi_set_category (const char *category) { if (!TRACE_smpi_is_enabled()) return; //declare category TRACE_category (category); char processid[INSTR_DEFAULT_STR_SIZE]; snprintf (processid, INSTR_DEFAULT_STR_SIZE, "%p", SIMIX_process_self()); if (xbt_dict_get_or_null (process_category, processid)) xbt_dict_remove (process_category, processid); if (category != nullptr) xbt_dict_set (process_category, processid, xbt_strdup(category), nullptr); } const char *TRACE_internal_smpi_get_category () { if (!TRACE_smpi_is_enabled()) return nullptr; char processid[INSTR_DEFAULT_STR_SIZE]; snprintf (processid, INSTR_DEFAULT_STR_SIZE, "%p", SIMIX_process_self()); return static_cast(xbt_dict_get_or_null (process_category, processid)); } void TRACE_smpi_alloc() { keys = xbt_dict_new_homogeneous(xbt_dynar_free_voidp); process_category = xbt_dict_new_homogeneous(xbt_free_f); } void TRACE_smpi_release() { xbt_dict_free(&keys); xbt_dict_free(&process_category); } void TRACE_smpi_init(int rank) { if (!TRACE_smpi_is_enabled()) return; char str[INSTR_DEFAULT_STR_SIZE]; smpi_container(rank, str, INSTR_DEFAULT_STR_SIZE); container_t father; if (TRACE_smpi_is_grouped()){ father = PJ_container_get (SIMIX_host_self_get_name()); }else{ father = PJ_container_get_root (); } xbt_assert(father!=nullptr, "Could not find a parent for mpi rank %s at function %s", str, __FUNCTION__); #if HAVE_PAPI container_t container = #endif PJ_container_new(str, INSTR_SMPI, father); #if HAVE_PAPI papi_counter_t counters = smpi_process_papi_counters(); for (auto& it : counters) { /** * Check whether this variable already exists or not. Otherwise, it will be created * multiple times but only the last one would be used... */ if (PJ_type_get_or_null(it.first.c_str(), container->type) == nullptr) { PJ_type_variable_new(it.first.c_str(), nullptr, container->type); } } #endif } void TRACE_smpi_finalize(int rank) { if (!TRACE_smpi_is_enabled()) return; char str[INSTR_DEFAULT_STR_SIZE]; container_t container = PJ_container_get(smpi_container(rank, str, INSTR_DEFAULT_STR_SIZE)); PJ_container_remove_from_parent (container); PJ_container_free (container); } void TRACE_smpi_collective_in(int rank, int root, const char *operation, instr_extra_data extra) { if (!TRACE_smpi_is_enabled()) { cleanup_extra_data(extra); return; } char str[INSTR_DEFAULT_STR_SIZE]; smpi_container(rank, str, INSTR_DEFAULT_STR_SIZE); container_t container = PJ_container_get (str); type_t type = PJ_type_get ("MPI_STATE", container->type); const char *color = instr_find_color (operation); val_t value = PJ_value_get_or_new (operation, color, type); new_pajePushStateWithExtra (SIMIX_get_clock(), container, type, value, static_cast(extra)); } void TRACE_smpi_collective_out(int rank, int root, const char *operation) { if (!TRACE_smpi_is_enabled()) return; char str[INSTR_DEFAULT_STR_SIZE]; smpi_container(rank, str, INSTR_DEFAULT_STR_SIZE); container_t container = PJ_container_get (str); type_t type = PJ_type_get ("MPI_STATE", container->type); new_pajePopState (SIMIX_get_clock(), container, type); } void TRACE_smpi_computing_init(int rank) { //first use, initialize the color in the trace if (!TRACE_smpi_is_enabled() || !TRACE_smpi_is_computing()) return; char str[INSTR_DEFAULT_STR_SIZE]; smpi_container(rank, str, INSTR_DEFAULT_STR_SIZE); container_t container = PJ_container_get (str); type_t type = PJ_type_get ("MPI_STATE", container->type); const char *color = instr_find_color ("computing"); val_t value = PJ_value_get_or_new ("computing", color, type); new_pajePushState (SIMIX_get_clock(), container, type, value); } void TRACE_smpi_computing_in(int rank, instr_extra_data extra) { //do not forget to set the color first, otherwise this will explode if (!TRACE_smpi_is_enabled()|| !TRACE_smpi_is_computing()) { cleanup_extra_data(extra); return; } char str[INSTR_DEFAULT_STR_SIZE]; smpi_container(rank, str, INSTR_DEFAULT_STR_SIZE); container_t container = PJ_container_get (str); type_t type = PJ_type_get ("MPI_STATE", container->type); val_t value = PJ_value_get_or_new ("computing", nullptr, type); new_pajePushStateWithExtra (SIMIX_get_clock(), container, type, value, static_cast(extra)); } void TRACE_smpi_computing_out(int rank) { if (!TRACE_smpi_is_enabled()|| !TRACE_smpi_is_computing()) return; char str[INSTR_DEFAULT_STR_SIZE]; smpi_container(rank, str, INSTR_DEFAULT_STR_SIZE); container_t container = PJ_container_get (str); type_t type = PJ_type_get ("MPI_STATE", container->type); new_pajePopState (SIMIX_get_clock(), container, type); } void TRACE_smpi_sleeping_init(int rank) { //first use, initialize the color in the trace if (!TRACE_smpi_is_enabled() || !TRACE_smpi_is_sleeping()) return; char str[INSTR_DEFAULT_STR_SIZE]; smpi_container(rank, str, INSTR_DEFAULT_STR_SIZE); container_t container = PJ_container_get (str); type_t type = PJ_type_get ("MPI_STATE", container->type); const char *color = instr_find_color ("sleeping"); val_t value = PJ_value_get_or_new ("sleeping", color, type); new_pajePushState (SIMIX_get_clock(), container, type, value); } void TRACE_smpi_sleeping_in(int rank, instr_extra_data extra) { //do not forget to set the color first, otherwise this will explode if (!TRACE_smpi_is_enabled()|| !TRACE_smpi_is_sleeping()) { cleanup_extra_data(extra); return; } char str[INSTR_DEFAULT_STR_SIZE]; smpi_container(rank, str, INSTR_DEFAULT_STR_SIZE); container_t container = PJ_container_get (str); type_t type = PJ_type_get ("MPI_STATE", container->type); val_t value = PJ_value_get_or_new ("sleeping", nullptr, type); new_pajePushStateWithExtra (SIMIX_get_clock(), container, type, value, static_cast(extra)); } void TRACE_smpi_sleeping_out(int rank) { if (!TRACE_smpi_is_enabled()|| !TRACE_smpi_is_sleeping()) return; char str[INSTR_DEFAULT_STR_SIZE]; smpi_container(rank, str, INSTR_DEFAULT_STR_SIZE); container_t container = PJ_container_get (str); type_t type = PJ_type_get ("MPI_STATE", container->type); new_pajePopState (SIMIX_get_clock(), container, type); } void TRACE_smpi_testing_in(int rank, instr_extra_data extra) { //do not forget to set the color first, otherwise this will explode if (!TRACE_smpi_is_enabled()) { cleanup_extra_data(extra); return; } char str[INSTR_DEFAULT_STR_SIZE]; smpi_container(rank, str, INSTR_DEFAULT_STR_SIZE); container_t container = PJ_container_get (str); type_t type = PJ_type_get ("MPI_STATE", container->type); val_t value = PJ_value_get_or_new ("test", nullptr, type); new_pajePushStateWithExtra (SIMIX_get_clock(), container, type, value, static_cast(extra)); } void TRACE_smpi_testing_out(int rank) { if (!TRACE_smpi_is_enabled()) return; char str[INSTR_DEFAULT_STR_SIZE]; smpi_container(rank, str, INSTR_DEFAULT_STR_SIZE); container_t container = PJ_container_get (str); type_t type = PJ_type_get ("MPI_STATE", container->type); new_pajePopState (SIMIX_get_clock(), container, type); } void TRACE_smpi_ptp_in(int rank, int src, int dst, const char *operation, instr_extra_data extra) { if (!TRACE_smpi_is_enabled()) { cleanup_extra_data(extra); return; } char str[INSTR_DEFAULT_STR_SIZE]; smpi_container(rank, str, INSTR_DEFAULT_STR_SIZE); container_t container = PJ_container_get (str); type_t type = PJ_type_get ("MPI_STATE", container->type); const char *color = instr_find_color (operation); val_t value = PJ_value_get_or_new (operation, color, type); new_pajePushStateWithExtra (SIMIX_get_clock(), container, type, value, static_cast(extra)); } void TRACE_smpi_ptp_out(int rank, int src, int dst, const char *operation) { if (!TRACE_smpi_is_enabled()) return; char str[INSTR_DEFAULT_STR_SIZE]; smpi_container(rank, str, INSTR_DEFAULT_STR_SIZE); container_t container = PJ_container_get (str); type_t type = PJ_type_get ("MPI_STATE", container->type); new_pajePopState (SIMIX_get_clock(), container, type); } void TRACE_smpi_send(int rank, int src, int dst, int tag, int size) { if (!TRACE_smpi_is_enabled()) return; char key[INSTR_DEFAULT_STR_SIZE] = {0}; TRACE_smpi_get_key(src, dst, tag, key, INSTR_DEFAULT_STR_SIZE,1); char str[INSTR_DEFAULT_STR_SIZE]; smpi_container(src, str, INSTR_DEFAULT_STR_SIZE); container_t container = PJ_container_get (str); type_t type = PJ_type_get ("MPI_LINK", PJ_type_get_root()); XBT_DEBUG("Send tracing from %d to %d, tag %d, with key %s", src, dst, tag, key); new_pajeStartLinkWithSize (SIMIX_get_clock(), PJ_container_get_root(), type, container, "PTP", key, size); } void TRACE_smpi_recv(int rank, int src, int dst, int tag) { if (!TRACE_smpi_is_enabled()) return; char key[INSTR_DEFAULT_STR_SIZE] = {0}; TRACE_smpi_get_key(src, dst, tag, key, INSTR_DEFAULT_STR_SIZE,0); char str[INSTR_DEFAULT_STR_SIZE]; smpi_container(dst, str, INSTR_DEFAULT_STR_SIZE); container_t container = PJ_container_get (str); type_t type = PJ_type_get ("MPI_LINK", PJ_type_get_root()); XBT_DEBUG("Recv tracing from %d to %d, tag %d, with key %s", src, dst, tag, key); new_pajeEndLink (SIMIX_get_clock(), PJ_container_get_root(), type, container, "PTP", key); } SimGrid-3.14.159/src/smpi/private.hpp000644 001750 001750 00000002302 13031031273 021010 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2016. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef SMPI_PRIVATE_HPP #define SMPI_PRIVATE_HPP #include "src/instr/instr_smpi.h" #include #include /** * Get the address of the beginning of the memory page where addr is located. * Note that we use an integer division here, so (a/b)*b is not a, unless a%b == 0 * * This is used when privatizing. */ #define TOPAGE(addr) (void *)(((unsigned long)(addr) / xbt_pagesize) * xbt_pagesize) #if HAVE_PAPI typedef std::vector> papi_counter_t; XBT_PRIVATE papi_counter_t& smpi_process_papi_counters(); XBT_PRIVATE int smpi_process_papi_event_set(); #endif extern std::unordered_map location2speedup; /** @brief Returns the last call location (filename, linenumber). Process-specific. */ extern "C" { XBT_PUBLIC(smpi_trace_call_location_t*) smpi_process_get_call_location(); XBT_PUBLIC(smpi_trace_call_location_t*) smpi_trace_get_call_location(); } #endif SimGrid-3.14.159/src/smpi/smpirun.in000755 001750 001750 00000037546 13031031276 020702 0ustar00mquinsonmquinson000000 000000 #! /bin/sh # Copyright (c) 2007-2016, The SimGrid Team. All rights reserved. # This program is free software; you can redistribute it and/or modify it # under the terms of the license (GNU LGPL) which comes with this package. @CMAKE_SMPI_COMMAND@ SIMGRID_VERSION="@SIMGRID_VERSION_STRING@" SIMGRID_GITHASH="@SIMGRID_GITHASH@" DEFAULT_LOOPBACK_BANDWIDTH="498000000Bps" DEFAULT_LOOPBACK_LATENCY="0.000004s" DEFAULT_NETWORK_BANDWIDTH="$((26 * 1024 * 1024))Bps" DEFAULT_NETWORK_LATENCY="0.000005s" DEFAULT_NUMPROCS="4" DEFAULT_SPEED="100flops" LOOPBACK_BANDWIDTH="${DEFAULT_LOOPBACK_BANDWIDTH}" LOOPBACK_LATENCY="${DEFAULT_LOOPBACK_LATENCY}" NETWORK_BANDWIDTH="${DEFAULT_NETWORK_BANDWIDTH}" NETWORK_LATENCY="${DEFAULT_NETWORK_LATENCY}" SPEED="${DEFAULT_SPEED}" PRIVATIZE="--cfg=smpi/privatize-global-variables:yes" SIMOPTS="--cfg=surf/precision:1e-9 --cfg=network/model:SMPI --cfg=network/TCP-gamma:4194304" #usage to print the way this script should be called usage () { cat < -hostfile program [program-options] Options: -keep-temps # don't remove the generated files after execution -wrapper # use command to run the program (e.g. "valgrind") -map # display the machine on which each process rank is mapped -np # use that amount of processes from the hostfile. # By default, all processes of the hostfile are used. -no-privatize # Disable the globals privatization, that is activated by default -trace-ti # activate time independant tracing (for replay, default in smpi_simgrid.txt) -trace # activate tracing (Paje, default in smpi_simgrid.trace) -trace-comment # put a comment on the top of the trace file -trace-comment-file # put file contents on the top of the trace file as comment -trace-grouped # group MPI processes by location -trace-resource # trace resource utilization -trace-viva # generate configuration for Viva's GraphView -trace-file # name of the tracefile (simgrid_smpi.trace) -ext # additional parameter (reserved) -version # Displays the SimGrid version (human readable) -git-version # Displays the git hash of SimGrid or (deprecated usage): $0 [-keep-temps] [-np ] [-bandwidth ] [-latency ] program [program-options] EOF } #check if we have at least one parameter if [ $# -eq 0 ] then usage exit fi EXTOPT="" WRAPPER="" HOSTFILE="" HOSTFILETMP=0 MAPOPT=0 REPLAY=0 unset pid trapped_signals="HUP INT QUIT ILL ABRT SEGV FPE ALRM TERM USR1 USR2 BUS" smpirun_cleanup() { if [ -z "${KEEP}" ] ; then if [ -z "${PLATFORM}" -a -n "$PLATFORMTMP" ]; then rm -f ${PLATFORMTMP} PLATFORMTMP="" fi if [ ${HOSTFILETMP} = 1 -a -n "$HOSTFILE" ] ; then rm -f ${HOSTFILE} HOSTFILE="" fi if [ ${UNROLLEDHOSTFILETMP} = 1 -a -n "$UNROLLEDHOSTFILE" ] ; then rm -f ${UNROLLEDHOSTFILE} UNROLLEDHOSTFILE="" fi if [ -n ${APPLICATIONTMP} ]; then rm -f ${APPLICATIONTMP} APPLICATIONTMP="" fi fi } smpirun_trap() { local sig sig="$1" # Cleanup and kill the child process: smpirun_cleanup if ! [ -z "$pid" ]; then kill -TERM $pid fi unset pid # Raise the same signal again (remove the traps first): trap - $trapped_signals kill -$sig $$ # This should never happen: kill -ABRT $$ kill -TERM $$ } for s in $trapped_signals; do trap "smpirun_trap $s" $s done while true; do case "$1" in "-np" | "-n") NUMPROCS="$2" shift 2 ;; "-bandwidth") NETWORK_BANDWIDTH="$2" shift 2 ;; "-latency") NETWORK_LATENCY="$2" shift 2 ;; "-platform") PLATFORM="$2" if [ ! -f "${PLATFORM}" ]; then echo "[`basename $0`] ** error: the file '${PLATFORM}' does not exist. Aborting." exit 1 fi shift 2 ;; "-hostfile") HOSTFILE="$2" if [ ! -f "${HOSTFILE}" ]; then echo "[`basename $0`] ** error: the file '${HOSTFILE}' does not exist. Aborting." exit 1 fi shift 2 ;; "-machinefile") HOSTFILE="$2" if [ ! -f "${HOSTFILE}" ]; then echo "[`basename $0`] ** error: the file '${HOSTFILE}' does not exist. Aborting." exit 1 fi shift 2 ;; "-ext") EXTOPT="$2" shift 2 ;; "-no-privatize") PRIVATIZE="" shift 1 ;; "-map") MAPOPT=1 shift 1 ;; "-trace") TRACE_ACTIVE="true" shift 1 ;; "-trace-ti") TRACE_ACTIVE="true" TRACE_TI_ACTIVE="true" shift 1 ;; "-trace-comment") TRACE_COMMENT="$2" shift 2 ;; "-trace-comment-file") TRACE_COMMENT_FILE="$2" shift 2 ;; "-trace-file") TRACE_FILENAME="$2" shift 2 ;; "-trace-grouped") TRACE_GROUPED="true" shift 1 ;; "-trace-resource") TRACE_RESOURCE="true" shift 1 ;; "-trace-viva") TRACE_VIVA="true" shift 1 ;; "-keep-temps") KEEP="true" shift 1 ;; "-wrapper") WRAPPER="$2" shift 2 ;; "-help" | "--help" | "-h") usage exit 0 ;; "-version" | "--version" | "-v") printf '%b\n' "$SIMGRID_VERSION" exit 0 ;; "-git-version" | "--git-version") printf '%b\n' "$SIMGRID_GITHASH" exit 0 ;; "--cfg="*|"--log="*) for OPT in ${1#*=} do SIMOPTS="$SIMOPTS ${1%%=*}=$OPT" done shift 1 ;; "-foreground") # Nothing to do, compatibility. shift 1 ;; *) break ;; esac done if [ -n "$WRAPPER" ]; then EXEC="$WRAPPER $1" else EXEC="$1" fi shift # steel --cfg and --logs options while [ $# -gt 0 ]; do case "$1" in "--cfg="*|"--log="*) for OPT in ${1#*=} do SIMOPTS="$SIMOPTS ${1%%=*}=$OPT" done shift 1 ;; *) PROC_ARGS="${PROC_ARGS:+$PROC_ARGS }$1" shift ;; esac done if [ -z "${HOSTFILE}" ] && [ -z "${PLATFORM}" ] ; then echo "No hostfile nor platform specified." usage exit 1 fi if [ -z "${HOSTFILE}" ] ; then HOSTFILETMP=1 HOSTFILE="$(mktemp smpitmp-hostfXXXXXX)" perl -ne 'print "$1\n" if /.*.*/' ${PLATFORM} > ${HOSTFILE} perl -ne 'if (/.*> ${HOSTFILE} fi UNROLLEDHOSTFILETMP=0 #parse if our lines are terminated by :num_process multiple_processes=`grep -c ":" $HOSTFILE` if [ "${multiple_processes}" -gt 0 ] ; then UNROLLEDHOSTFILETMP=1 UNROLLEDHOSTFILE="$(mktemp smpitmp-hostfXXXXXX)" perl -ne ' do{ for ( 1 .. $2 ) { print "$1\n" } } if /(.*?):(\d+).*/' ${HOSTFILE} > ${UNROLLEDHOSTFILE} if [ ${HOSTFILETMP} = 1 ] ; then rm ${HOSTFILE} HOSTFILETMP=0 fi HOSTFILE=$UNROLLEDHOSTFILE fi # Don't use wc -l to compute it to avoid issues with trailing \n at EOF hostfile_procs=`grep -c "[a-zA-Z0-9]" $HOSTFILE` if [ ${hostfile_procs} = 0 ] ; then echo "[`basename $0`] ** error: the hostfile '${HOSTFILE}' is empty. Aborting." >&2 exit 1 fi if [ -z "${NUMPROCS}" ] ; then # Use the amount of processes in the hostfile as default value for the -np parameter NUMPROCS=$hostfile_procs fi if [ ${NUMPROCS} -gt ${hostfile_procs} ] ; then echo "You requested to use ${NUMPROCS} processes, but there is only ${hostfile_procs} processes in your hostfile..." >&2 fi ##-------------------------------- DEFAULT or SPECIFIED PLATFORM -------------------------------------- if [ -z "${PLATFORM}" ]; then PLATFORMTMP="$(mktemp smpitmp-platfXXXXXX)" cat > ${PLATFORMTMP} < PLATFORMHEAD i=${NUMPROCS} while [ $i -gt 0 ]; do echo " " >> ${PLATFORMTMP} echo " " >> ${PLATFORMTMP} echo " " >> ${PLATFORMTMP} i=$((i - 1)) done i=${NUMPROCS} while [ $i -gt 0 ]; do j=${NUMPROCS} while [ $j -gt 0 ]; do if [ $i -eq $j ]; then echo " " >> ${PLATFORMTMP} else echo " " >> ${PLATFORMTMP} fi j=$((j - 1)) done i=$((i - 1)) done cat >> ${PLATFORMTMP} < PLATFORMFOOT else PLATFORMTMP=${PLATFORM} fi ##-------------------------------- end DEFAULT or SPECIFIED PLATFORM -------------------------------------- ##-------------------------------- DEFAULT APPLICATION -------------------------------------- APPLICATIONTMP="$(mktemp smpitmp-appXXXXXX)" #APPLICATIONTMP="app.xml" cat > ${APPLICATIONTMP} < APPLICATIONHEAD ##---- cache hostnames of hostfile--------------- if [ -n "${HOSTFILE}" ] && [ -f ${HOSTFILE} ]; then hostnames=$(cat ${HOSTFILE} | tr '\n\r' ' ') NUMHOSTS=$(cat ${HOSTFILE} | wc -l) fi if [ "${EXTOPT}" = "smpi_replay" ]; then APP_TRACES=$PROC_ARGS if [ -n "${APP_TRACES}" ] && [ -f "${APP_TRACES}" ]; then hosttraces=$(cat ${APP_TRACES} | tr '\n\r' ' ' ) NUMTRACES=$(cat ${APP_TRACES} | wc -l) REPLAY=1 else printf "File not found: %s\n" "${APP_TRACES:-\${APP_TRACES\}}" >&2 exit 1 fi fi ##---------------------------------------------------------- ## generate application.xml with hostnames from hostfile: ## the name of host_i (1<=i<=p, where -np p) is the line i in hostfile (where -hostfile hostfile), or "host$i" if ## hostfile has less than i lines. ##---------------------------------------------------------- HAVE_SEQ="`which seq 2>/dev/null`" if [ -n "${HAVE_SEQ}" ]; then SEQ=`${HAVE_SEQ} 0 $((${NUMPROCS}-1))` else cnt=0 while [ $cnt -lt ${NUMPROCS} ] ; do SEQ="$SEQ $cnt" cnt=$((cnt + 1)); done fi set -- $hostnames ##---- generate tags------------------------------ #prepare arguments at once for ARG in $PROC_ARGS; do XML_ARGS="${XML_ARGS}"" " done for i in ${SEQ} do j=$(( $i % ${NUMHOSTS} + 1 )) host=$(eval "echo \${$j}") ##---- optional display of ranks to process mapping if [ ${MAPOPT} = 1 ]; then echo "[rank $i] -> $host" fi echo " " >> ${APPLICATIONTMP} if [ ${REPLAY} = 1 ]; then if [ ${NUMTRACES} -gt 1 ]; then echo " " >> ${APPLICATIONTMP} else echo " " >> ${APPLICATIONTMP} fi else echo ${XML_ARGS} >> ${APPLICATIONTMP} fi echo " " >> ${APPLICATIONTMP} done cat >> ${APPLICATIONTMP} < APPLICATIONFOOT ##-------------------------------- end DEFAULT APPLICATION -------------------------------------- ##---------------------- SMPI TRACING OPTIONS --------------------------------- if [ -n "${TRACE_ACTIVE}" ]; then #define trace filename if [ -n "${TRACE_TI_ACTIVE}" ]; then if [ -z "${TRACE_FILENAME}" ]; then TRACE_FILENAME="smpi_simgrid.txt" fi TRACEOPTIONS="--cfg=tracing:yes --cfg=tracing/filename:${TRACE_FILENAME} --cfg=tracing/smpi:yes --cfg=tracing/smpi/format:TI --cfg=tracing/smpi/computing:yes" else if [ -z "${TRACE_FILENAME}" ]; then TRACE_FILENAME="smpi_simgrid.trace" fi TRACEOPTIONS="--cfg=tracing:yes --cfg=tracing/filename:${TRACE_FILENAME} --cfg=tracing/smpi:yes" fi if [ -n "${TRACE_COMMENT}" ]; then TRACEOPTIONS="${TRACEOPTIONS} --cfg=tracing/comment:${TRACE_COMMENT}" fi if [ -n "${TRACE_COMMENT_FILE}" ]; then TRACEOPTIONS="${TRACEOPTIONS} --cfg=tracing/comment-file:${TRACE_COMMENT_FILE}" fi if [ -n "${TRACE_GROUPED}" ]; then TRACEOPTIONS="${TRACEOPTIONS} --cfg=tracing/smpi/group:yes" fi if [ -n "${TRACE_RESOURCE}" ]; then TRACEOPTIONS="${TRACEOPTIONS} --cfg=tracing/categorized:yes --cfg=tracing/uncategorized:yes" fi if [ -n "${TRACE_VIVA}" ]; then TRACEOPTIONS="${TRACEOPTIONS} --cfg=viva/categorized:smpi_cat.plist --cfg=viva/uncategorized:smpi_uncat.plist" fi fi ##---------------------- end SMPI TRACING OPTIONS --------------------------------- export SMPI_GLOBAL_SIZE=${NUMPROCS} if [ -n "${KEEP}" ] ; then echo ${EXEC} ${TRACEOPTIONS} ${SIMOPTS} ${PRIVATIZE} ${PLATFORMTMP} ${APPLICATIONTMP} if [ ${HOSTFILETMP} = 1 ] ; then echo "Generated hostfile ${HOSTFILE} kept." fi if [ ${UNROLLEDHOSTFILETMP} = 1 ] ; then echo "Generated unrolled hostfile ${UNROLLEDHOSTFILE} kept." fi fi # Execute the process # # The shell still need to be alive for the duration in order to do some cleanup after the process. # # We are going through great lengths in order to both keep stdin and be able to handle signals: # # * The job is launched in the background in order to be able to handle signals. # # * The FD 3 is used to temporarily store FD 1. This is because the shell connects FD 1 to /dev/null when the command # is launched in the background: this can be overriden in bash but not in standard bourne shell. exec 3<&0 ${EXEC} ${TRACEOPTIONS} ${SIMOPTS} ${PLATFORMTMP} ${APPLICATIONTMP} <&3 3>&- & pid=$! exec 3>&- wait $pid status=$? pid="" # Keep temporary files on failures to help debugging # if [ ${status} -ne 0 ] ; then if [ -z ${KEEP} ] ; then echo ${EXEC} ${TRACEOPTIONS} ${SIMOPTS} ${PRIVATIZE} ${PLATFORMTMP} ${APPLICATIONTMP} if [ ${HOSTFILETMP} = 1 ] ; then echo "Generated hostfile ${HOSTFILE} kept." fi if [ ${UNROLLEDHOSTFILETMP} = 1 ] ; then echo "Generated unrolled hostfile ${UNROLLEDHOSTFILE} kept." fi fi echo "Execution failed with code ${status}." KEEP=true fi smpirun_cleanup exit $status SimGrid-3.14.159/src/smpi/smpi_group.cpp000644 001750 001750 00000011350 13031031272 021517 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010, 2013-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_group, smpi, "Logging specific to SMPI (group)"); typedef struct s_smpi_mpi_group { int size; int *rank_to_index_map; xbt_dict_t index_to_rank_map; int refcount; } s_smpi_mpi_group_t; static s_smpi_mpi_group_t mpi_MPI_GROUP_EMPTY = { 0, /* size */ nullptr, /* rank_to_index_map */ nullptr, /* index_to_rank_map */ 1, /* refcount: start > 0 so that this group never gets freed */ }; MPI_Group MPI_GROUP_EMPTY = &mpi_MPI_GROUP_EMPTY; MPI_Group smpi_group_new(int size) { MPI_Group group; int i; group = xbt_new(s_smpi_mpi_group_t, 1); group->size = size; group->rank_to_index_map = xbt_new(int, size); group->index_to_rank_map = xbt_dict_new_homogeneous(xbt_free_f); group->refcount = 1; for (i = 0; i < size; i++) { group->rank_to_index_map[i] = MPI_UNDEFINED; } return group; } MPI_Group smpi_group_copy(MPI_Group origin) { MPI_Group group=origin; char *key; char *ptr_rank; xbt_dict_cursor_t cursor = nullptr; int i; if(origin != MPI_GROUP_NULL && origin != MPI_GROUP_EMPTY) { group = xbt_new(s_smpi_mpi_group_t, 1); group->size = origin->size; group->rank_to_index_map = xbt_new(int, group->size); group->index_to_rank_map = xbt_dict_new_homogeneous(xbt_free_f); group->refcount = 1; for (i = 0; i < group->size; i++) { group->rank_to_index_map[i] = origin->rank_to_index_map[i]; } xbt_dict_foreach(origin->index_to_rank_map, cursor, key, ptr_rank) { int * cp = static_cast(xbt_malloc(sizeof(int))); *cp=*reinterpret_cast(ptr_rank); xbt_dict_set(group->index_to_rank_map, key, cp, nullptr); } } return group; } void smpi_group_destroy(MPI_Group group) { if(group!= smpi_comm_group(MPI_COMM_WORLD) && group != MPI_GROUP_NULL && group != MPI_GROUP_EMPTY) smpi_group_unuse(group); } void smpi_group_set_mapping(MPI_Group group, int index, int rank) { int * val_rank; if (rank < group->size) { group->rank_to_index_map[rank] = index; if (index!=MPI_UNDEFINED ) { val_rank = static_cast(xbt_malloc(sizeof(int))); *val_rank = rank; char * key = bprintf("%d", index); xbt_dict_set(group->index_to_rank_map, key, val_rank, nullptr); xbt_free(key); } } } int smpi_group_index(MPI_Group group, int rank) { int index = MPI_UNDEFINED; if (0 <= rank && rank < group->size) { index = group->rank_to_index_map[rank]; } return index; } int smpi_group_rank(MPI_Group group, int index) { int * ptr_rank = nullptr; char * key = bprintf("%d", index); ptr_rank = static_cast(xbt_dict_get_or_null(group->index_to_rank_map, key)); xbt_free(key); if (ptr_rank==nullptr) return MPI_UNDEFINED; return *ptr_rank; } int smpi_group_use(MPI_Group group) { group->refcount++; return group->refcount; } int smpi_group_unuse(MPI_Group group) { group->refcount--; if (group->refcount <= 0) { xbt_free(group->rank_to_index_map); xbt_dict_free(&group->index_to_rank_map); xbt_free(group); return 0; } return group->refcount; } int smpi_group_size(MPI_Group group) { return group->size; } int smpi_group_compare(MPI_Group group1, MPI_Group group2) { int result; int i, index, rank, size; result = MPI_IDENT; if (smpi_group_size(group1) != smpi_group_size(group2)) { result = MPI_UNEQUAL; } else { size = smpi_group_size(group2); for (i = 0; i < size; i++) { index = smpi_group_index(group1, i); rank = smpi_group_rank(group2, index); if (rank == MPI_UNDEFINED) { result = MPI_UNEQUAL; break; } if (rank != i) { result = MPI_SIMILAR; } } } return result; } int smpi_group_incl(MPI_Group group, int n, int* ranks, MPI_Group* newgroup) { int i=0, index=0; if (n == 0) { *newgroup = MPI_GROUP_EMPTY; } else if (n == smpi_group_size(group)) { *newgroup = group; if(group!= smpi_comm_group(MPI_COMM_WORLD) && group != MPI_GROUP_NULL && group != smpi_comm_group(MPI_COMM_SELF) && group != MPI_GROUP_EMPTY) smpi_group_use(group); } else { *newgroup = smpi_group_new(n); for (i = 0; i < n; i++) { index = smpi_group_index(group, ranks[i]); smpi_group_set_mapping(*newgroup, index, i); } } return MPI_SUCCESS; } SimGrid-3.14.159/src/smpi/smpi_rma.cpp000644 001750 001750 00000030640 13031031272 021145 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2007-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "private.h" #include XBT_LOG_NEW_DEFAULT_SUBCATEGORY(smpi_rma, smpi, "Logging specific to SMPI (RMA operations)"); msg_bar_t creation_bar = nullptr; typedef struct s_smpi_mpi_win{ void* base; MPI_Aint size; int disp_unit; MPI_Comm comm; MPI_Info info; int assert; std::vector *requests; msg_bar_t bar; MPI_Win* connected_wins; char* name; int opened; MPI_Group group; } s_smpi_mpi_win_t; MPI_Win smpi_mpi_win_create( void *base, MPI_Aint size, int disp_unit, MPI_Info info, MPI_Comm comm){ MPI_Win win; int comm_size = smpi_comm_size(comm); int rank=smpi_comm_rank(comm); XBT_DEBUG("Creating window"); win = xbt_new(s_smpi_mpi_win_t, 1); win->base = base; win->size = size; win->disp_unit = disp_unit; win->assert = 0; win->info = info; if(info!=MPI_INFO_NULL) info->refcount++; win->comm = comm; win->name = nullptr; win->opened = 0; win->group = MPI_GROUP_NULL; win->requests = new std::vector(); win->connected_wins = xbt_new0(MPI_Win, comm_size); win->connected_wins[rank] = win; if(rank==0){ win->bar = MSG_barrier_init(comm_size); } mpi_coll_allgather_fun(&(win->connected_wins[rank]), sizeof(MPI_Win), MPI_BYTE, win->connected_wins, sizeof(MPI_Win), MPI_BYTE, comm); mpi_coll_bcast_fun(&(win->bar), sizeof(msg_bar_t), MPI_BYTE, 0, comm); mpi_coll_barrier_fun(comm); return win; } int smpi_mpi_win_free( MPI_Win* win){ //As per the standard, perform a barrier to ensure every async comm is finished MSG_barrier_wait((*win)->bar); delete (*win)->requests; xbt_free((*win)->connected_wins); if ((*win)->name != nullptr){ xbt_free((*win)->name); } if((*win)->info!=MPI_INFO_NULL){ MPI_Info_free(&(*win)->info); } mpi_coll_barrier_fun((*win)->comm); int rank=smpi_comm_rank((*win)->comm); if(rank == 0) MSG_barrier_destroy((*win)->bar); xbt_free(*win); *win = MPI_WIN_NULL; return MPI_SUCCESS; } void smpi_mpi_win_get_name(MPI_Win win, char* name, int* length){ if(win->name==nullptr){ *length=0; name=nullptr; return; } *length = strlen(win->name); strncpy(name, win->name, *length+1); } void smpi_mpi_win_get_group(MPI_Win win, MPI_Group* group){ if(win->comm != MPI_COMM_NULL){ *group = smpi_comm_group(win->comm); } } void smpi_mpi_win_set_name(MPI_Win win, char* name){ win->name = xbt_strdup(name); } int smpi_mpi_win_fence( int assert, MPI_Win win){ XBT_DEBUG("Entering fence"); if(win->opened==0) win->opened=1; if(assert != MPI_MODE_NOPRECEDE){ MSG_barrier_wait(win->bar); std::vector *reqs = win->requests; int size = static_cast(reqs->size()); // start all requests that have been prepared by another process for(auto req: *reqs){ if (req && (req->flags & PREPARED)) smpi_mpi_start(req); } MPI_Request* treqs = &(*reqs)[0]; smpi_mpi_waitall(size,treqs,MPI_STATUSES_IGNORE); } win->assert = assert; MSG_barrier_wait(win->bar); XBT_DEBUG("Leaving fence "); return MPI_SUCCESS; } int smpi_mpi_put( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Win win) { if(win->opened==0)//check that post/start has been done return MPI_ERR_WIN; //get receiver pointer MPI_Win recv_win = win->connected_wins[target_rank]; void* recv_addr = static_cast ( static_cast(recv_win->base) + target_disp * recv_win->disp_unit); XBT_DEBUG("Entering MPI_Put to %d", target_rank); if(target_rank != smpi_comm_rank(win->comm)){ //prepare send_request MPI_Request sreq = smpi_rma_send_init(origin_addr, origin_count, origin_datatype, smpi_process_index(), smpi_group_index(smpi_comm_group(win->comm),target_rank), SMPI_RMA_TAG+1, win->comm, MPI_OP_NULL); //prepare receiver request MPI_Request rreq = smpi_rma_recv_init(recv_addr, target_count, target_datatype, smpi_process_index(), smpi_group_index(smpi_comm_group(win->comm),target_rank), SMPI_RMA_TAG+1, recv_win->comm, MPI_OP_NULL); //push request to receiver's win recv_win->requests->push_back(rreq); //start send smpi_mpi_start(sreq); //push request to sender's win win->requests->push_back(sreq); }else{ smpi_datatype_copy(origin_addr, origin_count, origin_datatype, recv_addr, target_count, target_datatype); } return MPI_SUCCESS; } int smpi_mpi_get( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Win win) { if(win->opened==0)//check that post/start has been done return MPI_ERR_WIN; //get sender pointer MPI_Win send_win = win->connected_wins[target_rank]; void* send_addr = static_cast(static_cast(send_win->base) + target_disp * send_win->disp_unit); XBT_DEBUG("Entering MPI_Get from %d", target_rank); if(target_rank != smpi_comm_rank(win->comm)){ //prepare send_request MPI_Request sreq = smpi_rma_send_init(send_addr, target_count, target_datatype, smpi_group_index(smpi_comm_group(win->comm),target_rank), smpi_process_index(), SMPI_RMA_TAG+2, send_win->comm, MPI_OP_NULL); //prepare receiver request MPI_Request rreq = smpi_rma_recv_init(origin_addr, origin_count, origin_datatype, smpi_group_index(smpi_comm_group(win->comm),target_rank), smpi_process_index(), SMPI_RMA_TAG+2, win->comm, MPI_OP_NULL); //start the send, with another process than us as sender. smpi_mpi_start(sreq); //push request to receiver's win send_win->requests->push_back(sreq); //start recv smpi_mpi_start(rreq); //push request to sender's win win->requests->push_back(rreq); }else{ smpi_datatype_copy(send_addr, target_count, target_datatype, origin_addr, origin_count, origin_datatype); } return MPI_SUCCESS; } int smpi_mpi_accumulate( void *origin_addr, int origin_count, MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype, MPI_Op op, MPI_Win win) { if(win->opened==0)//check that post/start has been done return MPI_ERR_WIN; //FIXME: local version //get receiver pointer MPI_Win recv_win = win->connected_wins[target_rank]; void* recv_addr = static_cast(static_cast(recv_win->base) + target_disp * recv_win->disp_unit); XBT_DEBUG("Entering MPI_Accumulate to %d", target_rank); //prepare send_request MPI_Request sreq = smpi_rma_send_init(origin_addr, origin_count, origin_datatype, smpi_process_index(), smpi_group_index(smpi_comm_group(win->comm),target_rank), SMPI_RMA_TAG+3, win->comm, op); //prepare receiver request MPI_Request rreq = smpi_rma_recv_init(recv_addr, target_count, target_datatype, smpi_process_index(), smpi_group_index(smpi_comm_group(win->comm),target_rank), SMPI_RMA_TAG+3, recv_win->comm, op); //push request to receiver's win recv_win->requests->push_back(rreq); //start send smpi_mpi_start(sreq); //push request to sender's win win->requests->push_back(sreq); return MPI_SUCCESS; } int smpi_mpi_win_start(MPI_Group group, int assert, MPI_Win win){ /* From MPI forum advices The call to MPI_WIN_COMPLETE does not return until the put call has completed at the origin; and the target window will be accessed by the put operation only after the call to MPI_WIN_START has matched a call to MPI_WIN_POST by the target process. This still leaves much choice to implementors. The call to MPI_WIN_START can block until the matching call to MPI_WIN_POST occurs at all target processes. One can also have implementations where the call to MPI_WIN_START is nonblocking, but the call to MPI_PUT blocks until the matching call to MPI_WIN_POST occurred; or implementations where the first two calls are nonblocking, but the call to MPI_WIN_COMPLETE blocks until the call to MPI_WIN_POST occurred; or even implementations where all three calls can complete before any target process called MPI_WIN_POST --- the data put must be buffered, in this last case, so as to allow the put to complete at the origin ahead of its completion at the target. However, once the call to MPI_WIN_POST is issued, the sequence above must complete, without further dependencies. */ //naive, blocking implementation. int i=0,j=0; int size = smpi_group_size(group); MPI_Request* reqs = xbt_new0(MPI_Request, size); while(j!=size){ int src=smpi_group_index(group,j); if(src!=smpi_process_index()){ reqs[i]=smpi_irecv_init(nullptr, 0, MPI_CHAR, src,SMPI_RMA_TAG+4, MPI_COMM_WORLD); i++; } j++; } size=i; smpi_mpi_startall(size, reqs); smpi_mpi_waitall(size, reqs, MPI_STATUSES_IGNORE); for(i=0;iopened++; //we're open for business ! win->group=group; smpi_group_use(group); return MPI_SUCCESS; } int smpi_mpi_win_post(MPI_Group group, int assert, MPI_Win win){ //let's make a synchronous send here int i=0,j=0; int size = smpi_group_size(group); MPI_Request* reqs = xbt_new0(MPI_Request, size); while(j!=size){ int dst=smpi_group_index(group,j); if(dst!=smpi_process_index()){ reqs[i]=smpi_mpi_send_init(nullptr, 0, MPI_CHAR, dst, SMPI_RMA_TAG+4, MPI_COMM_WORLD); i++; } j++; } size=i; smpi_mpi_startall(size, reqs); smpi_mpi_waitall(size, reqs, MPI_STATUSES_IGNORE); for(i=0;iopened++; //we're open for business ! win->group=group; smpi_group_use(group); return MPI_SUCCESS; } int smpi_mpi_win_complete(MPI_Win win){ if(win->opened==0) xbt_die("Complete called on already opened MPI_Win"); XBT_DEBUG("Entering MPI_Win_Complete"); int i=0,j=0; int size = smpi_group_size(win->group); MPI_Request* reqs = xbt_new0(MPI_Request, size); while(j!=size){ int dst=smpi_group_index(win->group,j); if(dst!=smpi_process_index()){ reqs[i]=smpi_mpi_send_init(nullptr, 0, MPI_CHAR, dst, SMPI_RMA_TAG+5, MPI_COMM_WORLD); i++; } j++; } size=i; XBT_DEBUG("Win_complete - Sending sync messages to %d processes", size); smpi_mpi_startall(size, reqs); smpi_mpi_waitall(size, reqs, MPI_STATUSES_IGNORE); for(i=0;i *reqqs = win->requests; size = static_cast(reqqs->size()); XBT_DEBUG("Win_complete - Finishing %d RMA calls", size); // start all requests that have been prepared by another process for (auto req: *reqqs){ if (req && (req->flags & PREPARED)) smpi_mpi_start(req); } MPI_Request* treqs = &(*reqqs)[0]; smpi_mpi_waitall(size,treqs,MPI_STATUSES_IGNORE); reqqs->clear(); smpi_group_unuse(win->group); win->opened--; //we're closed for business ! return MPI_SUCCESS; } int smpi_mpi_win_wait(MPI_Win win){ //naive, blocking implementation. XBT_DEBUG("Entering MPI_Win_Wait"); int i=0,j=0; int size = smpi_group_size(win->group); MPI_Request* reqs = xbt_new0(MPI_Request, size); while(j!=size){ int src=smpi_group_index(win->group,j); if(src!=smpi_process_index()){ reqs[i]=smpi_irecv_init(nullptr, 0, MPI_CHAR, src,SMPI_RMA_TAG+5, MPI_COMM_WORLD); i++; } j++; } size=i; XBT_DEBUG("Win_wait - Receiving sync messages from %d processes", size); smpi_mpi_startall(size, reqs); smpi_mpi_waitall(size, reqs, MPI_STATUSES_IGNORE); for(i=0;i *reqqs = win->requests; size = static_cast(reqqs->size()); XBT_DEBUG("Win_complete - Finishing %d RMA calls", size); // start all requests that have been prepared by another process for(auto req: *reqqs){ if (req && (req->flags & PREPARED)) smpi_mpi_start(req); } MPI_Request* treqs = &(*reqqs)[0]; smpi_mpi_waitall(size,treqs,MPI_STATUSES_IGNORE); reqqs->clear(); smpi_group_unuse(win->group); win->opened--; //we're opened for business ! return MPI_SUCCESS; } SimGrid-3.14.159/src/s4u/000755 001750 001750 00000000000 13031031271 016371 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/src/s4u/s4u_file.cpp000644 001750 001750 00000003024 13031031271 020606 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2015. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "xbt/log.h" #include "src/msg/msg_private.h" #include "simgrid/s4u/Actor.hpp" #include "simgrid/s4u/comm.hpp" #include "simgrid/s4u/host.hpp" #include "simgrid/s4u/Mailbox.hpp" XBT_LOG_NEW_DEFAULT_CATEGORY(s4u_file,"S4U files"); #include "simgrid/s4u/file.hpp" #include "simgrid/s4u/host.hpp" #include "simgrid/simix.h" namespace simgrid { namespace s4u { File::File(const char* fullpath, void* userdata) : path_(fullpath) { // this cannot fail because we get a xbt_die if the mountpoint does not exist pimpl_ = simcall_file_open(fullpath, Host::current()); } File::~File() { simcall_file_close(pimpl_, Host::current()); } sg_size_t File::read(sg_size_t size) { return simcall_file_read(pimpl_, size, Host::current()); } sg_size_t File::write(sg_size_t size) { return simcall_file_write(pimpl_,size, Host::current()); } sg_size_t File::size() { return simcall_file_get_size(pimpl_); } void File::seek(sg_size_t pos) { simcall_file_seek(pimpl_,pos,SEEK_SET); } sg_size_t File::tell() { return simcall_file_tell(pimpl_); } void File::move(const char*fullpath) { simcall_file_move(pimpl_,fullpath); } void File::unlink() { sg_host_t attached = Host::current(); // FIXME: we should check where this file is attached simcall_file_unlink(pimpl_,attached); } }} // namespace simgrid::s4u SimGrid-3.14.159/src/s4u/s4u_activity.cpp000644 001750 001750 00000001341 13031031271 021523 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "xbt/asserts.h" #include "xbt/log.h" #include "simgrid/s4u/Activity.hpp" XBT_LOG_EXTERNAL_CATEGORY(s4u); XBT_LOG_NEW_DEFAULT_SUBCATEGORY(s4u_activity,s4u,"S4U activities"); namespace simgrid { namespace s4u { Activity::Activity() { } Activity::~Activity() { } void Activity::setRemains(double remains) { xbt_assert(state_ == inited, "Cannot change the remaining amount of work once the Activity is started"); remains_ = remains; } } } SimGrid-3.14.159/src/s4u/s4u_mailbox.cpp000644 001750 001750 00000003542 13031031271 021327 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "xbt/log.h" #include "src/msg/msg_private.h" #include "src/simix/ActorImpl.hpp" #include "src/simix/smx_network_private.h" #include "simgrid/s4u/Mailbox.hpp" XBT_LOG_EXTERNAL_CATEGORY(s4u); XBT_LOG_NEW_DEFAULT_SUBCATEGORY(s4u_channel,s4u,"S4U Communication Mailboxes"); namespace simgrid { namespace s4u { const char *Mailbox::name() { return pimpl_->name; } MailboxPtr Mailbox::byName(const char*name) { // FIXME: there is a race condition here where two actors run Mailbox::byName // on a non-existent mailbox during the same scheduling round. Both will be // interrupted in the simcall creating the underlying simix mbox. // Only one simix object will be created, but two S4U objects will be created. // Only one S4U object will be stored in the hashmap and used, and the other // one will be leaked. smx_mailbox_t mbox = SIMIX_mbox_get_by_name(name); if (mbox == nullptr) mbox = simcall_mbox_create(name); return MailboxPtr(&mbox->piface_, true); } MailboxPtr Mailbox::byName(std::string name) { return byName(name.c_str()); } bool Mailbox::empty() { return pimpl_->comm_queue.empty(); } smx_activity_t Mailbox::front() { return pimpl_->comm_queue.empty() ? nullptr : pimpl_->comm_queue.front(); } void Mailbox::setReceiver(ActorPtr actor) { simcall_mbox_set_receiver(pimpl_, actor == nullptr ? nullptr : actor->pimpl_); } /** @brief get the receiver (process associated to the mailbox) */ ActorPtr Mailbox::receiver() { if(pimpl_->permanent_receiver == nullptr) return ActorPtr(); return ActorPtr(&pimpl_->permanent_receiver->getIface()); } } } SimGrid-3.14.159/src/s4u/s4u_storage.cpp000644 001750 001750 00000002752 13031031271 021342 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/s4u/storage.hpp" #include "xbt/lib.h" extern xbt_lib_t storage_lib; namespace simgrid { namespace s4u { boost::unordered_map *Storage::storages_ = new boost::unordered_map (); Storage::Storage(std::string name, smx_storage_t inferior) : name_(name), pimpl_(inferior) { storages_->insert({name, this}); } Storage::~Storage() = default; smx_storage_t Storage::inferior() { return pimpl_; } Storage &Storage::byName(const char*name) { s4u::Storage *res = nullptr; try { res = storages_->at(name); } catch (std::out_of_range& e) { smx_storage_t inferior = xbt_lib_get_elm_or_null(storage_lib,name); if (inferior == nullptr) xbt_die("Storage %s does not exist. Please only use the storages that are defined in your platform.", name); res = new Storage(name,inferior); } return *res; } const char*Storage::name() { return name_.c_str(); } sg_size_t Storage::sizeFree() { return simcall_storage_get_free_size(pimpl_); } sg_size_t Storage::sizeUsed() { return simcall_storage_get_used_size(pimpl_); } sg_size_t Storage::size() { return SIMIX_storage_get_size(pimpl_); } } /* namespace s4u */ } /* namespace simgrid */ SimGrid-3.14.159/src/s4u/s4u_mutex.cpp000644 001750 001750 00000002143 13031031271 021032 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "xbt/log.h" #include "src/msg/msg_private.h" #include "src/simix/smx_synchro_private.h" #include "simgrid/s4u/Mutex.hpp" namespace simgrid { namespace s4u { /** @brief Blocks the calling actor until the mutex can be obtained */ void Mutex::lock() { simcall_mutex_lock(mutex_); } /** @brief Release the ownership of the mutex, unleashing a blocked actor (if any) * * Will fail if the calling actor does not own the mutex. */ void Mutex::unlock() { simcall_mutex_unlock(mutex_); } /** @brief Acquire the mutex if it's free, and return false (without blocking) if not */ bool Mutex::try_lock() { return simcall_mutex_trylock(mutex_); } /** @brief Create a new mutex * * See @ref s4u_raii. */ MutexPtr Mutex::createMutex() { smx_mutex_t mutex = simcall_mutex_init(); return MutexPtr(&mutex->mutex(), false); } } } SimGrid-3.14.159/src/s4u/s4u_comm.cpp000644 001750 001750 00000011145 13031031271 020625 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "xbt/log.h" #include "src/msg/msg_private.h" #include "simgrid/s4u/comm.hpp" #include XBT_LOG_NEW_DEFAULT_SUBCATEGORY(s4u_comm,s4u_activity,"S4U asynchronous communications"); namespace simgrid { namespace s4u { Comm::~Comm() { } s4u::Comm &Comm::send_init(s4u::MailboxPtr chan) { s4u::Comm *res = new s4u::Comm(); res->sender_ = SIMIX_process_self(); res->mailbox_ = chan; return *res; } s4u::Comm &Comm::recv_init(s4u::MailboxPtr chan) { s4u::Comm *res = new s4u::Comm(); res->receiver_ = SIMIX_process_self(); res->mailbox_ = chan; return *res; } void Comm::setRate(double rate) { xbt_assert(state_==inited); rate_ = rate; } void Comm::setSrcData(void * buff) { xbt_assert(state_==inited); xbt_assert(dstBuff_ == nullptr, "Cannot set the src and dst buffers at the same time"); srcBuff_ = buff; } void Comm::setSrcDataSize(size_t size){ xbt_assert(state_==inited); srcBuffSize_ = size; } void Comm::setSrcData(void * buff, size_t size) { xbt_assert(state_==inited); xbt_assert(dstBuff_ == nullptr, "Cannot set the src and dst buffers at the same time"); srcBuff_ = buff; srcBuffSize_ = size; } void Comm::setDstData(void ** buff) { xbt_assert(state_==inited); xbt_assert(srcBuff_ == nullptr, "Cannot set the src and dst buffers at the same time"); dstBuff_ = buff; } size_t Comm::getDstDataSize(){ xbt_assert(state_==finished); return dstBuffSize_; } void Comm::setDstData(void ** buff, size_t size) { xbt_assert(state_==inited); xbt_assert(srcBuff_ == nullptr, "Cannot set the src and dst buffers at the same time"); dstBuff_ = buff; dstBuffSize_ = size; } void Comm::start() { xbt_assert(state_ == inited); if (srcBuff_ != nullptr) { // Sender side pimpl_ = simcall_comm_isend(sender_, mailbox_->getImpl(), remains_, rate_, srcBuff_, srcBuffSize_, matchFunction_, cleanFunction_, copyDataFunction_, userData_, detached_); } else if (dstBuff_ != nullptr) { // Receiver side pimpl_ = simcall_comm_irecv(receiver_, mailbox_->getImpl(), dstBuff_, &dstBuffSize_, matchFunction_, copyDataFunction_, userData_, rate_); } else { xbt_die("Cannot start a communication before specifying whether we are the sender or the receiver"); } state_ = started; } void Comm::wait() { xbt_assert(state_ == started || state_ == inited); if (state_ == started) simcall_comm_wait(pimpl_, -1/*timeout*/); else {// p_state == inited. Save a simcall and do directly a blocking send/recv if (srcBuff_ != nullptr) { simcall_comm_send(sender_, mailbox_->getImpl(), remains_, rate_, srcBuff_, srcBuffSize_, matchFunction_, copyDataFunction_, userData_, -1 /*timeout*/); } else { simcall_comm_recv(receiver_, mailbox_->getImpl(), dstBuff_, &dstBuffSize_, matchFunction_, copyDataFunction_, userData_, -1/*timeout*/, rate_); } } state_ = finished; delete this; } void Comm::wait(double timeout) { xbt_assert(state_ == started || state_ == inited); if (state_ == started) { simcall_comm_wait(pimpl_, timeout); state_ = finished; return; } // It's not started yet. Do it in one simcall if (srcBuff_ != nullptr) { simcall_comm_send(sender_, mailbox_->getImpl(), remains_, rate_, srcBuff_, srcBuffSize_, matchFunction_, copyDataFunction_, userData_, timeout); } else { // Receiver simcall_comm_recv(receiver_, mailbox_->getImpl(), dstBuff_, &dstBuffSize_, matchFunction_, copyDataFunction_, userData_, timeout, rate_); } state_ = finished; delete this; } s4u::Comm &Comm::send_async(MailboxPtr dest, void *data, int simulatedSize) { s4u::Comm &res = s4u::Comm::send_init(dest); res.setRemains(simulatedSize); res.srcBuff_ = data; res.srcBuffSize_ = sizeof(void*); res.start(); return res; } s4u::Comm &Comm::recv_async(MailboxPtr dest, void **data) { s4u::Comm &res = s4u::Comm::recv_init(dest); res.setDstData(data); res.start(); return res; } bool Comm::test() { xbt_assert(state_ == inited || state_ == started || state_ == finished); if (state_ == finished) xbt_die("Don't call test on a finished comm."); if (state_ == inited) { this->start(); } if(simcall_comm_test(pimpl_)){ state_ = finished; delete this; return true; } return false; } } } SimGrid-3.14.159/src/s4u/s4u_conditionVariable.cpp000644 001750 001750 00000003611 13031031271 023325 0ustar00mquinsonmquinson000000 000000 #include #include #include #include #include "src/simix/smx_synchro_private.h" #include "simgrid/s4u/conditionVariable.hpp" #include "simgrid/simix.h" namespace simgrid { namespace s4u { ConditionVariablePtr ConditionVariable::createConditionVariable() { smx_cond_t cond = simcall_cond_init(); return ConditionVariablePtr(&cond->cond_, false); } /** * Wait functions */ void ConditionVariable::wait(std::unique_lock& lock) { simcall_cond_wait(cond_, lock.mutex()->mutex_); } std::cv_status s4u::ConditionVariable::wait_for(std::unique_lock& lock, double timeout) { // The simcall uses -1 for "any timeout" but we don't want this: if (timeout < 0) timeout = 0.0; try { simcall_cond_wait_timeout(cond_, lock.mutex()->mutex_, timeout); return std::cv_status::no_timeout; } catch (xbt_ex& e) { // If the exception was a timeout, we have to take the lock again: if (e.category == timeout_error) { try { lock.mutex()->lock(); return std::cv_status::timeout; } catch (...) { std::terminate(); } } // Another exception: should we reaquire the lock? std::terminate(); } catch (...) { std::terminate(); } } std::cv_status ConditionVariable::wait_until(std::unique_lock& lock, double timeout_time) { double now = SIMIX_get_clock(); double timeout; if (timeout_time < now) timeout = 0.0; else timeout = timeout_time - now; return this->wait_for(lock, timeout); } /** * Notify functions */ void ConditionVariable::notify_one() { simcall_cond_signal(cond_); } void ConditionVariable::notify_all() { simcall_cond_broadcast(cond_); } void intrusive_ptr_add_ref(ConditionVariable* cond) { intrusive_ptr_add_ref(cond->cond_); } void intrusive_ptr_release(ConditionVariable* cond) { intrusive_ptr_release(cond->cond_); } } } SimGrid-3.14.159/src/s4u/s4u_host.cpp000644 001750 001750 00000015574 13031031271 020661 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include #include #include #include "simgrid/s4u/engine.hpp" #include "simgrid/s4u/host.hpp" #include "simgrid/s4u/storage.hpp" #include "simgrid/simix.hpp" #include "src/kernel/routing/NetCard.hpp" #include "src/msg/msg_private.h" #include "src/simix/ActorImpl.hpp" #include "src/simix/smx_private.h" #include "src/surf/HostImpl.hpp" #include "src/surf/cpu_interface.hpp" #include "xbt/log.h" XBT_LOG_EXTERNAL_CATEGORY(surf_route); std::unordered_map host_list; // FIXME: move it to Engine int MSG_HOST_LEVEL = -1; int USER_HOST_LEVEL = -1; namespace simgrid { namespace xbt { template class Extendable; } namespace s4u { simgrid::xbt::signal Host::onCreation; simgrid::xbt::signal Host::onDestruction; simgrid::xbt::signal Host::onStateChange; Host::Host(const char* name) : name_(name) { xbt_assert(Host::by_name_or_null(name) == nullptr, "Refusing to create a second host named '%s'.", name); host_list[name_] = this; } Host::~Host() { xbt_assert(currentlyDestroying_, "Please call h->destroy() instead of manually deleting it."); delete pimpl_; if (pimpl_netcard != nullptr) // not removed yet by a children class simgrid::s4u::Engine::instance()->netcardUnregister(pimpl_netcard); delete pimpl_cpu; delete mounts; } /** @brief Fire the required callbacks and destroy the object * * Don't delete directly an Host, call h->destroy() instead. * * This is cumbersome but this is the simplest solution to ensure that the * onDestruction() callback receives a valid object (because of the destructor * order in a class hierarchy). */ void Host::destroy() { if (!currentlyDestroying_) { currentlyDestroying_ = true; onDestruction(*this); host_list.erase(name_); delete this; } } Host* Host::by_name(std::string name) { return host_list.at(name); // Will raise a std::out_of_range if the host does not exist } Host* Host::by_name_or_null(const char* name) { return by_name_or_null(std::string(name)); } Host* Host::by_name_or_null(std::string name) { if (host_list.find(name) == host_list.end()) return nullptr; return host_list.at(name); } Host *Host::current(){ smx_actor_t smx_proc = SIMIX_process_self(); if (smx_proc == nullptr) xbt_die("Cannot call Host::current() from the maestro context"); return smx_proc->host; } void Host::turnOn() { if (isOff()) { simgrid::simix::kernelImmediate([&]{ this->extension()->turnOn(); this->pimpl_cpu->turnOn(); }); } } void Host::turnOff() { if (isOn()) { simgrid::simix::kernelImmediate(std::bind(SIMIX_host_off, this, SIMIX_process_self())); } } bool Host::isOn() { return this->pimpl_cpu->isOn(); } int Host::pstatesCount() const { return this->pimpl_cpu->getNbPStates(); } /** * \brief Find a route toward another host * * \param dest [IN] where to * \param links [OUT] where to store the list of links (must exist, cannot be nullptr). * \param latency [OUT] where to store the latency experienced on the path (or nullptr if not interested) * It is the caller responsibility to initialize latency to 0 (we add to provided route) * \pre links!=nullptr * * walk through the routing components tree and find a route between hosts * by calling each "get_route" function in each routing component. */ void Host::routeTo(Host* dest, std::vector* links, double* latency) { simgrid::kernel::routing::NetZoneImpl::getGlobalRoute(pimpl_netcard, dest->pimpl_netcard, links, latency); if (XBT_LOG_ISENABLED(surf_route, xbt_log_priority_debug)) { XBT_CDEBUG(surf_route, "Route from '%s' to '%s' (latency: %f):", cname(), dest->cname(), (latency == nullptr ? -1 : *latency)); for (auto link : *links) XBT_CDEBUG(surf_route, "Link %s", link->getName()); } } boost::unordered_map const& Host::mountedStorages() { if (mounts == nullptr) { mounts = new boost::unordered_map (); xbt_dict_t dict = this->mountedStoragesAsDict(); xbt_dict_cursor_t cursor; char *mountname; char *storagename; xbt_dict_foreach(dict, cursor, mountname, storagename) { mounts->insert({mountname, &Storage::byName(storagename)}); } xbt_dict_free(&dict); } return *mounts; } /** Get the properties assigned to a host */ xbt_dict_t Host::properties() { return simgrid::simix::kernelImmediate([&] { return this->pimpl_->getProperties(); }); } /** Retrieve the property value (or nullptr if not set) */ const char*Host::property(const char*key) { return this->pimpl_->getProperty(key); } void Host::setProperty(const char*key, const char *value){ simgrid::simix::kernelImmediate([&] { this->pimpl_->setProperty(key, value); }); } /** Get the processes attached to the host */ xbt_swag_t Host::processes() { return simgrid::simix::kernelImmediate([&]() { return this->extension()->process_list; }); } /** Get the peak power of a host */ double Host::getPstateSpeedCurrent() { return simgrid::simix::kernelImmediate([&] { return this->pimpl_cpu->getPstateSpeedCurrent(); }); } /** Get one power peak (in flops/s) of a host at a given pstate */ double Host::getPstateSpeed(int pstate_index) { return simgrid::simix::kernelImmediate([&] { return this->pimpl_cpu->getPstateSpeed(pstate_index); }); } /** @brief Get the speed of the cpu associated to a host */ double Host::speed() { return pimpl_cpu->getSpeed(1.0); } /** @brief Returns the number of core of the processor. */ int Host::coreCount() { return pimpl_cpu->coreCount(); } /** @brief Set the pstate at which the host should run */ void Host::setPstate(int pstate_index) { simgrid::simix::kernelImmediate(std::bind( &simgrid::surf::Cpu::setPState, pimpl_cpu, pstate_index )); } /** @brief Retrieve the pstate at which the host is currently running */ int Host::pstate() { return pimpl_cpu->getPState(); } /** * \ingroup simix_storage_management * \brief Returns the list of storages mounted on an host. * \return a dict containing all storages mounted on the host */ xbt_dict_t Host::mountedStoragesAsDict() { return simgrid::simix::kernelImmediate([&] { return this->pimpl_->getMountedStorageList(); }); } /** * \ingroup simix_storage_management * \brief Returns the list of storages attached to an host. * \return a dict containing all storages attached to the host */ xbt_dynar_t Host::attachedStorages() { return simgrid::simix::kernelImmediate([&] { return this->pimpl_->getAttachedStorageList(); }); } } // namespace simgrid } // namespace s4u SimGrid-3.14.159/src/s4u/s4u_actor.cpp000644 001750 001750 00000007436 13031031271 021012 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2014. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "xbt/log.h" #include "simgrid/s4u/Actor.hpp" #include "simgrid/s4u/comm.hpp" #include "simgrid/s4u/host.hpp" #include "simgrid/s4u/Mailbox.hpp" #include "src/kernel/context/Context.hpp" XBT_LOG_NEW_DEFAULT_CATEGORY(s4u_actor,"S4U actors"); namespace simgrid { namespace s4u { // ***** Actor creation ***** ActorPtr Actor::self() { smx_context_t self_context = SIMIX_context_self(); if (self_context == nullptr) return simgrid::s4u::ActorPtr(); return simgrid::s4u::ActorPtr(&self_context->process()->getIface()); } ActorPtr Actor::createActor(const char* name, s4u::Host *host, double killTime, std::function code) { // TODO, when autorestart is used, the std::function is copied so the new // instance will get a fresh (reinitialized) state. Is this what we want? smx_actor_t process = simcall_process_create( name, std::move(code), nullptr, host, killTime, nullptr, 0); return ActorPtr(&process->getIface()); } ActorPtr Actor::createActor(const char* name, s4u::Host *host, double killTime, const char* function, std::vector args) { simgrid::simix::ActorCodeFactory& factory = SIMIX_get_actor_code_factory(function); simgrid::simix::ActorCode code = factory(std::move(args)); smx_actor_t process = simcall_process_create( name, std::move(code), nullptr, host, killTime, nullptr, 0); return ActorPtr(&process->getIface()); } // ***** Actor methods ***** void Actor::join() { simcall_process_join(pimpl_, -1); } void Actor::setAutoRestart(bool autorestart) { simcall_process_auto_restart_set(pimpl_,autorestart); } s4u::Host *Actor::getHost() { return pimpl_->host; } simgrid::xbt::string Actor::getName() { return pimpl_->name; } int Actor::getPid(){ return pimpl_->pid; } int Actor::getPpid() { return pimpl_->ppid; } void Actor::setKillTime(double time) { simcall_process_set_kill_time(pimpl_,time); } double Actor::getKillTime() { return simcall_process_get_kill_time(pimpl_); } void Actor::kill(int pid) { msg_process_t process = SIMIX_process_from_PID(pid); if(process != nullptr) { simcall_process_kill(process); } else { std::ostringstream oss; oss << "kill: ("<< pid <<") - No such process" << std::endl; throw std::runtime_error(oss.str()); } } smx_actor_t Actor::getImpl() { return pimpl_; } void Actor::kill() { simcall_process_kill(pimpl_); } // ***** Static functions ***** ActorPtr Actor::forPid(int pid) { smx_actor_t process = SIMIX_process_from_PID(pid); if (process != nullptr) return ActorPtr(&process->getIface()); else return nullptr; } void Actor::killAll() { simcall_process_killall(1); } // ***** this_actor ***** namespace this_actor { void sleep_for(double duration) { if (duration > 0) simcall_process_sleep(duration); } XBT_PUBLIC(void) sleep_until(double timeout) { double now = SIMIX_get_clock(); if (timeout > now) simcall_process_sleep(timeout - now); } e_smx_state_t execute(double flops) { smx_activity_t s = simcall_execution_start(nullptr,flops,1.0/*priority*/,0./*bound*/); return simcall_execution_wait(s); } void* recv(MailboxPtr chan) { void *res = nullptr; Comm& c = Comm::recv_init(chan); c.setDstData(&res,sizeof(res)); c.wait(); return res; } void send(MailboxPtr chan, void *payload, size_t simulatedSize) { Comm& c = Comm::send_init(chan); c.setRemains(simulatedSize); c.setSrcData(payload); // c.start() is optional. c.wait(); } int getPid() { return SIMIX_process_self()->pid; } int getPpid() { return SIMIX_process_self()->ppid; } } } } SimGrid-3.14.159/src/s4u/s4u_netzone.cpp000644 001750 001750 00000004635 13031031271 021362 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2006-2016. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "xbt/log.h" #include "simgrid/s4u/NetZone.hpp" #include "simgrid/s4u/host.hpp" #include "simgrid/simix.hpp" #include "src/kernel/routing/NetCard.hpp" #include "src/surf/network_interface.hpp" // Link FIXME: move to proper header XBT_LOG_NEW_DEFAULT_CATEGORY(s4u_netzone, "S4U Networking Zones"); namespace simgrid { namespace s4u { simgrid::xbt::signal* link_list)> NetZone::onRouteCreation; NetZone::NetZone(NetZone* father, const char* name) : father_(father), name_(xbt_strdup(name)) { } void NetZone::seal() { sealed_ = true; } NetZone::~NetZone() { xbt_dict_cursor_t cursor = nullptr; char* key; NetZone_t elem; xbt_dict_foreach (children_, cursor, key, elem) { delete static_cast(elem); } xbt_dict_free(&children_); xbt_free(name_); } std::unordered_map* NetZone::properties() { return simgrid::simix::kernelImmediate([=] { return &properties_; }); } /** Retrieve the property value (or nullptr if not set) */ const char* NetZone::property(const char* key) { return properties_.at(key).c_str(); } void NetZone::setProperty(const char* key, const char* value) { simgrid::simix::kernelImmediate([=,&key,&value] { properties_[key] = value; }); } xbt_dict_t NetZone::children() { return children_; } char* NetZone::name() { return name_; } NetZone* NetZone::father() { return father_; } xbt_dynar_t NetZone::hosts() { xbt_dynar_t res = xbt_dynar_new(sizeof(sg_host_t), nullptr); for (auto card : vertices_) { s4u::Host* host = simgrid::s4u::Host::by_name_or_null(card->name()); if (host != nullptr) xbt_dynar_push(res, &host); } return res; } int NetZone::addComponent(kernel::routing::NetCard* elm) { vertices_.push_back(elm); return vertices_.size() - 1; // The rank of the newly created object } void NetZone::addRoute(sg_platf_route_cbarg_t /*route*/) { xbt_die("NetZone '%s' does not accept new routes (wrong class).", name_); } } }; // namespace simgrid::s4u SimGrid-3.14.159/src/s4u/s4u_engine.cpp000644 001750 001750 00000007376 13031031271 021152 0ustar00mquinsonmquinson000000 000000 /* s4u::Engine Simulation Engine and global functions. */ /* Copyright (c) 2006-2015. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "instr/instr_interface.h" #include "mc/mc.h" #include "simgrid/s4u/Mailbox.hpp" #include "simgrid/s4u/NetZone.hpp" #include "simgrid/s4u/engine.hpp" #include "simgrid/s4u/host.hpp" #include "simgrid/s4u/storage.hpp" #include "simgrid/simix.h" #include "src/kernel/EngineImpl.hpp" #include "src/kernel/routing/NetZoneImpl.hpp" #include "src/kernel/routing/NetCard.hpp" #include "src/surf/network_interface.hpp" #include "surf/surf.h" // routing_platf. FIXME:KILLME. SOON XBT_LOG_NEW_CATEGORY(s4u,"Log channels of the S4U (Simgrid for you) interface"); namespace simgrid { namespace s4u { Engine *Engine::instance_ = nullptr; /* That singleton is awful, but I don't see no other solution right now. */ Engine::Engine(int *argc, char **argv) { xbt_assert(s4u::Engine::instance_ == nullptr, "It is currently forbidden to create more than one instance of s4u::Engine"); s4u::Engine::instance_ = this; pimpl = new kernel::EngineImpl(); TRACE_global_init(argc, argv); SIMIX_global_init(argc, argv); } Engine::~Engine() { delete pimpl; } Engine *Engine::instance() { if (s4u::Engine::instance_ == nullptr) new Engine(0,nullptr); return s4u::Engine::instance_; } void Engine::shutdown() { delete s4u::Engine::instance_; s4u::Engine::instance_ = nullptr; delete s4u::Storage::storages_; } double Engine::getClock() { return SIMIX_get_clock(); } void Engine::loadPlatform(const char *platf) { SIMIX_create_environment(platf); } void Engine::registerFunction(const char*name, int (*code)(int,char**)) { SIMIX_function_register(name,code); } void Engine::registerDefault(int (*code)(int,char**)) { SIMIX_function_register_default(code); } void Engine::loadDeployment(const char *deploy) { SIMIX_launch_application(deploy); } void Engine::run() { if (MC_is_active()) { MC_run(); } else { SIMIX_run(); } } s4u::NetZone* Engine::netRoot() { return pimpl->netRoot_; } static s4u::NetZone* netzoneByNameRecursive(s4u::NetZone* current, const char* name) { if(!strcmp(current->name(), name)) return current; xbt_dict_cursor_t cursor = nullptr; char *key; NetZone_t elem; xbt_dict_foreach(current->children(), cursor, key, elem) { simgrid::s4u::NetZone* tmp = netzoneByNameRecursive(elem, name); if (tmp != nullptr ) return tmp; } return nullptr; } /** @brief Retrieve the NetZone of the given name (or nullptr if not found) */ NetZone* Engine::netzoneByNameOrNull(const char* name) { return netzoneByNameRecursive(netRoot(), name); } /** @brief Retrieve the netcard of the given name (or nullptr if not found) */ simgrid::kernel::routing::NetCard* Engine::netcardByNameOrNull(const char* name) { if (pimpl->netcards_.find(name) == pimpl->netcards_.end()) return nullptr; return pimpl->netcards_.at(name); } /** @brief Fill the provided vector with all existing netcards */ void Engine::netcardList(std::vector* list) { for (auto kv: pimpl->netcards_) list->push_back(kv.second); } /** @brief Register a new netcard to the system */ void Engine::netcardRegister(simgrid::kernel::routing::NetCard* card) { // simgrid::simix::kernelImmediate([&]{ FIXME: this segfaults in set_thread pimpl->netcards_[card->name()] = card; // }); } /** @brief Unregister a given netcard */ void Engine::netcardUnregister(simgrid::kernel::routing::NetCard* card) { simgrid::simix::kernelImmediate([&]{ pimpl->netcards_.erase(card->name()); delete card; }); } } } SimGrid-3.14.159/src/instr/000755 001750 001750 00000000000 13031031272 017016 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/src/instr/instr_paje_types.cpp000644 001750 001750 00000012700 13031031272 023104 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012, 2014-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "src/instr/instr_private.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY (instr_paje_types, instr, "Paje tracing event system (types)"); static type_t rootType = nullptr; /* the root type */ void PJ_type_alloc () { } void PJ_type_release () { rootType = nullptr; } type_t PJ_type_get_root () { return rootType; } static type_t newType (const char *typeNameBuff, const char *key, const char *color, e_entity_types kind, type_t father) { if (typeNameBuff == nullptr || key == nullptr){ THROWF(tracing_error, 0, "can't create a new type with name or key equal nullptr"); } type_t ret = xbt_new0(s_type_t, 1); ret->name = xbt_strdup (typeNameBuff); ret->father = father; ret->kind = kind; ret->children = xbt_dict_new_homogeneous(nullptr); ret->values = xbt_dict_new_homogeneous(nullptr); ret->color = xbt_strdup (color); char str_id[INSTR_DEFAULT_STR_SIZE]; snprintf (str_id, INSTR_DEFAULT_STR_SIZE, "%lld", instr_new_paje_id()); ret->id = xbt_strdup (str_id); if (father != nullptr){ xbt_dict_set (father->children, key, ret, nullptr); XBT_DEBUG("new type %s, child of %s", typeNameBuff, father->name); } return ret; } void PJ_type_free (type_t type) { val_t value; char *value_name; xbt_dict_cursor_t cursor = nullptr; xbt_dict_foreach(type->values, cursor, value_name, value) { PJ_value_free (value); } xbt_dict_free (&type->values); xbt_free (type->name); xbt_free (type->id); xbt_free (type->color); xbt_dict_free (&type->children); xbt_free (type); type = nullptr; } static void recursiveDestroyType (type_t type) { XBT_DEBUG("recursiveDestroyType %s", type->name); xbt_dict_cursor_t cursor = nullptr; type_t child; char *child_name; xbt_dict_foreach(type->children, cursor, child_name, child) { recursiveDestroyType (child); } PJ_type_free(type); } void PJ_type_free_all () { recursiveDestroyType (PJ_type_get_root()); rootType = nullptr; } type_t PJ_type_get (const char *name, type_t father) { type_t ret = PJ_type_get_or_null (name, father); if (ret == nullptr){ THROWF (tracing_error, 2, "type with name (%s) not found in father type (%s)", name, father->name); } return ret; } type_t PJ_type_get_or_null (const char *name, type_t father) { if (name == nullptr || father == nullptr){ THROWF (tracing_error, 0, "can't get type with a nullptr name or from a nullptr father"); } type_t ret = nullptr; type_t child; char *child_name; xbt_dict_cursor_t cursor = nullptr; xbt_dict_foreach(father->children, cursor, child_name, child) { if (strcmp (child->name, name) == 0){ if (ret != nullptr){ THROWF (tracing_error, 0, "there are two children types with the same name?"); }else{ ret = child; } } } return ret; } type_t PJ_type_container_new (const char *name, type_t father) { if (name == nullptr){ THROWF (tracing_error, 0, "can't create a container type with a nullptr name"); } type_t ret = nullptr; ret = newType (name, name, nullptr, TYPE_CONTAINER, father); if (father == nullptr){ rootType = ret; } if(father){ XBT_DEBUG("ContainerType %s(%s), child of %s(%s)", ret->name, ret->id, father->name, father->id); new_pajeDefineContainerType (ret); } return ret; } type_t PJ_type_event_new (const char *name, type_t father) { if (name == nullptr){ THROWF (tracing_error, 0, "can't create an event type with a nullptr name"); } type_t ret = newType (name, name, nullptr, TYPE_EVENT, father); XBT_DEBUG("EventType %s(%s), child of %s(%s)", ret->name, ret->id, father->name, father->id); new_pajeDefineEventType(ret); return ret; } type_t PJ_type_variable_new (const char *name, const char *color, type_t father) { if (name == nullptr){ THROWF (tracing_error, 0, "can't create a variable type with a nullptr name"); } type_t ret = nullptr; char white[INSTR_DEFAULT_STR_SIZE] = "1 1 1"; if (!color){ ret = newType (name, name, white, TYPE_VARIABLE, father); }else{ ret = newType (name, name, color, TYPE_VARIABLE, father); } XBT_DEBUG("VariableType %s(%s), child of %s(%s)", ret->name, ret->id, father->name, father->id); new_pajeDefineVariableType (ret); return ret; } type_t PJ_type_link_new (const char *name, type_t father, type_t source, type_t dest) { if (name == nullptr){ THROWF (tracing_error, 0, "can't create a link type with a nullptr name"); } type_t ret = nullptr; char key[INSTR_DEFAULT_STR_SIZE]; snprintf (key, INSTR_DEFAULT_STR_SIZE, "%s-%s-%s", name, source->id, dest->id); ret = newType (name, key, nullptr, TYPE_LINK, father); XBT_DEBUG("LinkType %s(%s), child of %s(%s) %s(%s)->%s(%s)", ret->name, ret->id, father->name, father->id, source->name, source->id, dest->name, dest->id); new_pajeDefineLinkType(ret, source, dest); return ret; } type_t PJ_type_state_new (const char *name, type_t father) { if (name == nullptr){ THROWF (tracing_error, 0, "can't create a state type with a nullptr name"); } type_t ret = nullptr; ret = newType (name, name, nullptr, TYPE_STATE, father); XBT_DEBUG("StateType %s(%s), child of %s(%s)", ret->name, ret->id, father->name, father->id); new_pajeDefineStateType(ret); return ret; } SimGrid-3.14.159/src/instr/instr_paje_containers.cpp000644 001750 001750 00000020322 13031031272 024104 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010, 2012-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/s4u/engine.hpp" #include "simgrid/s4u/host.hpp" #include #include #include #include #include #include "src/instr/instr_private.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY (instr_paje_containers, instr, "Paje tracing event system (containers)"); static container_t rootContainer = nullptr; /* the root container */ static xbt_dict_t allContainers = nullptr; /* all created containers indexed by name */ xbt_dict_t trivaNodeTypes = nullptr; /* all host types defined */ xbt_dict_t trivaEdgeTypes = nullptr; /* all link types defined */ long long int instr_new_paje_id () { static long long int type_id = 0; return type_id++; } void PJ_container_alloc () { allContainers = xbt_dict_new_homogeneous(nullptr); trivaNodeTypes = xbt_dict_new_homogeneous(xbt_free_f); trivaEdgeTypes = xbt_dict_new_homogeneous(xbt_free_f); } void PJ_container_release () { xbt_dict_free (&allContainers); xbt_dict_free (&trivaNodeTypes); xbt_dict_free (&trivaEdgeTypes); } void PJ_container_set_root (container_t root) { rootContainer = root; } container_t PJ_container_new (const char *name, e_container_types kind, container_t father) { if (name == nullptr){ THROWF (tracing_error, 0, "can't create a container with a nullptr name"); } static long long int container_id = 0; char id_str[INSTR_DEFAULT_STR_SIZE]; snprintf (id_str, INSTR_DEFAULT_STR_SIZE, "%lld", container_id); container_id++; container_t newContainer = xbt_new0(s_container_t, 1); newContainer->name = xbt_strdup (name); // name of the container newContainer->id = xbt_strdup (id_str); // id (or alias) of the container newContainer->father = father; sg_host_t sg_host = sg_host_by_name(name); //Search for network_element_t switch (kind){ case INSTR_HOST: newContainer->netcard = sg_host->pimpl_netcard; xbt_assert(newContainer->netcard, "Element '%s' not found",name); break; case INSTR_ROUTER: newContainer->netcard = simgrid::s4u::Engine::instance()->netcardByNameOrNull(name); xbt_assert(newContainer->netcard, "Element '%s' not found",name); break; case INSTR_AS: newContainer->netcard = simgrid::s4u::Engine::instance()->netcardByNameOrNull(name); xbt_assert(newContainer->netcard, "Element '%s' not found",name); break; default: newContainer->netcard = nullptr; break; } // level depends on level of father if (newContainer->father){ newContainer->level = newContainer->father->level+1; XBT_DEBUG("new container %s, child of %s", name, father->name); }else{ newContainer->level = 0; } // type definition (method depends on kind of this new container) newContainer->kind = kind; if (newContainer->kind == INSTR_AS){ //if this container is of an AS, its type name depends on its level char as_typename[INSTR_DEFAULT_STR_SIZE]; snprintf (as_typename, INSTR_DEFAULT_STR_SIZE, "L%d", newContainer->level); if (newContainer->father){ newContainer->type = PJ_type_get_or_null (as_typename, newContainer->father->type); if (newContainer->type == nullptr){ newContainer->type = PJ_type_container_new (as_typename, newContainer->father->type); } }else{ newContainer->type = PJ_type_container_new ("0", nullptr); } }else{ //otherwise, the name is its kind char typeNameBuff[INSTR_DEFAULT_STR_SIZE]; switch (newContainer->kind){ case INSTR_HOST: snprintf (typeNameBuff, INSTR_DEFAULT_STR_SIZE, "HOST"); break; case INSTR_LINK: snprintf (typeNameBuff, INSTR_DEFAULT_STR_SIZE, "LINK"); break; case INSTR_ROUTER: snprintf (typeNameBuff, INSTR_DEFAULT_STR_SIZE, "ROUTER"); break; case INSTR_SMPI: snprintf (typeNameBuff, INSTR_DEFAULT_STR_SIZE, "MPI"); break; case INSTR_MSG_PROCESS: snprintf (typeNameBuff, INSTR_DEFAULT_STR_SIZE, "MSG_PROCESS"); break; case INSTR_MSG_VM: snprintf (typeNameBuff, INSTR_DEFAULT_STR_SIZE, "MSG_VM"); break; case INSTR_MSG_TASK: snprintf (typeNameBuff, INSTR_DEFAULT_STR_SIZE, "MSG_TASK"); break; default: THROWF (tracing_error, 0, "new container kind is unknown."); break; } type_t type = PJ_type_get_or_null (typeNameBuff, newContainer->father->type); if (type == nullptr){ newContainer->type = PJ_type_container_new (typeNameBuff, newContainer->father->type); }else{ newContainer->type = type; } } newContainer->children = xbt_dict_new_homogeneous(nullptr); if (newContainer->father){ xbt_dict_set(newContainer->father->children, newContainer->name, newContainer, nullptr); new_pajeCreateContainer (newContainer); } //register all kinds by name if (xbt_dict_get_or_null(allContainers, newContainer->name) != nullptr){ THROWF(tracing_error, 1, "container %s already present in allContainers data structure", newContainer->name); } xbt_dict_set (allContainers, newContainer->name, newContainer, nullptr); XBT_DEBUG("Add container name '%s'",newContainer->name); //register NODE types for triva configuration if (newContainer->kind == INSTR_HOST || newContainer->kind == INSTR_LINK || newContainer->kind == INSTR_ROUTER) { xbt_dict_set (trivaNodeTypes, newContainer->type->name, xbt_strdup("1"), nullptr); } return newContainer; } container_t PJ_container_get (const char *name) { container_t ret = PJ_container_get_or_null (name); if (ret == nullptr){ THROWF(tracing_error, 1, "container with name %s not found", name); } return ret; } container_t PJ_container_get_or_null (const char *name) { return static_cast(name != nullptr ? xbt_dict_get_or_null(allContainers, name) : nullptr); } container_t PJ_container_get_root () { return rootContainer; } void PJ_container_remove_from_parent (container_t child) { if (child == nullptr){ THROWF (tracing_error, 0, "can't remove from parent with a nullptr child"); } container_t parent = child->father; if (parent){ XBT_DEBUG("removeChildContainer (%s) FromContainer (%s) ", child->name, parent->name); xbt_dict_remove (parent->children, child->name); } } void PJ_container_free (container_t container) { if (container == nullptr){ THROWF (tracing_error, 0, "trying to free a nullptr container"); } XBT_DEBUG("destroy container %s", container->name); //obligation to dump previous events because they might //reference the container that is about to be destroyed TRACE_last_timestamp_to_dump = surf_get_clock(); TRACE_paje_dump_buffer(1); //trace my destruction if (!TRACE_disable_destroy() && container != PJ_container_get_root()){ //do not trace the container destruction if user requests //or if the container is root new_pajeDestroyContainer(container); } //remove it from allContainers data structure xbt_dict_remove (allContainers, container->name); //free xbt_free (container->name); xbt_free (container->id); xbt_dict_free (&container->children); xbt_free (container); container = nullptr; } static void recursiveDestroyContainer (container_t container) { if (container == nullptr){ THROWF (tracing_error, 0, "trying to recursively destroy a nullptr container"); } XBT_DEBUG("recursiveDestroyContainer %s", container->name); xbt_dict_cursor_t cursor = nullptr; container_t child; char *child_name; xbt_dict_foreach(container->children, cursor, child_name, child) { recursiveDestroyContainer (child); } PJ_container_free (container); } void PJ_container_free_all () { container_t root = PJ_container_get_root(); if (root == nullptr){ THROWF (tracing_error, 0, "trying to free all containers, but root is nullptr"); } recursiveDestroyContainer (root); rootContainer = nullptr; //checks if (!xbt_dict_is_empty(allContainers)){ THROWF(tracing_error, 0, "some containers still present even after destroying all of them"); } } SimGrid-3.14.159/src/instr/instr_TI_trace.cpp000644 001750 001750 00000020620 13031031272 022433 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010-2015. The SimGrid Team. All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "src/instr/instr_private.h" #include "xbt/virtu.h" /* sg_cmdline */ #include "xbt/xbt_os_time.h" #include "simgrid/sg_config.h" #include #include #include #ifdef WIN32 #include // _mkdir #endif XBT_LOG_NEW_DEFAULT_SUBCATEGORY(instr_TI_trace, instr_trace, "tracing event system"); extern FILE *tracing_file; double prefix=0.0; xbt_dict_t tracing_files = nullptr; extern s_instr_trace_writer_t active_writer; void TRACE_TI_init() { active_writer.print_PushState = &print_TIPushState; active_writer.print_CreateContainer = &print_TICreateContainer; active_writer.print_DestroyContainer = &print_TIDestroyContainer; } void TRACE_TI_start() { char *filename = TRACE_get_filename(); tracing_file = fopen(filename, "w"); if (tracing_file == nullptr) { THROWF(system_error, 1, "Tracefile %s could not be opened for writing.", filename); } XBT_DEBUG("Filename %s is open for writing", filename); /* output one line comment */ dump_comment(TRACE_get_comment()); /* output comment file */ dump_comment_file(TRACE_get_comment_file()); } void TRACE_TI_end() { xbt_dict_free(&tracing_files); fclose(tracing_file); char *filename = TRACE_get_filename(); XBT_DEBUG("Filename %s is closed", filename); } void print_TICreateContainer(paje_event_t event) { //if we are in the mode with only one file static FILE *temp = nullptr; if (tracing_files == nullptr) { tracing_files = xbt_dict_new_homogeneous(nullptr); //generate unique run id with time prefix = xbt_os_time(); } if (!xbt_cfg_get_boolean("tracing/smpi/format/ti-one-file") || temp == nullptr) { char *folder_name = bprintf("%s_files", TRACE_get_filename()); char *filename = bprintf("%s/%f_%s.txt", folder_name, prefix, ((createContainer_t) event->data)->container->name); #ifdef WIN32 _mkdir(folder_name); #else mkdir(folder_name, S_IRWXU | S_IRWXG | S_IRWXO); #endif temp = fopen(filename, "w"); xbt_assert(temp, "Tracefile %s could not be opened for writing: %s", filename, strerror(errno)); fprintf(tracing_file, "%s\n", filename); xbt_free(folder_name); xbt_free(filename); } xbt_dict_set(tracing_files, ((createContainer_t) event->data)->container->name, (void *) temp, nullptr); } void print_TIDestroyContainer(paje_event_t event) { if (!xbt_cfg_get_boolean("tracing/smpi/format/ti-one-file")|| xbt_dict_length(tracing_files) == 1) { FILE* f = (FILE*)xbt_dict_get_or_null(tracing_files, ((destroyContainer_t) event->data)->container->name); fclose(f); } xbt_dict_remove(tracing_files, ((destroyContainer_t) event->data)->container->name); } void print_TIPushState(paje_event_t event) { int i; //char* function=nullptr; if (((pushState_t) event->data)->extra == nullptr) return; instr_extra_data extra = (instr_extra_data) (((pushState_t) event->data)->extra); char *process_id = nullptr; //FIXME: dirty extract "rank-" from the name, as we want the bare process id here if (strstr(((pushState_t) event->data)->container->name, "rank-") == nullptr) process_id = xbt_strdup(((pushState_t) event->data)->container->name); else process_id = xbt_strdup(((pushState_t) event->data)->container->name + 5); FILE* trace_file = (FILE* )xbt_dict_get(tracing_files, ((pushState_t) event->data)->container->name); switch (extra->type) { case TRACING_INIT: fprintf(trace_file, "%s init\n", process_id); break; case TRACING_FINALIZE: fprintf(trace_file, "%s finalize\n", process_id); break; case TRACING_SEND: fprintf(trace_file, "%s send %d %d %s\n", process_id, extra->dst, extra->send_size, extra->datatype1); break; case TRACING_ISEND: fprintf(trace_file, "%s isend %d %d %s\n", process_id, extra->dst, extra->send_size, extra->datatype1); break; case TRACING_RECV: fprintf(trace_file, "%s recv %d %d %s\n", process_id, extra->src, extra->send_size, extra->datatype1); break; case TRACING_IRECV: fprintf(trace_file, "%s irecv %d %d %s\n", process_id, extra->src, extra->send_size, extra->datatype1); break; case TRACING_TEST: fprintf(trace_file, "%s test\n", process_id); break; case TRACING_WAIT: fprintf(trace_file, "%s wait\n", process_id); break; case TRACING_WAITALL: fprintf(trace_file, "%s waitall\n", process_id); break; case TRACING_BARRIER: fprintf(trace_file, "%s barrier\n", process_id); break; case TRACING_BCAST: // rank bcast size (root) (datatype) fprintf(trace_file, "%s bcast %d ", process_id, extra->send_size); if (extra->root != 0 || (extra->datatype1 && strcmp(extra->datatype1, ""))) fprintf(trace_file, "%d %s", extra->root, extra->datatype1); fprintf(trace_file, "\n"); break; case TRACING_REDUCE: // rank reduce comm_size comp_size (root) (datatype) fprintf(trace_file, "%s reduce %d %f ", process_id, extra->send_size, extra->comp_size); if (extra->root != 0 || (extra->datatype1 && strcmp(extra->datatype1, ""))) fprintf(trace_file, "%d %s", extra->root, extra->datatype1); fprintf(trace_file, "\n"); break; case TRACING_ALLREDUCE: // rank allreduce comm_size comp_size (datatype) fprintf(trace_file, "%s allreduce %d %f %s\n", process_id, extra->send_size, extra->comp_size, extra->datatype1); break; case TRACING_ALLTOALL: // rank alltoall send_size recv_size (sendtype) (recvtype) fprintf(trace_file, "%s alltoall %d %d %s %s\n", process_id, extra->send_size, extra->recv_size, extra->datatype1, extra->datatype2); break; case TRACING_ALLTOALLV: // rank alltoallv send_size [sendcounts] recv_size [recvcounts] (sendtype) (recvtype) fprintf(trace_file, "%s alltoallv %d ", process_id, extra->send_size); for (i = 0; i < extra->num_processes; i++) fprintf(trace_file, "%d ", extra->sendcounts[i]); fprintf(trace_file, "%d ", extra->recv_size); for (i = 0; i < extra->num_processes; i++) fprintf(trace_file, "%d ", extra->recvcounts[i]); fprintf(trace_file, "%s %s \n", extra->datatype1, extra->datatype2); break; case TRACING_GATHER: // rank gather send_size recv_size root (sendtype) (recvtype) fprintf(trace_file, "%s gather %d %d %d %s %s\n", process_id, extra->send_size, extra->recv_size, extra->root, extra->datatype1, extra->datatype2); break; case TRACING_ALLGATHERV: // rank allgatherv send_size [recvcounts] (sendtype) (recvtype) fprintf(trace_file, "%s allgatherv %d ", process_id, extra->send_size); for (i = 0; i < extra->num_processes; i++) fprintf(trace_file, "%d ", extra->recvcounts[i]); fprintf(trace_file, "%s %s \n", extra->datatype1, extra->datatype2); break; case TRACING_REDUCE_SCATTER: // rank reducescatter [recvcounts] comp_size (sendtype) fprintf(trace_file, "%s reducescatter ", process_id); for (i = 0; i < extra->num_processes; i++) fprintf(trace_file, "%d ", extra->recvcounts[i]); fprintf(trace_file, "%f %s\n", extra->comp_size, extra->datatype1); break; case TRACING_COMPUTING: fprintf(trace_file, "%s compute %f\n", process_id, extra->comp_size); break; case TRACING_SLEEPING: fprintf(trace_file, "%s sleep %f\n", process_id, extra->sleep_duration); break; case TRACING_GATHERV: // rank gatherv send_size [recvcounts] root (sendtype) (recvtype) fprintf(trace_file, "%s gatherv %d ", process_id, extra->send_size); for (i = 0; i < extra->num_processes; i++) fprintf(trace_file, "%d ", extra->recvcounts[i]); fprintf(trace_file, "%d %s %s\n", extra->root, extra->datatype1, extra->datatype2); break; case TRACING_WAITANY: case TRACING_SENDRECV: case TRACING_SCATTER: case TRACING_SCATTERV: case TRACING_ALLGATHER: case TRACING_SCAN: case TRACING_EXSCAN: case TRACING_COMM_SIZE: case TRACING_COMM_SPLIT: case TRACING_COMM_DUP: case TRACING_SSEND: case TRACING_ISSEND: default: XBT_WARN ("Call from %s impossible to translate into replay command : Not implemented (yet)", ((pushState_t) event->data)->value->name); break; } if (extra->recvcounts != nullptr) xbt_free(extra->recvcounts); if (extra->sendcounts != nullptr) xbt_free(extra->sendcounts); xbt_free(process_id); xbt_free(extra); } SimGrid-3.14.159/src/instr/instr_trace.cpp000644 001750 001750 00000041114 13031031272 022040 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "src/instr/instr_private.h" #include "src/instr/instr_smpi.h" #include "src/smpi/private.hpp" #include "xbt/virtu.h" /* sg_cmdline */ XBT_LOG_NEW_DEFAULT_SUBCATEGORY(instr_trace, instr, "tracing event system"); FILE *tracing_file = nullptr; void print_NULL(paje_event_t event){} /* The active set of functions for the selected trace format * By default, they all do nothing, hence the print_NULL to avoid segfaults */ s_instr_trace_writer_t active_writer = {&print_NULL, &print_NULL, &print_NULL, &print_NULL, &print_NULL, &print_NULL, &print_NULL, &print_NULL, &print_NULL, &print_NULL, &print_NULL, &print_NULL, &print_NULL, &print_NULL, &print_NULL, &print_NULL, &print_NULL, &print_NULL}; std::vector buffer; void dump_comment (const char *comment) { if (!strlen(comment)) return; fprintf (tracing_file, "# %s\n", comment); } void dump_comment_file (const char *filename) { if (!strlen(filename)) return; FILE *file = fopen (filename, "r"); if (!file){ THROWF (system_error, 1, "Comment file %s could not be opened for reading.", filename); } while (!feof(file)){ char c; c = fgetc(file); if (feof(file)) break; fprintf (tracing_file, "# "); while (c != '\n'){ fprintf (tracing_file, "%c", c); c = fgetc(file); if (feof(file)) break; } fprintf (tracing_file, "\n"); } fclose(file); } double TRACE_last_timestamp_to_dump = 0; //dumps the trace file until the timestamp TRACE_last_timestamp_to_dump void TRACE_paje_dump_buffer (int force) { if (!TRACE_is_enabled()) return; XBT_DEBUG("%s: dump until %f. starts", __FUNCTION__, TRACE_last_timestamp_to_dump); if (force){ for (auto event :buffer){ event->print (event); event->free (event); } buffer.clear(); }else{ std::vector::iterator i = buffer.begin(); for (auto event :buffer){ double head_timestamp = event->timestamp; if (head_timestamp > TRACE_last_timestamp_to_dump) break; event->print (event); event->free (event); ++i; } buffer.erase(buffer.begin(), i); } XBT_DEBUG("%s: ends", __FUNCTION__); } /* internal do the instrumentation module */ static void insert_into_buffer (paje_event_t tbi) { if (TRACE_buffer() == 0){ tbi->print (tbi); tbi->free (tbi); return; } XBT_DEBUG("%s: insert event_type=%d, timestamp=%f, buffersize=%zu)", __FUNCTION__, (int)tbi->event_type, tbi->timestamp, buffer.size()); std::vector::reverse_iterator i; for (i = buffer.rbegin(); i != buffer.rend(); ++i) { paje_event_t e1 = *i; if (e1->timestamp <= tbi->timestamp) break; } buffer.insert(i.base(), tbi); if (i == buffer.rend()) XBT_DEBUG("%s: inserted at beginning", __FUNCTION__); else XBT_DEBUG("%s: inserted at%s %zd", __FUNCTION__, (i == buffer.rbegin()) ? " end" :"pos =", std::distance(buffer.rend(),i)); } static void free_paje_event (paje_event_t event) { XBT_DEBUG("%s: event_type=%d, timestamp=%f", __FUNCTION__, (int)event->event_type, event->timestamp); switch (event->event_type){ case PAJE_StartLink: xbt_free (((startLink_t)(event->data))->value); xbt_free (((startLink_t)(event->data))->key); break; case PAJE_EndLink: xbt_free (((endLink_t)(event->data))->value); xbt_free (((endLink_t)(event->data))->key); break; default: break; } xbt_free (event->data); xbt_free (event); } void new_pajeDefineContainerType(type_t type) { paje_event_t event = xbt_new0(s_paje_event_t, 1); event->event_type = PAJE_DefineContainerType; event->timestamp = 0; event->print = active_writer.print_DefineContainerType; event->free = &free_paje_event; event->data = xbt_new0(s_defineContainerType_t, 1); ((defineContainerType_t)(event->data))->type = type; XBT_DEBUG("%s: event_type=%d", __FUNCTION__, (int)event->event_type); //print it event->print (event); event->free (event); } void new_pajeDefineVariableType(type_t type) { paje_event_t event = xbt_new0(s_paje_event_t, 1); event->event_type = PAJE_DefineVariableType; event->timestamp = 0; event->print = active_writer.print_DefineVariableType; event->free = &free_paje_event; event->data = xbt_new0(s_defineVariableType_t, 1); ((defineVariableType_t)(event->data))->type = type; XBT_DEBUG("%s: event_type=%d", __FUNCTION__, (int)event->event_type); //print it event->print (event); event->free (event); } void new_pajeDefineStateType(type_t type) { paje_event_t event = xbt_new0(s_paje_event_t, 1); event->event_type = PAJE_DefineStateType; event->timestamp = 0; event->print = active_writer.print_DefineStateType; event->free = &free_paje_event; event->data = xbt_new0(s_defineStateType_t, 1); ((defineStateType_t)(event->data))->type = type; XBT_DEBUG("%s: event_type=%d", __FUNCTION__, (int)event->event_type); //print it event->print (event); event->free (event); } void new_pajeDefineEventType(type_t type) { paje_event_t event = xbt_new0(s_paje_event_t, 1); event->event_type = PAJE_DefineEventType; event->timestamp = 0; event->print = active_writer.print_DefineEventType; event->free = &free_paje_event; event->data = xbt_new0(s_defineEventType_t, 1); ((defineEventType_t)(event->data))->type = type; XBT_DEBUG("%s: event_type=%d", __FUNCTION__, (int)event->event_type); //print it event->print (event); event->free (event); } void new_pajeDefineLinkType(type_t type, type_t source, type_t dest) { paje_event_t event = xbt_new0(s_paje_event_t, 1); event->event_type = PAJE_DefineLinkType; event->timestamp = 0; event->print = active_writer.print_DefineLinkType; event->free = &free_paje_event; event->data = xbt_new0(s_defineLinkType_t, 1); ((defineLinkType_t)(event->data))->type = type; ((defineLinkType_t)(event->data))->source = source; ((defineLinkType_t)(event->data))->dest = dest; XBT_DEBUG("%s: event_type=%d", __FUNCTION__, (int)event->event_type); //print it event->print (event); event->free (event); } void new_pajeDefineEntityValue (val_t value) { paje_event_t event = xbt_new0(s_paje_event_t, 1); event->event_type = PAJE_DefineEntityValue; event->timestamp = 0; event->print = active_writer.print_DefineEntityValue; event->free = &free_paje_event; event->data = xbt_new0(s_defineEntityValue_t, 1); ((defineEntityValue_t)(event->data))->value = value; XBT_DEBUG("%s: event_type=%d", __FUNCTION__, (int)event->event_type); //print it event->print (event); event->free (event); } void new_pajeCreateContainer (container_t container) { paje_event_t event = xbt_new0(s_paje_event_t, 1); event->event_type = PAJE_CreateContainer; event->timestamp = SIMIX_get_clock(); event->print = active_writer.print_CreateContainer; event->free = &free_paje_event; event->data = xbt_new0(s_createContainer_t, 1); ((createContainer_t)(event->data))->container = container; XBT_DEBUG("%s: event_type=%d, timestamp=%f", __FUNCTION__, (int)event->event_type, event->timestamp); //print it event->print (event); event->free (event); } void new_pajeDestroyContainer (container_t container) { paje_event_t event = xbt_new0(s_paje_event_t, 1); event->event_type = PAJE_DestroyContainer; event->timestamp = SIMIX_get_clock(); event->print = active_writer.print_DestroyContainer; event->free = &free_paje_event; event->data = xbt_new0(s_destroyContainer_t, 1); ((destroyContainer_t)(event->data))->container = container; XBT_DEBUG("%s: event_type=%d, timestamp=%f", __FUNCTION__, (int)event->event_type, event->timestamp); //print it event->print (event); event->free (event); } void new_pajeSetVariable (double timestamp, container_t container, type_t type, double value) { paje_event_t event = xbt_new0(s_paje_event_t, 1); event->event_type = PAJE_SetVariable; event->timestamp = timestamp; event->print = active_writer.print_SetVariable; event->free = &free_paje_event; event->data = xbt_new0(s_setVariable_t, 1); ((setVariable_t)(event->data))->type = type; ((setVariable_t)(event->data))->container = container; ((setVariable_t)(event->data))->value = value; XBT_DEBUG("%s: event_type=%d, timestamp=%f", __FUNCTION__, (int)event->event_type, event->timestamp); insert_into_buffer (event); } void new_pajeAddVariable (double timestamp, container_t container, type_t type, double value) { paje_event_t event = xbt_new0(s_paje_event_t, 1); event->event_type = PAJE_AddVariable; event->timestamp = timestamp; event->print = active_writer.print_AddVariable; event->free = &free_paje_event; event->data = xbt_new0(s_addVariable_t, 1); ((addVariable_t)(event->data))->type = type; ((addVariable_t)(event->data))->container = container; ((addVariable_t)(event->data))->value = value; XBT_DEBUG("%s: event_type=%d, timestamp=%f", __FUNCTION__, (int)event->event_type, event->timestamp); insert_into_buffer (event); } void new_pajeSubVariable (double timestamp, container_t container, type_t type, double value) { paje_event_t event = xbt_new0(s_paje_event_t, 1); event->event_type = PAJE_SubVariable; event->timestamp = timestamp; event->print = active_writer.print_SubVariable; event->free = &free_paje_event; event->data = xbt_new0(s_subVariable_t, 1); ((subVariable_t)(event->data))->type = type; ((subVariable_t)(event->data))->container = container; ((subVariable_t)(event->data))->value = value; XBT_DEBUG("%s: event_type=%d, timestamp=%f", __FUNCTION__, (int)event->event_type, event->timestamp); insert_into_buffer (event); } void new_pajeSetState (double timestamp, container_t container, type_t type, val_t value) { paje_event_t event = xbt_new0(s_paje_event_t, 1); event->event_type = PAJE_SetState; event->timestamp = timestamp; event->print = active_writer.print_SetState; event->free = &free_paje_event; event->data = xbt_new0(s_setState_t, 1); ((setState_t)(event->data))->type = type; ((setState_t)(event->data))->container = container; ((setState_t)(event->data))->value = value; #if HAVE_SMPI if (xbt_cfg_get_boolean("smpi/trace-call-location")) { smpi_trace_call_location_t* loc = smpi_trace_get_call_location(); ((setState_t)(event->data))->filename = loc->filename; ((setState_t)(event->data))->linenumber = loc->linenumber; } #endif XBT_DEBUG("%s: event_type=%d, timestamp=%f", __FUNCTION__, (int)event->event_type, event->timestamp); insert_into_buffer (event); } void new_pajePushStateWithExtra (double timestamp, container_t container, type_t type, val_t value, void* extra) { paje_event_t event = xbt_new0(s_paje_event_t, 1); event->event_type = PAJE_PushState; event->timestamp = timestamp; event->print = active_writer.print_PushState; event->free = &free_paje_event; event->data = xbt_new0(s_pushState_t, 1); ((pushState_t)(event->data))->type = type; ((pushState_t)(event->data))->container = container; ((pushState_t)(event->data))->value = value; ((pushState_t)(event->data))->extra = extra; #if HAVE_SMPI if (xbt_cfg_get_boolean("smpi/trace-call-location")) { smpi_trace_call_location_t* loc = smpi_trace_get_call_location(); ((pushState_t)(event->data))->filename = loc->filename; ((pushState_t)(event->data))->linenumber = loc->linenumber; } #endif XBT_DEBUG("%s: event_type=%d, timestamp=%f", __FUNCTION__, (int)event->event_type, event->timestamp); insert_into_buffer (event); } void new_pajePushState (double timestamp, container_t container, type_t type, val_t value) { new_pajePushStateWithExtra(timestamp, container, type, value, nullptr); } void new_pajePopState (double timestamp, container_t container, type_t type) { paje_event_t event = xbt_new0(s_paje_event_t, 1); event->event_type = PAJE_PopState; event->timestamp = timestamp; event->print = active_writer.print_PopState; event->free = &free_paje_event; event->data = xbt_new0(s_popState_t, 1); ((popState_t)(event->data))->type = type; ((popState_t)(event->data))->container = container; XBT_DEBUG("%s: event_type=%d, timestamp=%f", __FUNCTION__, (int)event->event_type, event->timestamp); insert_into_buffer (event); } void new_pajeResetState (double timestamp, container_t container, type_t type) { paje_event_t event = xbt_new0(s_paje_event_t, 1); event->event_type = PAJE_ResetState; event->timestamp = timestamp; event->print = active_writer.print_ResetState; event->free = &free_paje_event; event->data = xbt_new0(s_resetState_t, 1); ((resetState_t)(event->data))->type = type; ((resetState_t)(event->data))->container = container; XBT_DEBUG("%s: event_type=%d, timestamp=%f", __FUNCTION__, (int)event->event_type, event->timestamp); insert_into_buffer (event); } void new_pajeStartLink (double timestamp, container_t container, type_t type, container_t sourceContainer, const char *value, const char *key) { paje_event_t event = xbt_new0(s_paje_event_t, 1); event->event_type = PAJE_StartLink; event->timestamp = timestamp; event->print = active_writer.print_StartLink; event->free = &free_paje_event; event->data = xbt_new0(s_startLink_t, 1); ((startLink_t)(event->data))->type = type; ((startLink_t)(event->data))->container = container; ((startLink_t)(event->data))->sourceContainer = sourceContainer; ((startLink_t)(event->data))->value = xbt_strdup(value); ((startLink_t)(event->data))->key = xbt_strdup(key); ((startLink_t)(event->data))->size = -1; XBT_DEBUG("%s: event_type=%d, timestamp=%f", __FUNCTION__, (int)event->event_type, event->timestamp); insert_into_buffer (event); } void new_pajeStartLinkWithSize (double timestamp, container_t container, type_t type, container_t sourceContainer, const char *value, const char *key, int size) { paje_event_t event = xbt_new0(s_paje_event_t, 1); event->event_type = PAJE_StartLink; event->timestamp = timestamp; event->print = active_writer.print_StartLink; event->free = &free_paje_event; event->data = xbt_new0(s_startLink_t, 1); ((startLink_t)(event->data))->type = type; ((startLink_t)(event->data))->container = container; ((startLink_t)(event->data))->sourceContainer = sourceContainer; ((startLink_t)(event->data))->value = xbt_strdup(value); ((startLink_t)(event->data))->key = xbt_strdup(key); ((startLink_t)(event->data))->size = size; XBT_DEBUG("%s: event_type=%d, timestamp=%f", __FUNCTION__, (int)event->event_type, event->timestamp); insert_into_buffer (event); } void new_pajeEndLink (double timestamp, container_t container, type_t type, container_t destContainer, const char *value, const char *key) { paje_event_t event = xbt_new0(s_paje_event_t, 1); event->event_type = PAJE_EndLink; event->timestamp = timestamp; event->print = active_writer.print_EndLink; event->free = &free_paje_event; event->data = xbt_new0(s_endLink_t, 1); ((endLink_t)(event->data))->type = type; ((endLink_t)(event->data))->container = container; ((endLink_t)(event->data))->destContainer = destContainer; ((endLink_t)(event->data))->value = xbt_strdup(value); ((endLink_t)(event->data))->key = xbt_strdup(key); XBT_DEBUG("%s: event_type=%d, timestamp=%f", __FUNCTION__, (int)event->event_type, event->timestamp); insert_into_buffer (event); } void new_pajeNewEvent (double timestamp, container_t container, type_t type, val_t value) { paje_event_t event = xbt_new0(s_paje_event_t, 1); event->event_type = PAJE_NewEvent; event->timestamp = timestamp; event->print = active_writer.print_NewEvent; event->free = &free_paje_event; event->data = xbt_new0(s_newEvent_t, 1); ((newEvent_t)(event->data))->type = type; ((newEvent_t)(event->data))->container = container; ((newEvent_t)(event->data))->value = value; XBT_DEBUG("%s: event_type=%d, timestamp=%f", __FUNCTION__, (int)event->event_type, event->timestamp); insert_into_buffer (event); } SimGrid-3.14.159/src/instr/instr_config.cpp000644 001750 001750 00000063156 13031031272 022221 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "src/instr/instr_private.h" #include "surf/surf.h" #include XBT_LOG_NEW_CATEGORY(instr, "Logging the behavior of the tracing system (used for Visualization/Analysis of simulations)"); XBT_LOG_NEW_DEFAULT_SUBCATEGORY (instr_config, instr, "Configuration"); #define OPT_TRACING_BASIC "tracing/basic" #define OPT_TRACING_BUFFER "tracing/buffer" #define OPT_TRACING_CATEGORIZED "tracing/categorized" #define OPT_TRACING_COMMENT_FILE "tracing/comment-file" #define OPT_TRACING_COMMENT "tracing/comment" #define OPT_TRACING_DISABLE_DESTROY "tracing/disable-destroy" #define OPT_TRACING_DISABLE_LINK "tracing/disable-link" #define OPT_TRACING_DISABLE_POWER "tracing/disable-power" #define OPT_TRACING_DISPLAY_SIZES "tracing/smpi/display-sizes" #define OPT_TRACING_FILENAME "tracing/filename" #define OPT_TRACING_FORMAT_TI_ONEFILE "tracing/smpi/format/ti-one-file" #define OPT_TRACING_FORMAT "tracing/smpi/format" #define OPT_TRACING_MSG_PROCESS "tracing/msg/process" #define OPT_TRACING_MSG_VM "tracing/msg/vm" #define OPT_TRACING_ONELINK_ONLY "tracing/onelink-only" #define OPT_TRACING_PLATFORM "tracing/platform" #define OPT_TRACING_PRECISION "tracing/precision" #define OPT_TRACING_SMPI_COMPUTING "tracing/smpi/computing" #define OPT_TRACING_SMPI_GROUP "tracing/smpi/group" #define OPT_TRACING_SMPI_INTERNALS "tracing/smpi/internals" #define OPT_TRACING_SMPI_SLEEPING "tracing/smpi/sleeping" #define OPT_TRACING_SMPI "tracing/smpi" #define OPT_TRACING_TOPOLOGY "tracing/platform/topology" #define OPT_TRACING "tracing" #define OPT_TRACING_UNCATEGORIZED "tracing/uncategorized" #define OPT_VIVA_CAT_CONF "viva/categorized" #define OPT_VIVA_UNCAT_CONF "viva/uncategorized" static bool trace_enabled = false; static bool trace_platform; static bool trace_platform_topology; static bool trace_smpi_enabled; static bool trace_smpi_grouped; static bool trace_smpi_computing; static bool trace_smpi_sleeping; static bool trace_view_internals; static bool trace_categorized; static bool trace_uncategorized; static bool trace_msg_process_enabled; static bool trace_msg_vm_enabled; static bool trace_buffer; static bool trace_onelink_only; static bool trace_disable_destroy; static bool trace_basic; static bool trace_display_sizes = false; static bool trace_disable_link; static bool trace_disable_power; static int trace_precision; static bool trace_configured = false; static bool trace_active = false; static void TRACE_getopts() { trace_enabled = xbt_cfg_get_boolean(OPT_TRACING); trace_platform = xbt_cfg_get_boolean(OPT_TRACING_PLATFORM); trace_platform_topology = xbt_cfg_get_boolean(OPT_TRACING_TOPOLOGY); trace_smpi_enabled = xbt_cfg_get_boolean(OPT_TRACING_SMPI); trace_smpi_grouped = xbt_cfg_get_boolean(OPT_TRACING_SMPI_GROUP); trace_smpi_computing = xbt_cfg_get_boolean(OPT_TRACING_SMPI_COMPUTING); trace_smpi_sleeping = xbt_cfg_get_boolean(OPT_TRACING_SMPI_SLEEPING); trace_view_internals = xbt_cfg_get_boolean(OPT_TRACING_SMPI_INTERNALS); trace_categorized = xbt_cfg_get_boolean(OPT_TRACING_CATEGORIZED); trace_uncategorized = xbt_cfg_get_boolean(OPT_TRACING_UNCATEGORIZED); trace_msg_process_enabled = xbt_cfg_get_boolean(OPT_TRACING_MSG_PROCESS); trace_msg_vm_enabled = xbt_cfg_get_boolean(OPT_TRACING_MSG_VM); trace_buffer = xbt_cfg_get_boolean(OPT_TRACING_BUFFER); trace_onelink_only = xbt_cfg_get_boolean(OPT_TRACING_ONELINK_ONLY); trace_disable_destroy = xbt_cfg_get_boolean(OPT_TRACING_DISABLE_DESTROY); trace_basic = xbt_cfg_get_boolean(OPT_TRACING_BASIC); trace_display_sizes = xbt_cfg_get_boolean(OPT_TRACING_DISPLAY_SIZES); trace_disable_link = xbt_cfg_get_boolean(OPT_TRACING_DISABLE_LINK); trace_disable_power = xbt_cfg_get_boolean(OPT_TRACING_DISABLE_POWER); trace_precision = xbt_cfg_get_int(OPT_TRACING_PRECISION); } static std::vector> TRACE_start_functions; void TRACE_add_start_function(void (*func) ()) { TRACE_start_functions.push_back(func); } int TRACE_start() { if (TRACE_is_configured()) TRACE_getopts(); // tracing system must be: // - enabled (with --cfg=tracing:yes) // - already configured (TRACE_global_init already called) if (TRACE_is_enabled()) { XBT_DEBUG("Tracing starts"); /* init the tracing module to generate the right output */ /* open the trace file(s) */ const char* format = xbt_cfg_get_string(OPT_TRACING_FORMAT); XBT_DEBUG("Tracing format %s\n", format); if(!strcmp(format, "Paje")){ TRACE_paje_init(); TRACE_paje_start(); }else if (!strcmp(format, "TI")){ TRACE_TI_init(); TRACE_TI_start(); }else{ xbt_die("Unknown trace format :%s ", format); } /* activate trace */ if (trace_active == 1) { THROWF(tracing_error, 0, "Tracing is already active"); } trace_active = 1; XBT_DEBUG("Tracing is on"); /* other trace initialization */ created_categories = xbt_dict_new_homogeneous(xbt_free_f); declared_marks = xbt_dict_new_homogeneous(xbt_free_f); user_host_variables = xbt_dict_new_homogeneous(xbt_free_f); user_vm_variables = xbt_dict_new_homogeneous(xbt_free_f); user_link_variables = xbt_dict_new_homogeneous(xbt_free_f); for (auto func: TRACE_start_functions) func(); } TRACE_start_functions.clear(); return 0; } static std::vector> TRACE_end_functions; void TRACE_add_end_function(void (*func) (void)) { TRACE_end_functions.push_back(func); } int TRACE_end() { int retval; if (!trace_active) { retval = 1; } else { retval = 0; TRACE_generate_viva_uncat_conf(); TRACE_generate_viva_cat_conf(); /* dump trace buffer */ TRACE_last_timestamp_to_dump = surf_get_clock(); TRACE_paje_dump_buffer(1); /* destroy all data structures of tracing (and free) */ PJ_container_free_all(); PJ_type_free_all(); PJ_container_release(); PJ_type_release(); for (auto func: TRACE_end_functions) func(); TRACE_start_functions.clear(); xbt_dict_free(&user_link_variables); xbt_dict_free(&user_host_variables); xbt_dict_free(&user_vm_variables); xbt_dict_free(&declared_marks); xbt_dict_free(&created_categories); /* close the trace files */ const char* format = xbt_cfg_get_string(OPT_TRACING_FORMAT); XBT_DEBUG("Tracing format %s\n", format); if(!strcmp(format, "Paje")){ TRACE_paje_end(); }else if (!strcmp(format, "TI")){ TRACE_TI_end(); }else{ xbt_die("Unknown trace format :%s ", format); } /* de-activate trace */ trace_active = 0; XBT_DEBUG("Tracing is off"); XBT_DEBUG("Tracing system is shutdown"); } return retval; } bool TRACE_needs_platform () { return TRACE_msg_process_is_enabled() || TRACE_msg_vm_is_enabled() || TRACE_categorized() || TRACE_uncategorized() || TRACE_platform () || (TRACE_smpi_is_enabled() && TRACE_smpi_is_grouped()); } bool TRACE_is_enabled() { return trace_enabled; } bool TRACE_platform() { return trace_platform; } bool TRACE_platform_topology() { return trace_platform_topology; } bool TRACE_is_configured() { return trace_configured; } bool TRACE_smpi_is_enabled() { return (trace_smpi_enabled || TRACE_smpi_is_grouped()) && TRACE_is_enabled(); } bool TRACE_smpi_is_grouped() { return trace_smpi_grouped; } bool TRACE_smpi_is_computing() { return trace_smpi_computing; } bool TRACE_smpi_is_sleeping() { return trace_smpi_sleeping; } bool TRACE_smpi_view_internals() { return trace_view_internals; } bool TRACE_categorized () { return trace_categorized; } bool TRACE_uncategorized () { return trace_uncategorized; } bool TRACE_msg_process_is_enabled() { return trace_msg_process_enabled && TRACE_is_enabled(); } bool TRACE_msg_vm_is_enabled() { return trace_msg_vm_enabled && TRACE_is_enabled(); } bool TRACE_disable_link() { return trace_disable_link && TRACE_is_enabled(); } bool TRACE_disable_speed() { return trace_disable_power && TRACE_is_enabled(); } bool TRACE_buffer () { return trace_buffer && TRACE_is_enabled(); } bool TRACE_onelink_only () { return trace_onelink_only && TRACE_is_enabled(); } bool TRACE_disable_destroy () { return trace_disable_destroy && TRACE_is_enabled(); } bool TRACE_basic () { return trace_basic && TRACE_is_enabled(); } bool TRACE_display_sizes () { return trace_display_sizes && trace_smpi_enabled && TRACE_is_enabled(); } char *TRACE_get_comment () { return xbt_cfg_get_string(OPT_TRACING_COMMENT); } char *TRACE_get_comment_file () { return xbt_cfg_get_string(OPT_TRACING_COMMENT_FILE); } int TRACE_precision () { return xbt_cfg_get_int(OPT_TRACING_PRECISION); } char *TRACE_get_filename() { return xbt_cfg_get_string(OPT_TRACING_FILENAME); } char *TRACE_get_viva_uncat_conf () { return xbt_cfg_get_string(OPT_VIVA_UNCAT_CONF); } char *TRACE_get_viva_cat_conf () { return xbt_cfg_get_string(OPT_VIVA_CAT_CONF); } void TRACE_global_init(int *argc, char **argv) { static int is_initialised = 0; if (is_initialised) return; is_initialised = 1; /* name of the tracefile */ xbt_cfg_register_string (OPT_TRACING_FILENAME, "simgrid.trace", nullptr, "Trace file created by the instrumented SimGrid."); xbt_cfg_register_boolean(OPT_TRACING, "no", nullptr, "Enable Tracing."); xbt_cfg_register_boolean(OPT_TRACING_PLATFORM, "no", nullptr, "Register the platform in the trace as a hierarchy."); xbt_cfg_register_boolean(OPT_TRACING_TOPOLOGY, "yes", nullptr, "Register the platform topology in the trace as a graph."); xbt_cfg_register_boolean(OPT_TRACING_SMPI, "no", nullptr, "Tracing of the SMPI interface."); xbt_cfg_register_boolean(OPT_TRACING_SMPI_GROUP,"no", nullptr, "Group MPI processes by host."); xbt_cfg_register_boolean(OPT_TRACING_SMPI_COMPUTING, "no", nullptr, "Generate states for timing out of SMPI parts of the application"); xbt_cfg_register_boolean(OPT_TRACING_SMPI_SLEEPING, "no", nullptr, "Generate states for timing out of SMPI parts of the application"); xbt_cfg_register_boolean(OPT_TRACING_SMPI_INTERNALS, "no", nullptr, "View internal messages sent by Collective communications in SMPI"); xbt_cfg_register_boolean(OPT_TRACING_CATEGORIZED, "no", nullptr, "Tracing categorized resource utilization of hosts and links."); xbt_cfg_register_boolean(OPT_TRACING_UNCATEGORIZED, "no", nullptr, "Tracing uncategorized resource utilization of hosts and links."); xbt_cfg_register_boolean(OPT_TRACING_MSG_PROCESS, "no", nullptr, "Tracing of MSG process behavior."); xbt_cfg_register_boolean(OPT_TRACING_MSG_VM, "no", nullptr, "Tracing of MSG process behavior."); xbt_cfg_register_boolean(OPT_TRACING_DISABLE_LINK, "no", nullptr, "Do not trace link bandwidth and latency."); xbt_cfg_register_boolean(OPT_TRACING_DISABLE_POWER, "no", nullptr, "Do not trace host power."); xbt_cfg_register_boolean(OPT_TRACING_BUFFER, "yes", nullptr, "Buffer trace events to put them in temporal order."); xbt_cfg_register_boolean(OPT_TRACING_ONELINK_ONLY, "no", nullptr, "Use only routes with one link to trace platform."); xbt_cfg_register_boolean(OPT_TRACING_DISABLE_DESTROY, "no", nullptr, "Disable platform containers destruction."); xbt_cfg_register_boolean(OPT_TRACING_BASIC, "no", nullptr, "Avoid extended events (impoverished trace file)."); xbt_cfg_register_boolean(OPT_TRACING_DISPLAY_SIZES, "no", nullptr, "(smpi only) Extended events with message size information"); xbt_cfg_register_string(OPT_TRACING_FORMAT, "Paje", nullptr, "(smpi only) Switch the output format of Tracing"); xbt_cfg_register_boolean(OPT_TRACING_FORMAT_TI_ONEFILE, "no", nullptr, "(smpi only) For replay format only : output to one file only"); xbt_cfg_register_string(OPT_TRACING_COMMENT, "", nullptr, "Comment to be added on the top of the trace file."); xbt_cfg_register_string(OPT_TRACING_COMMENT_FILE, "", nullptr, "The contents of the file are added to the top of the trace file as comment."); xbt_cfg_register_int(OPT_TRACING_PRECISION, 6, nullptr, "Numerical precision used when timestamping events " "(expressed in number of digits after decimal point)"); /* Viva graph configuration for uncategorized tracing */ xbt_cfg_register_string(OPT_VIVA_UNCAT_CONF, "", nullptr, "Viva Graph configuration file for uncategorized resource utilization traces."); xbt_cfg_register_string(OPT_VIVA_CAT_CONF, "", nullptr, "Viva Graph configuration file for categorized resource utilization traces."); xbt_cfg_register_alias(OPT_TRACING_COMMENT_FILE,"tracing/comment_file"); xbt_cfg_register_alias(OPT_TRACING_DISABLE_DESTROY, "tracing/disable_destroy"); xbt_cfg_register_alias(OPT_TRACING_DISABLE_LINK, "tracing/disable_link"); xbt_cfg_register_alias(OPT_TRACING_DISABLE_POWER, "tracing/disable_power"); xbt_cfg_register_alias(OPT_TRACING_DISPLAY_SIZES, "tracing/smpi/display_sizes"); xbt_cfg_register_alias(OPT_TRACING_FORMAT_TI_ONEFILE, "tracing/smpi/format/ti_one_file"); xbt_cfg_register_alias(OPT_TRACING_ONELINK_ONLY, "tracing/onelink_only"); /* instrumentation can be considered configured now */ trace_configured = 1; } static void print_line (const char *option, const char *desc, const char *longdesc, int detailed) { char str[INSTR_DEFAULT_STR_SIZE]; snprintf (str, INSTR_DEFAULT_STR_SIZE, "--cfg=%s ", option); int len = strlen (str); printf ("%s%*.*s %s\n", str, 30-len, 30-len, "", desc); if (longdesc != nullptr && detailed){ printf ("%s\n\n", longdesc); } } void TRACE_help (int detailed) { printf("Description of the tracing options accepted by this simulator:\n\n"); print_line (OPT_TRACING, "Enable the tracing system", " It activates the tracing system and register the simulation platform\n" " in the trace file. You have to enable this option to others take effect.", detailed); print_line (OPT_TRACING_CATEGORIZED, "Trace categorized resource utilization", " It activates the categorized resource utilization tracing. It should\n" " be enabled if tracing categories are used by this simulator.", detailed); print_line (OPT_TRACING_UNCATEGORIZED, "Trace uncategorized resource utilization", " It activates the uncategorized resource utilization tracing. Use it if\n" " this simulator do not use tracing categories and resource use have to be\n" " traced.", detailed); print_line (OPT_TRACING_FILENAME, "Filename to register traces", " A file with this name will be created to register the simulation. The file\n" " is in the Paje format and can be analyzed using Viva, Paje, and PajeNG visualization\n" " tools. More information can be found in these webpages:\n" " http://github.com/schnorr/viva/\n" " http://github.com/schnorr/pajeng/\n" " http://paje.sourceforge.net/", detailed); print_line (OPT_TRACING_SMPI, "Trace the MPI Interface (SMPI)", " This option only has effect if this simulator is SMPI-based. Traces the MPI\n" " interface and generates a trace that can be analyzed using Gantt-like\n" " visualizations. Every MPI function (implemented by SMPI) is transformed in a\n" " state, and point-to-point communications can be analyzed with arrows.", detailed); print_line (OPT_TRACING_SMPI_GROUP, "Group MPI processes by host (SMPI)", " This option only has effect if this simulator is SMPI-based. The processes\n" " are grouped by the hosts where they were executed.", detailed); print_line (OPT_TRACING_SMPI_COMPUTING, "Generates a \" Computing \" State", " This option aims at tracing computations in the application, outside SMPI\n" " to allow further study of simulated or real computation time", detailed); print_line (OPT_TRACING_SMPI_SLEEPING, "Generates a \" Sleeping \" State", " This option aims at tracing sleeps in the application, outside SMPI\n" " to allow further study of simulated or real sleep time", detailed); print_line (OPT_TRACING_SMPI_INTERNALS, "Generates tracing events corresponding", " to point-to-point messages sent by collective communications", detailed); print_line (OPT_TRACING_MSG_PROCESS, "Trace processes behavior (MSG)", " This option only has effect if this simulator is MSG-based. It traces the\n" " behavior of all categorized MSG processes, grouping them by hosts. This option\n" " can be used to track process location if this simulator has process migration.", detailed); print_line (OPT_TRACING_BUFFER, "Buffer events to put them in temporal order", " This option put some events in a time-ordered buffer using the insertion\n" " sort algorithm. The process of acquiring and releasing locks to access this\n" " buffer and the cost of the sorting algorithm make this process slow. The\n" " simulator performance can be severely impacted if this option is activated,\n" " but you are sure to get a trace file with events sorted.", detailed); print_line (OPT_TRACING_ONELINK_ONLY, "Consider only one link routes to trace platform", " This option changes the way SimGrid register its platform on the trace file.\n" " Normally, the tracing considers all routes (no matter their size) on the\n" " platform file to re-create the resource topology. If this option is activated,\n" " only the routes with one link are used to register the topology within an AS.\n" " Routes among AS continue to be traced as usual.", detailed); print_line (OPT_TRACING_DISABLE_DESTROY, "Disable platform containers destruction", " Disable the destruction of containers at the end of simulation. This can be\n" " used with simulators that have a different notion of time (different from\n" " the simulated time).", detailed); print_line (OPT_TRACING_BASIC, "Avoid extended events (impoverished trace file).", " Some visualization tools are not able to parse correctly the Paje file format.\n" " Use this option if you are using one of these tools to visualize the simulation\n" " trace. Keep in mind that the trace might be incomplete, without all the\n" " information that would be registered otherwise.", detailed); print_line (OPT_TRACING_DISPLAY_SIZES, "Only works for SMPI now. Add message size information", " Message size (in bytes) is added to links, and to states. For collectives,\n" " the displayed value is the more relevant to the collective (total sent by\n" " the process, usually)", detailed); print_line (OPT_TRACING_FORMAT, "Only works for SMPI now. Switch output format", " Default format is Paje. Time independent traces are also supported,\n" " to output traces that can later be used by the trace replay tool", detailed); print_line (OPT_TRACING_FORMAT_TI_ONEFILE, "Only works for SMPI now, and TI output format", " By default, each process outputs to a separate file, inside a filename_files folder\n" " By setting this option to yes, all processes will output to only one file\n" " This is meant to avoid opening thousands of files with large simulations", detailed); print_line (OPT_TRACING_COMMENT, "Comment to be added on the top of the trace file.", " Use this to add a comment line to the top of the trace file.", detailed); print_line (OPT_TRACING_COMMENT_FILE, "File contents added to trace file as comment.", " Use this to add the contents of a file to the top of the trace file as comment.", detailed); print_line (OPT_VIVA_UNCAT_CONF, "Generate a graph configuration for Viva", " This option can be used in all types of simulators build with SimGrid\n" " to generate a uncategorized resource utilization graph to be used as\n" " configuration for the Viva visualization tool. This option\n" " can be used with tracing/categorized:1 and tracing:1 options to\n" " analyze an unmodified simulator before changing it to contain\n" " categories.", detailed); print_line (OPT_VIVA_CAT_CONF, "Generate an uncategorized graph configuration for Viva", " This option can be used if this simulator uses tracing categories\n" " in its code. The file specified by this option holds a graph configuration\n" " file for the Viva visualization tool that can be used to analyze a categorized\n" " resource utilization.", detailed); print_line (OPT_TRACING_TOPOLOGY, "Register the platform topology as a graph", " This option (enabled by default) can be used to disable the tracing of\n" " the platform topology in the trace file. Sometimes, such task is really\n" " time consuming, since it must get the route from each host to other hosts\n" " within the same Autonomous System (AS).", detailed); } static void output_types (const char *name, xbt_dynar_t types, FILE *file) { unsigned int i; fprintf (file, " %s = (", name); for (i = xbt_dynar_length(types); i > 0; i--) { char *type = *(static_cast(xbt_dynar_get_ptr(types, i - 1))); fprintf (file, "\"%s\"", type); if (i - 1 > 0){ fprintf (file, ","); }else{ fprintf (file, ");\n"); } } xbt_dynar_free (&types); } static void output_categories (const char *name, xbt_dynar_t cats, FILE *file) { unsigned int i; fprintf (file, " values = ("); for (i = xbt_dynar_length(cats); i > 0; i--) { char *cat = *(static_cast(xbt_dynar_get_ptr(cats, i - 1))); fprintf (file, "\"%s%s\"", name, cat); if (i - 1 > 0){ fprintf (file, ","); }else{ fprintf (file, ");\n"); } } xbt_dynar_free (&cats); } static void uncat_configuration (FILE *file) { //register NODE and EDGE types output_types ("node", TRACE_get_node_types(), file); output_types ("edge", TRACE_get_edge_types(), file); fprintf (file, "\n"); //configuration for all nodes fprintf (file, " host = {\n" " type = \"square\";\n" " size = \"power\";\n" " values = (\"power_used\");\n" " };\n" " link = {\n" " type = \"rhombus\";\n" " size = \"bandwidth\";\n" " values = (\"bandwidth_used\");\n" " };\n"); //close } static void cat_configuration (FILE *file) { //register NODE and EDGE types output_types ("node", TRACE_get_node_types(), file); output_types ("edge", TRACE_get_edge_types(), file); fprintf (file, "\n"); //configuration for all nodes fprintf (file, " host = {\n" " type = \"square\";\n" " size = \"power\";\n"); output_categories ("p", TRACE_get_categories(), file); fprintf (file, " };\n" " link = {\n" " type = \"rhombus\";\n" " size = \"bandwidth\";\n"); output_categories ("b", TRACE_get_categories(), file); fprintf (file, " };\n"); //close } static void generate_uncat_configuration (const char *output, const char *name, int brackets) { if (output && strlen(output) > 0){ FILE *file = fopen (output, "w"); if (file == nullptr){ THROWF (system_error, 1, "Unable to open file (%s) for writing %s graph configuration (uncategorized).", output, name); } if (brackets) fprintf (file, "{\n"); uncat_configuration (file); if (brackets) fprintf (file, "}\n"); fclose (file); } } static void generate_cat_configuration (const char *output, const char *name, int brackets) { if (output && strlen(output) > 0){ //check if we do have categories declared if (xbt_dict_is_empty(created_categories)){ XBT_INFO("No categories declared, ignoring generation of %s graph configuration", name); return; } FILE *file = fopen (output, "w"); if (file == nullptr){ THROWF (system_error, 1, "Unable to open file (%s) for writing %s graph " "configuration (categorized).", output, name); } if (brackets) fprintf (file, "{\n"); cat_configuration (file); if (brackets) fprintf (file, "}\n"); fclose (file); } } void TRACE_generate_viva_uncat_conf () { generate_uncat_configuration (TRACE_get_viva_uncat_conf (), "viva", 0); } void TRACE_generate_viva_cat_conf () { generate_cat_configuration (TRACE_get_viva_cat_conf(), "viva", 0); } static int previous_trace_state = -1; void instr_pause_tracing () { previous_trace_state = trace_enabled; if (!TRACE_is_enabled()){ XBT_DEBUG ("Tracing is already paused, therefore do nothing."); }else{ XBT_DEBUG ("Tracing is being paused."); } trace_enabled = 0; XBT_DEBUG ("Tracing is paused."); } void instr_resume_tracing () { if (TRACE_is_enabled()){ XBT_DEBUG ("Tracing is already running while trying to resume, therefore do nothing."); }else{ XBT_DEBUG ("Tracing is being resumed."); } if (previous_trace_state != -1){ trace_enabled = previous_trace_state; }else{ trace_enabled = 1; } XBT_DEBUG ("Tracing is resumed."); previous_trace_state = -1; } #undef OPT_TRACING #undef OPT_TRACING_PLATFORM #undef OPT_TRACING_TOPOLOGY #undef OPT_TRACING_SMPI #undef OPT_TRACING_SMPI_GROUP #undef OPT_TRACING_CATEGORIZED #undef OPT_TRACING_UNCATEGORIZED #undef OPT_TRACING_MSG_PROCESS #undef OPT_TRACING_FILENAME #undef OPT_TRACING_BUFFER #undef OPT_TRACING_ONELINK_ONLY #undef OPT_TRACING_DISABLE_DESTROY #undef OPT_TRACING_BASIC #undef OPT_TRACING_COMMENT #undef OPT_TRACING_COMMENT_FILE #undef OPT_VIVA_UNCAT_CONF #undef OPT_VIVA_CAT_CONF SimGrid-3.14.159/src/instr/instr_interface.cpp000644 001750 001750 00000126506 13031031272 022713 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid_config.h" #include "src/instr/instr_private.h" #include "src/kernel/routing/NetCard.hpp" #include "src/surf/network_interface.hpp" #include "src/surf/surf_private.h" #include "surf/surf.h" typedef enum { INSTR_US_DECLARE, INSTR_US_SET, INSTR_US_ADD, INSTR_US_SUB } InstrUserVariable; XBT_LOG_NEW_DEFAULT_SUBCATEGORY (instr_api, instr, "API"); xbt_dict_t created_categories = nullptr; xbt_dict_t declared_marks = nullptr; xbt_dict_t user_host_variables = nullptr; xbt_dict_t user_vm_variables = nullptr; xbt_dict_t user_link_variables = nullptr; extern xbt_dict_t trivaNodeTypes; extern xbt_dict_t trivaEdgeTypes; static xbt_dynar_t instr_dict_to_dynar (xbt_dict_t filter) { if (!TRACE_is_enabled() || !TRACE_needs_platform()) return nullptr; xbt_dynar_t ret = xbt_dynar_new (sizeof(char*), &xbt_free_ref); xbt_dict_cursor_t cursor = nullptr; char *name; char *value; xbt_dict_foreach(filter, cursor, name, value) { xbt_dynar_push_as (ret, char*, xbt_strdup(name)); } return ret; } /** \ingroup TRACE_category * \brief Declare a new category with a random color. * * This function should be used to define a user category. The category can be used to differentiate the tasks that * are created during the simulation (for example, tasks from server1, server2, or request tasks, computation tasks, * communication tasks). All resource utilization (host power and link bandwidth) will be classified according to the * task category. Tasks that do not belong to a category are not traced. The color for the category that is being * declared is random. This function has no effect if a category with the same name has been already declared. * * See \ref outcomes_vizu for details on how to trace the (categorized) resource utilization. * * \param category The name of the new tracing category to be created. * * \see TRACE_category_with_color, MSG_task_set_category, SD_task_set_category */ void TRACE_category(const char *category) { TRACE_category_with_color (category, nullptr); } /** \ingroup TRACE_category * \brief Declare a new category with a color. * * Same as #TRACE_category, but let user specify a color encoded as a RGB-like string with three floats from 0 to 1. * So, to specify a red color, pass "1 0 0" as color parameter. A light-gray color can be specified using "0.7 0.7 0.7" * as color. This function has no effect if a category with the same name has been already declared. * * See \ref outcomes_vizu for details on how to trace the (categorized) resource utilization. * * \param category The name of the new tracing category to be created. * \param color The color of the category (see \ref outcomes_vizu to * know how to correctly specify the color) * * \see MSG_task_set_category, SD_task_set_category */ void TRACE_category_with_color (const char *category, const char *color) { /* safe switches. tracing has to be activated and if platform is not traced, we can't deal with categories */ if (!TRACE_is_enabled() || !TRACE_needs_platform()) return; if (!(TRACE_categorized() && category != nullptr)) return; //check if category is already created if (xbt_dict_get_or_null(created_categories, category) != nullptr) return; xbt_dict_set (created_categories, category, xbt_strdup("1"), nullptr); //define final_color char final_color[INSTR_DEFAULT_STR_SIZE]; if (!color){ //generate a random color double red = drand48(); double green = drand48(); double blue = drand48(); snprintf (final_color, INSTR_DEFAULT_STR_SIZE, "%f %f %f", red, green, blue); }else{ snprintf (final_color, INSTR_DEFAULT_STR_SIZE, "%s", color); } XBT_DEBUG("CAT,declare %s, \"%s\" \"%s\"", category, color, final_color); //define the type of this category on top of hosts and links instr_new_variable_type (category, final_color); } /** \ingroup TRACE_category * \brief Get declared categories * * This function should be used to get categories that were already declared with #TRACE_category or with * #TRACE_category_with_color. * * See \ref outcomes_vizu for details on how to trace the (categorized) resource utilization. * * \return A dynar with the declared categories, must be freed with xbt_dynar_free. * * \see MSG_task_set_category, SD_task_set_category */ xbt_dynar_t TRACE_get_categories () { if (!TRACE_is_enabled() || !TRACE_categorized()) return nullptr; return instr_dict_to_dynar (created_categories); } /** \ingroup TRACE_mark * \brief Declare a new type for tracing mark. * * This function declares a new Paje event type in the trace file that can be used by simulators to declare * application-level marks. This function is independent of which API is used in SimGrid. * * \param mark_type The name of the new type. * * \see TRACE_mark */ void TRACE_declare_mark(const char *mark_type) { /* safe switchs. tracing has to be activated and if platform is not traced, we can't deal with marks */ if (!TRACE_is_enabled() || !TRACE_needs_platform()) return; if (!mark_type) THROWF (tracing_error, 1, "mark_type is nullptr"); //check if mark_type is already declared if (xbt_dict_get_or_null(declared_marks, mark_type) != nullptr) { THROWF (tracing_error, 1, "mark_type with name (%s) is already declared", mark_type); } XBT_DEBUG("MARK,declare %s", mark_type); PJ_type_event_new(mark_type, PJ_type_get_root()); xbt_dict_set (declared_marks, mark_type, xbt_strdup("1"), nullptr); } /** \ingroup TRACE_mark * \brief Declare a new colored value for a previously declared mark type. * * This function declares a new colored value for a Paje event type in the trace file that can be used by simulators to * declare application-level marks. This function is independent of which API is used in SimGrid. The color needs to be * a string with three numbers separated by spaces in the range [0,1]. * A light-gray color can be specified using "0.7 0.7 0.7" as color. If a nullptr color is provided, the color used will * be white ("1 1 1"). * * \param mark_type The name of the new type. * \param mark_value The name of the new value for this type. * \param mark_color The color of the new value for this type. * * \see TRACE_mark */ void TRACE_declare_mark_value_with_color (const char *mark_type, const char *mark_value, const char *mark_color) { /* safe switches. tracing has to be activated and if platform is not traced, we can't deal with marks */ if (!TRACE_is_enabled() || !TRACE_needs_platform()) return; if (!mark_type) THROWF (tracing_error, 1, "mark_type is nullptr"); if (!mark_value) THROWF (tracing_error, 1, "mark_value is nullptr"); type_t type = PJ_type_get (mark_type, PJ_type_get_root()); if (!type){ THROWF (tracing_error, 1, "mark_type with name (%s) is not declared", mark_type); } char white[INSTR_DEFAULT_STR_SIZE] = "1.0 1.0 1.0"; if (!mark_color) mark_color = white; XBT_DEBUG("MARK,declare_value %s %s %s", mark_type, mark_value, mark_color); PJ_value_new (mark_value, mark_color, type); } /** \ingroup TRACE_mark * \brief Declare a new value for a previously declared mark type. * * This function declares a new value for a Paje event type in the trace file that can be used by simulators to declare * application-level marks. This function is independent of which API is used in SimGrid. Calling this function is the * same as calling \ref TRACE_declare_mark_value_with_color with a nullptr color. * * \param mark_type The name of the new type. * \param mark_value The name of the new value for this type. * * \see TRACE_mark */ void TRACE_declare_mark_value (const char *mark_type, const char *mark_value) { TRACE_declare_mark_value_with_color (mark_type, mark_value, nullptr); } /** * \ingroup TRACE_mark * \brief Create a new instance of a tracing mark type. * * This function creates a mark in the trace file. The first parameter had to be previously declared using * #TRACE_declare_mark, the second is the identifier for this mark instance. We recommend that the mark_value is a * unique value for the whole simulation. Nevertheless, this is not a strong requirement: the trace will be valid even * if there are multiple mark identifiers for the same trace. * * \param mark_type The name of the type for which the new instance will belong. * \param mark_value The name of the new instance mark. * * \see TRACE_declare_mark */ void TRACE_mark(const char *mark_type, const char *mark_value) { /* safe switches. tracing has to be activated and if platform is not traced, we can't deal with marks */ if (!TRACE_is_enabled() || !TRACE_needs_platform()) return; if (!mark_type) THROWF (tracing_error, 1, "mark_type is nullptr"); if (!mark_value) THROWF (tracing_error, 1, "mark_value is nullptr"); //check if mark_type is already declared type_t type = PJ_type_get (mark_type, PJ_type_get_root()); if (!type){ THROWF (tracing_error, 1, "mark_type with name (%s) is not declared", mark_type); } val_t value = PJ_value_get (mark_value, type); XBT_DEBUG("MARK %s %s", mark_type, mark_value); new_pajeNewEvent (MSG_get_clock(), PJ_container_get_root(), type, value); } /** \ingroup TRACE_mark * \brief Get declared marks * * This function should be used to get marks that were already declared with #TRACE_declare_mark. * * \return A dynar with the declared marks, must be freed with xbt_dynar_free. */ xbt_dynar_t TRACE_get_marks () { if (!TRACE_is_enabled()) return nullptr; return instr_dict_to_dynar (declared_marks); } static void instr_user_variable(double time, const char *resource, const char *variable, const char *father_type, double value, InstrUserVariable what, const char *color, xbt_dict_t filter) { /* safe switches. tracing has to be activated and if platform is not traced, we don't allow user variables */ if (!TRACE_is_enabled() || !TRACE_needs_platform()) return; //check if variable is already declared char *created = (char*)xbt_dict_get_or_null(filter, variable); if (what == INSTR_US_DECLARE){ if (created){//already declared return; }else{ xbt_dict_set (filter, variable, xbt_strdup("1"), nullptr); instr_new_user_variable_type (father_type, variable, color); } }else{ if (!created){//not declared, ignore return; } else { char valuestr[100]; snprintf(valuestr, 100, "%g", value); container_t container = PJ_container_get(resource); type_t type = PJ_type_get (variable, container->type); switch (what){ case INSTR_US_SET: new_pajeSetVariable(time, container, type, value); break; case INSTR_US_ADD: new_pajeAddVariable(time, container, type, value); break; case INSTR_US_SUB: new_pajeSubVariable(time, container, type, value); break; default: THROW_IMPOSSIBLE; break; } } } } static void instr_user_srcdst_variable(double time, const char *src, const char *dst, const char *variable, const char *father_type, double value, InstrUserVariable what) { sg_netcard_t src_elm = sg_netcard_by_name_or_null(src); if(!src_elm) xbt_die("Element '%s' not found!",src); sg_netcard_t dst_elm = sg_netcard_by_name_or_null(dst); if(!dst_elm) xbt_die("Element '%s' not found!",dst); std::vector route; simgrid::kernel::routing::NetZoneImpl::getGlobalRoute(src_elm, dst_elm, &route, nullptr); for (auto link : route) instr_user_variable (time, link->getName(), variable, father_type, value, what, nullptr, user_link_variables); } /** \ingroup TRACE_API * \brief Creates a file with the topology of the platform file used for the simulator. * * The graph topology will have the following properties: all hosts, links and routers of the platform file are mapped * to graph nodes; routes are mapped to edges. * The platform's AS are not represented in the output. * * \param filename The name of the file that will hold the graph. * * \return 1 of successful, 0 otherwise. */ int TRACE_platform_graph_export_graphviz (const char *filename) { /* returns 1 if successful, 0 otherwise */ if (!TRACE_is_enabled()) return 0; xbt_graph_t g = instr_routing_platform_graph(); if (g == nullptr) return 0; instr_routing_platform_graph_export_graphviz (g, filename); xbt_graph_free_graph(g, xbt_free_f, xbt_free_f, nullptr); return 1; } /* * Derived functions that use instr_user_variable and TRACE_user_srcdst_variable. They were previously defined as * pre-processors directives, but were transformed into functions so the user can track them using gdb. */ /* for VM variables */ /** \ingroup TRACE_user_variables * \brief Declare a new user variable associated to VMs. * * Declare a user variable that will be associated to VMs. A user vm variable can be used to trace user variables * such as the number of tasks in a VM, the number of clients in an application (for VMs), and so on. The color * associated to this new variable will be random. * * \param variable The name of the new variable to be declared. * * \see TRACE_vm_variable_declare_with_color */ void TRACE_vm_variable_declare (const char *variable) { instr_user_variable(0, nullptr, variable, "MSG_VM", 0, INSTR_US_DECLARE, nullptr, user_vm_variables); } /** \ingroup TRACE_user_variables * \brief Declare a new user variable associated to VMs with a color. * * Same as #TRACE_vm_variable_declare, but associated a color to the newly created user host variable. The color needs * to be a string with three numbers separated by spaces in the range [0,1]. * A light-gray color can be specified using "0.7 0.7 0.7" as color. * * \param variable The name of the new variable to be declared. * \param color The color for the new variable. */ void TRACE_vm_variable_declare_with_color (const char *variable, const char *color) { instr_user_variable(0, nullptr, variable, "MSG_VM", 0, INSTR_US_DECLARE, color, user_vm_variables); } /** \ingroup TRACE_user_variables * \brief Set the value of a variable of a host. * * \param vm The name of the VM to be considered. * \param variable The name of the variable to be considered. * \param value The new value of the variable. * * \see TRACE_vm_variable_declare, TRACE_vm_variable_add, TRACE_vm_variable_sub */ void TRACE_vm_variable_set (const char *vm, const char *variable, double value) { TRACE_vm_variable_set_with_time (MSG_get_clock(), vm, variable, value); } /** \ingroup TRACE_user_variables * \brief Add a value to a variable of a VM. * * \param vm The name of the VM to be considered. * \param variable The name of the variable to be considered. * \param value The value to be added to the variable. * * \see TRACE_vm_variable_declare, TRACE_vm_variable_set, TRACE_vm_variable_sub */ void TRACE_vm_variable_add (const char *vm, const char *variable, double value) { TRACE_vm_variable_add_with_time (MSG_get_clock(), vm, variable, value); } /** \ingroup TRACE_user_variables * \brief Subtract a value from a variable of a VM. * * \param vm The name of the vm to be considered. * \param variable The name of the variable to be considered. * \param value The value to be subtracted from the variable. * * \see TRACE_vm_variable_declare, TRACE_vm_variable_set, TRACE_vm_variable_add */ void TRACE_vm_variable_sub (const char *vm, const char *variable, double value) { TRACE_vm_variable_sub_with_time (MSG_get_clock(), vm, variable, value); } /** \ingroup TRACE_user_variables * \brief Set the value of a variable of a VM at a given timestamp. * * Same as #TRACE_vm_variable_set, but let user specify the time used to trace it. Users can specify a time that * is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param vm The name of the VM to be considered. * \param variable The name of the variable to be considered. * \param value The new value of the variable. * * \see TRACE_vm_variable_declare, TRACE_vm_variable_add_with_time, TRACE_vm_variable_sub_with_time */ void TRACE_vm_variable_set_with_time (double time, const char *vm, const char *variable, double value) { instr_user_variable(time, vm, variable, "MSG_VM", value, INSTR_US_SET, nullptr, user_vm_variables); } /** \ingroup TRACE_user_variables * \brief Add a value to a variable of a VM at a given timestamp. * * Same as #TRACE_vm_variable_add, but let user specify the time used to trace it. Users can specify a time that * is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param vm The name of the VM to be considered. * \param variable The name of the variable to be considered. * \param value The value to be added to the variable. * * \see TRACE_vm_variable_declare, TRACE_vm_variable_set_with_time, TRACE_vm_variable_sub_with_time */ void TRACE_vm_variable_add_with_time (double time, const char *vm, const char *variable, double value) { instr_user_variable(time, vm, variable, "MSG_VM", value, INSTR_US_ADD, nullptr, user_vm_variables); } /** \ingroup TRACE_user_variables * \brief Subtract a value from a variable of a VM at a given timestamp. * * Same as #TRACE_vm_variable_sub, but let user specify the time used to trace it. Users can specify a time that * is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param vm The name of the VM to be considered. * \param variable The name of the variable to be considered. * \param value The value to be subtracted from the variable. * * \see TRACE_vm_variable_declare, TRACE_vm_variable_set_with_time, TRACE_vm_variable_add_with_time */ void TRACE_vm_variable_sub_with_time (double time, const char *vm, const char *variable, double value) { instr_user_variable(time, vm, variable, "MSG_VM", value, INSTR_US_SUB, nullptr, user_vm_variables); } /** \ingroup TRACE_user_variables * \brief Get declared user vm variables * * This function should be used to get VM variables that were already declared with #TRACE_vm_variable_declare or with * #TRACE_vm_variable_declare_with_color. * * \return A dynar with the declared host variables, must be freed with xbt_dynar_free. */ xbt_dynar_t TRACE_get_vm_variables () { return instr_dict_to_dynar (user_vm_variables); } /* for host variables */ /** \ingroup TRACE_user_variables * \brief Declare a new user variable associated to hosts. * * Declare a user variable that will be associated to hosts. * A user host variable can be used to trace user variables such as the number of tasks in a server, the number of * clients in an application (for hosts), and so on. The color associated to this new variable will be random. * * \param variable The name of the new variable to be declared. * * \see TRACE_host_variable_declare_with_color */ void TRACE_host_variable_declare (const char *variable) { instr_user_variable(0, nullptr, variable, "HOST", 0, INSTR_US_DECLARE, nullptr, user_host_variables); } /** \ingroup TRACE_user_variables * \brief Declare a new user variable associated to hosts with a color. * * Same as #TRACE_host_variable_declare, but associated a color to the newly created user host variable. The color * needs to be a string with three numbers separated by spaces in the range [0,1]. * A light-gray color can be specified using "0.7 0.7 0.7" as color. * * \param variable The name of the new variable to be declared. * \param color The color for the new variable. */ void TRACE_host_variable_declare_with_color (const char *variable, const char *color) { instr_user_variable(0, nullptr, variable, "HOST", 0, INSTR_US_DECLARE, color, user_host_variables); } /** \ingroup TRACE_user_variables * \brief Set the value of a variable of a host. * * \param host The name of the host to be considered. * \param variable The name of the variable to be considered. * \param value The new value of the variable. * * \see TRACE_host_variable_declare, TRACE_host_variable_add, TRACE_host_variable_sub */ void TRACE_host_variable_set (const char *host, const char *variable, double value) { TRACE_host_variable_set_with_time (MSG_get_clock(), host, variable, value); } /** \ingroup TRACE_user_variables * \brief Add a value to a variable of a host. * * \param host The name of the host to be considered. * \param variable The name of the variable to be considered. * \param value The value to be added to the variable. * * \see TRACE_host_variable_declare, TRACE_host_variable_set, TRACE_host_variable_sub */ void TRACE_host_variable_add (const char *host, const char *variable, double value) { TRACE_host_variable_add_with_time (MSG_get_clock(), host, variable, value); } /** \ingroup TRACE_user_variables * \brief Subtract a value from a variable of a host. * * \param host The name of the host to be considered. * \param variable The name of the variable to be considered. * \param value The value to be subtracted from the variable. * * \see TRACE_host_variable_declare, TRACE_host_variable_set, TRACE_host_variable_add */ void TRACE_host_variable_sub (const char *host, const char *variable, double value) { TRACE_host_variable_sub_with_time (MSG_get_clock(), host, variable, value); } /** \ingroup TRACE_user_variables * \brief Set the value of a variable of a host at a given timestamp. * * Same as #TRACE_host_variable_set, but let user specify the time used to trace it. Users can specify a time that * is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param host The name of the host to be considered. * \param variable The name of the variable to be considered. * \param value The new value of the variable. * * \see TRACE_host_variable_declare, TRACE_host_variable_add_with_time, TRACE_host_variable_sub_with_time */ void TRACE_host_variable_set_with_time (double time, const char *host, const char *variable, double value) { instr_user_variable(time, host, variable, "HOST", value, INSTR_US_SET, nullptr, user_host_variables); } /** \ingroup TRACE_user_variables * \brief Add a value to a variable of a host at a given timestamp. * * Same as #TRACE_host_variable_add, but let user specify the time used to trace it. Users can specify a time that * is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param host The name of the host to be considered. * \param variable The name of the variable to be considered. * \param value The value to be added to the variable. * * \see TRACE_host_variable_declare, TRACE_host_variable_set_with_time, TRACE_host_variable_sub_with_time */ void TRACE_host_variable_add_with_time (double time, const char *host, const char *variable, double value) { instr_user_variable(time, host, variable, "HOST", value, INSTR_US_ADD, nullptr, user_host_variables); } /** \ingroup TRACE_user_variables * \brief Subtract a value from a variable of a host at a given timestamp. * * Same as #TRACE_host_variable_sub, but let user specify the time used to trace it. Users can specify a time that * is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param host The name of the host to be considered. * \param variable The name of the variable to be considered. * \param value The value to be subtracted from the variable. * * \see TRACE_host_variable_declare, TRACE_host_variable_set_with_time, TRACE_host_variable_add_with_time */ void TRACE_host_variable_sub_with_time (double time, const char *host, const char *variable, double value) { instr_user_variable(time, host, variable, "HOST", value, INSTR_US_SUB, nullptr, user_host_variables); } /** \ingroup TRACE_user_variables * \brief Get declared user host variables * * This function should be used to get host variables that were already declared with #TRACE_host_variable_declare or * with #TRACE_host_variable_declare_with_color. * * \return A dynar with the declared host variables, must be freed with xbt_dynar_free. */ xbt_dynar_t TRACE_get_host_variables () { return instr_dict_to_dynar (user_host_variables); } /* for link variables */ /** \ingroup TRACE_user_variables * \brief Declare a new user variable associated to links. * * Declare a user variable that will be associated to links. * A user link variable can be used, for example, to trace user variables such as the number of messages being * transferred through network links. The color associated to this new variable will be random. * * \param variable The name of the new variable to be declared. * * \see TRACE_link_variable_declare_with_color */ void TRACE_link_variable_declare (const char *variable) { instr_user_variable (0, nullptr, variable, "LINK", 0, INSTR_US_DECLARE, nullptr, user_link_variables); } /** \ingroup TRACE_user_variables * \brief Declare a new user variable associated to links with a color. * * Same as #TRACE_link_variable_declare, but associated a color to the newly created user link variable. The color * needs to be a string with three numbers separated by spaces in the range [0,1]. * A light-gray color can be specified using "0.7 0.7 0.7" as color. * * \param variable The name of the new variable to be declared. * \param color The color for the new variable. */ void TRACE_link_variable_declare_with_color (const char *variable, const char *color) { instr_user_variable (0, nullptr, variable, "LINK", 0, INSTR_US_DECLARE, color, user_link_variables); } /** \ingroup TRACE_user_variables * \brief Set the value of a variable of a link. * * \param link The name of the link to be considered. * \param variable The name of the variable to be considered. * \param value The new value of the variable. * * \see TRACE_link_variable_declare, TRACE_link_variable_add, TRACE_link_variable_sub */ void TRACE_link_variable_set (const char *link, const char *variable, double value) { TRACE_link_variable_set_with_time (MSG_get_clock(), link, variable, value); } /** \ingroup TRACE_user_variables * \brief Add a value to a variable of a link. * * \param link The name of the link to be considered. * \param variable The name of the variable to be considered. * \param value The value to be added to the variable. * * \see TRACE_link_variable_declare, TRACE_link_variable_set, TRACE_link_variable_sub */ void TRACE_link_variable_add (const char *link, const char *variable, double value) { TRACE_link_variable_add_with_time (MSG_get_clock(), link, variable, value); } /** \ingroup TRACE_user_variables * \brief Subtract a value from a variable of a link. * * \param link The name of the link to be considered. * \param variable The name of the variable to be considered. * \param value The value to be subtracted from the variable. * * \see TRACE_link_variable_declare, TRACE_link_variable_set, TRACE_link_variable_add */ void TRACE_link_variable_sub (const char *link, const char *variable, double value) { TRACE_link_variable_sub_with_time (MSG_get_clock(), link, variable, value); } /** \ingroup TRACE_user_variables * \brief Set the value of a variable of a link at a given timestamp. * * Same as #TRACE_link_variable_set, but let user specify the time used to trace it. Users can specify a time that * is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param link The name of the link to be considered. * \param variable The name of the variable to be considered. * \param value The new value of the variable. * * \see TRACE_link_variable_declare, TRACE_link_variable_add_with_time, TRACE_link_variable_sub_with_time */ void TRACE_link_variable_set_with_time (double time, const char *link, const char *variable, double value) { instr_user_variable (time, link, variable, "LINK", value, INSTR_US_SET, nullptr, user_link_variables); } /** \ingroup TRACE_user_variables * \brief Add a value to a variable of a link at a given timestamp. * * Same as #TRACE_link_variable_add, but let user specify the time used to trace it. Users can specify a time that * is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param link The name of the link to be considered. * \param variable The name of the variable to be considered. * \param value The value to be added to the variable. * * \see TRACE_link_variable_declare, TRACE_link_variable_set_with_time, TRACE_link_variable_sub_with_time */ void TRACE_link_variable_add_with_time (double time, const char *link, const char *variable, double value) { instr_user_variable (time, link, variable, "LINK", value, INSTR_US_ADD, nullptr, user_link_variables); } /** \ingroup TRACE_user_variables * \brief Subtract a value from a variable of a link at a given timestamp. * * Same as #TRACE_link_variable_sub, but let user specify the time used to trace it. Users can specify a time that * is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param link The name of the link to be considered. * \param variable The name of the variable to be considered. * \param value The value to be subtracted from the variable. * * \see TRACE_link_variable_declare, TRACE_link_variable_set_with_time, TRACE_link_variable_add_with_time */ void TRACE_link_variable_sub_with_time (double time, const char *link, const char *variable, double value) { instr_user_variable (time, link, variable, "LINK", value, INSTR_US_SUB, nullptr, user_link_variables); } /* for link variables, but with src and dst used for get_route */ /** \ingroup TRACE_user_variables * \brief Set the value of the variable present in the links connecting source and destination. * * Same as #TRACE_link_variable_set, but instead of providing the name of link to be considered, provide the source * and destination hosts. All links that are part of the route between source and destination will have the variable * set to the provided value. * * \param src The name of the source host for get route. * \param dst The name of the destination host for get route. * \param variable The name of the variable to be considered. * \param value The new value of the variable. * * \see TRACE_link_variable_declare, TRACE_link_srcdst_variable_add, TRACE_link_srcdst_variable_sub */ void TRACE_link_srcdst_variable_set (const char *src, const char *dst, const char *variable, double value) { TRACE_link_srcdst_variable_set_with_time (MSG_get_clock(), src, dst, variable, value); } /** \ingroup TRACE_user_variables * \brief Add a value to the variable present in the links connecting source and destination. * * Same as #TRACE_link_variable_add, but instead of providing the name of link to be considered, provide the source * and destination hosts. All links that are part of the route between source and destination will have the value * passed as parameter added to the current value of the variable name to be considered. * * \param src The name of the source host for get route. * \param dst The name of the destination host for get route. * \param variable The name of the variable to be considered. * \param value The value to be added to the variable. * * \see TRACE_link_variable_declare, TRACE_link_srcdst_variable_set, TRACE_link_srcdst_variable_sub */ void TRACE_link_srcdst_variable_add (const char *src, const char *dst, const char *variable, double value) { TRACE_link_srcdst_variable_add_with_time (MSG_get_clock(), src, dst, variable, value); } /** \ingroup TRACE_user_variables * \brief Subtract a value from the variable present in the links connecting source and destination. * * Same as #TRACE_link_variable_sub, but instead of providing the name of link to be considered, provide the source * and destination hosts. All links that are part of the route between source and destination will have the value * passed as parameter subtracted from the current value of the variable name to be considered. * * \param src The name of the source host for get route. * \param dst The name of the destination host for get route. * \param variable The name of the variable to be considered. * \param value The value to be subtracted from the variable. * * \see TRACE_link_variable_declare, TRACE_link_srcdst_variable_set, TRACE_link_srcdst_variable_add */ void TRACE_link_srcdst_variable_sub (const char *src, const char *dst, const char *variable, double value) { TRACE_link_srcdst_variable_sub_with_time (MSG_get_clock(), src, dst, variable, value); } /** \ingroup TRACE_user_variables * \brief Set the value of the variable present in the links connecting source and destination at a given timestamp. * * Same as #TRACE_link_srcdst_variable_set, but let user specify the time used to trace it. Users can specify a time * that is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param src The name of the source host for get route. * \param dst The name of the destination host for get route. * \param variable The name of the variable to be considered. * \param value The new value of the variable. * * \see TRACE_link_variable_declare, TRACE_link_srcdst_variable_add_with_time, TRACE_link_srcdst_variable_sub_with_time */ void TRACE_link_srcdst_variable_set_with_time (double time, const char *src, const char *dst, const char *variable, double value) { instr_user_srcdst_variable (time, src, dst, variable, "LINK", value, INSTR_US_SET); } /** \ingroup TRACE_user_variables * \brief Add a value to the variable present in the links connecting source and destination at a given timestamp. * * Same as #TRACE_link_srcdst_variable_add, but let user specify the time used to trace it. Users can specify a time * that is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param src The name of the source host for get route. * \param dst The name of the destination host for get route. * \param variable The name of the variable to be considered. * \param value The value to be added to the variable. * * \see TRACE_link_variable_declare, TRACE_link_srcdst_variable_set_with_time, TRACE_link_srcdst_variable_sub_with_time */ void TRACE_link_srcdst_variable_add_with_time (double time, const char *src, const char *dst, const char *variable, double value) { instr_user_srcdst_variable (time, src, dst, variable, "LINK", value, INSTR_US_ADD); } /** \ingroup TRACE_user_variables * \brief Subtract a value from the variable present in the links connecting source and dest. at a given timestamp. * * Same as #TRACE_link_srcdst_variable_sub, but let user specify the time used to trace it. Users can specify a time * that is not the simulated clock time as defined by the core simulator. This allows a fine-grain control of time * definition, but should be used with caution since the trace can be inconsistent if resource utilization traces are * also traced. * * \param time The timestamp to be used to tag this change of value. * \param src The name of the source host for get route. * \param dst The name of the destination host for get route. * \param variable The name of the variable to be considered. * \param value The value to be subtracted from the variable. * * \see TRACE_link_variable_declare, TRACE_link_srcdst_variable_set_with_time, TRACE_link_srcdst_variable_add_with_time */ void TRACE_link_srcdst_variable_sub_with_time (double time, const char *src, const char *dst, const char *variable, double value) { instr_user_srcdst_variable (time, src, dst, variable, "LINK", value, INSTR_US_SUB); } /** \ingroup TRACE_user_variables * \brief Get declared user link variables * * This function should be used to get link variables that were already declared with #TRACE_link_variable_declare or * with #TRACE_link_variable_declare_with_color. * * \return A dynar with the declared link variables, must be freed with xbt_dynar_free. */ xbt_dynar_t TRACE_get_link_variables () { return instr_dict_to_dynar (user_link_variables); } /** \ingroup TRACE_user_variables * \brief Declare a new user state associated to hosts. * * Declare a user state that will be associated to hosts. * A user host state can be used to trace application states. * * \param state The name of the new state to be declared. * * \see TRACE_host_state_declare_value */ void TRACE_host_state_declare (const char *state) { instr_new_user_state_type("HOST", state); } /** \ingroup TRACE_user_variables * \brief Declare a new value for a user state associated to hosts. * * Declare a value for a state. The color needs to be a string with 3 numbers separated by spaces in the range [0,1]. * A light-gray color can be specified using "0.7 0.7 0.7" as color. * * \param state The name of the new state to be declared. * \param value The name of the value * \param color The color of the value * * \see TRACE_host_state_declare */ void TRACE_host_state_declare_value (const char *state, const char *value, const char *color) { instr_new_value_for_user_state_type (state, value, color); } /** \ingroup TRACE_user_variables * \brief Set the user state to the given value. * * Change a user state previously declared to the given value. * * \param host The name of the host to be considered. * \param state The name of the state previously declared. * \param value The new value of the state. * * \see TRACE_host_state_declare, TRACE_host_push_state, TRACE_host_pop_state, TRACE_host_reset_state */ void TRACE_host_set_state (const char *host, const char *state, const char *value) { container_t container = PJ_container_get(host); type_t type = PJ_type_get (state, container->type); val_t val = PJ_value_get_or_new (value, nullptr, type); /* if user didn't declare a value with a color, use nullptr color */ new_pajeSetState(MSG_get_clock(), container, type, val); } /** \ingroup TRACE_user_variables * \brief Push a new value for a state of a given host. * * Change a user state previously declared by pushing the new value to the state. * * \param host The name of the host to be considered. * \param state The name of the state previously declared. * \param value The value to be pushed. * * \see TRACE_host_state_declare, TRACE_host_set_state, TRACE_host_pop_state, TRACE_host_reset_state */ void TRACE_host_push_state (const char *host, const char *state, const char *value) { container_t container = PJ_container_get(host); type_t type = PJ_type_get (state, container->type); val_t val = PJ_value_get_or_new (value, nullptr, type); /* if user didn't declare a value with a color, use nullptr color */ new_pajePushState(MSG_get_clock(), container, type, val); } /** \ingroup TRACE_user_variables * \brief Pop the last value of a state of a given host. * * Change a user state previously declared by removing the last value of the state. * * \param host The name of the host to be considered. * \param state The name of the state to be popped. * * \see TRACE_host_state_declare, TRACE_host_set_state, TRACE_host_push_state, TRACE_host_reset_state */ void TRACE_host_pop_state (const char *host, const char *state) { container_t container = PJ_container_get(host); type_t type = PJ_type_get (state, container->type); new_pajePopState(MSG_get_clock(), container, type); } /** \ingroup TRACE_user_variables * \brief Reset the state of a given host. * * Clear all previous values of a user state. * * \param host The name of the host to be considered. * \param state The name of the state to be cleared. * * \see TRACE_host_state_declare, TRACE_host_set_state, TRACE_host_push_state, TRACE_host_pop_state */ void TRACE_host_reset_state (const char *host, const char *state) { container_t container = PJ_container_get(host); type_t type = PJ_type_get (state, container->type); new_pajeResetState(MSG_get_clock(), container, type); } /** \ingroup TRACE_API * \brief Get Paje container types that can be mapped to the nodes of a graph. * * This function can be used to create a user made graph configuration file for Triva. Normally, it is used with the * functions defined in \ref TRACE_user_variables. * * \return A dynar with the types, must be freed with xbt_dynar_free. */ xbt_dynar_t TRACE_get_node_types () { return instr_dict_to_dynar (trivaNodeTypes); } /** \ingroup TRACE_API * \brief Get Paje container types that can be mapped to the edges of a graph. * * This function can be used to create a user made graph configuration file for Triva. Normally, it is used with the * functions defined in \ref TRACE_user_variables. * * \return A dynar with the types, must be freed with xbt_dynar_free. */ xbt_dynar_t TRACE_get_edge_types () { return instr_dict_to_dynar (trivaEdgeTypes); } /** \ingroup TRACE_API * \brief Pauses all tracing activities. * \see TRACE_resume */ void TRACE_pause () { instr_pause_tracing(); } /** \ingroup TRACE_API * \brief Resumes all tracing activities. * \see TRACE_pause */ void TRACE_resume () { instr_resume_tracing(); } SimGrid-3.14.159/src/instr/instr_paje_trace.cpp000644 001750 001750 00000026033 13031031272 023042 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "src/instr/instr_private.h" #include "xbt/virtu.h" /* sg_cmdline */ #include #include /** std::setprecision **/ #include "simgrid/sg_config.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(instr_paje_trace, instr_trace, "tracing event system"); extern FILE * tracing_file; extern s_instr_trace_writer_t active_writer; static std::stringstream stream; static void print_paje_debug(std::string functionName, paje_event_t event) { XBT_DEBUG("%s: event_type=%d, timestamp=%.*f", __FUNCTION__, (int)event->event_type, TRACE_precision(), event->timestamp); } template static void init_stream(paje_event_t event) { stream << std::fixed << std::setprecision(TRACE_precision()); stream << (int) event->event_type; } static void print_row() { stream << std::endl; fprintf(tracing_file, "%s", stream.str().c_str()); stream.str(""); stream.clear(); } static void print_timestamp(paje_event_t event) { stream << " "; /* prevent 0.0000 in the trace - this was the behavior before the transition to c++ */ if (event->timestamp < 1e-12) stream << 0; else stream << event->timestamp; } template static void print_default_pajeLink_row(paje_event_t& event) { init_stream(event); print_timestamp(event); stream << " " << static_cast(event->data)->type->id << " " << static_cast(event->data)->container->id << " " << static_cast(event->data)->value; } template static void print_default_pajeState_row(paje_event_t& event) { init_stream(event); print_timestamp(event); stream << " " << static_cast(event->data)->type->id << " " << static_cast(event->data)->container->id; } template static void print_default_pajeType_row(paje_event_t& event) { init_stream(event); stream << " " << static_cast(event->data)->type->id << " " << static_cast(event->data)->type->father->id << " " << static_cast(event->data)->type->name; } template static void print_default_pajeVariable_row(paje_event_t& event) { init_stream(event); print_timestamp(event); stream << " " << static_cast(event->data)->type->id << " " << static_cast(event->data)->container->id << " " << static_cast(event->data)->value; print_row(); } void TRACE_paje_init() { active_writer.print_DefineContainerType = &print_pajeDefineContainerType; active_writer.print_DefineVariableType = &print_pajeDefineVariableType; active_writer.print_DefineStateType = &print_pajeDefineStateType; active_writer.print_DefineEventType = &print_pajeDefineEventType; active_writer.print_DefineLinkType = &print_pajeDefineLinkType; active_writer.print_DefineEntityValue = &print_pajeDefineEntityValue; active_writer.print_CreateContainer = &print_pajeCreateContainer; active_writer.print_DestroyContainer = &print_pajeDestroyContainer; active_writer.print_SetVariable = &print_pajeSetVariable; active_writer.print_AddVariable = &print_pajeAddVariable; active_writer.print_SubVariable = &print_pajeSubVariable; active_writer.print_SetState = &print_pajeSetState; active_writer.print_PushState = &print_pajePushState; active_writer.print_PopState = &print_pajePopState; active_writer.print_ResetState = &print_pajeResetState; active_writer.print_StartLink = &print_pajeStartLink; active_writer.print_EndLink = &print_pajeEndLink; active_writer.print_NewEvent = &print_pajeNewEvent; } void TRACE_paje_start() { char *filename = TRACE_get_filename(); tracing_file = fopen(filename, "w"); if (tracing_file == nullptr){ THROWF (system_error, 1, "Tracefile %s could not be opened for writing.", filename); } XBT_DEBUG("Filename %s is open for writing", filename); /* output generator version */ fprintf (tracing_file, "#This file was generated using SimGrid-%d.%d.%d\n", SIMGRID_VERSION_MAJOR, SIMGRID_VERSION_MINOR, SIMGRID_VERSION_PATCH); fprintf (tracing_file, "#["); unsigned int cpt; char *str; xbt_dynar_foreach (xbt_cmdline, cpt, str){ fprintf(tracing_file, "%s ",str); } fprintf (tracing_file, "]\n"); /* output one line comment */ dump_comment (TRACE_get_comment()); /* output comment file */ dump_comment_file (TRACE_get_comment_file()); /* output header */ TRACE_header(TRACE_basic(),TRACE_display_sizes()); } void TRACE_paje_end() { fclose(tracing_file); char *filename = TRACE_get_filename(); XBT_DEBUG("Filename %s is closed", filename); } void print_pajeDefineContainerType(paje_event_t event) { print_paje_debug(__FUNCTION__, event); print_default_pajeType_row(event); print_row(); } void print_pajeDefineVariableType(paje_event_t event) { print_paje_debug(__FUNCTION__, event); print_default_pajeType_row(event); if(static_cast(event->data)->type->color) stream << " \"" << static_cast(event->data)->type->color << "\""; print_row(); } void print_pajeDefineStateType(paje_event_t event) { print_paje_debug(__FUNCTION__, event); print_default_pajeType_row(event); print_row(); } void print_pajeDefineEventType(paje_event_t event) { print_paje_debug(__FUNCTION__, event); print_default_pajeType_row(event); print_row(); } void print_pajeDefineLinkType(paje_event_t event) { print_paje_debug(__FUNCTION__, event); init_stream(event); stream << " " << static_cast(event->data)->type->id << " " << static_cast(event->data)->type->father->id << " " << static_cast(event->data)->source->id << " " << static_cast(event->data)->dest->id << " " << static_cast(event->data)->type->name; print_row(); } void print_pajeDefineEntityValue (paje_event_t event) { print_paje_debug(__FUNCTION__, event); init_stream(event); stream << " " << static_cast(event->data)->value->id << " " << static_cast(event->data)->value->father->id << " " << static_cast(event->data)->value->name; if(static_cast(event->data)->value->color) stream << " \"" << static_cast(event->data)->value->color << "\""; print_row(); } void print_pajeCreateContainer(paje_event_t event) { print_paje_debug(__FUNCTION__, event); init_stream(event); print_timestamp(event); stream << " " << static_cast(event->data)->container->id << " " << static_cast(event->data)->container->type->id << " " << static_cast(event->data)->container->father->id << " \"" << static_cast(event->data)->container->name << "\""; print_row(); } void print_pajeDestroyContainer(paje_event_t event) { print_paje_debug(__FUNCTION__, event); init_stream(event); print_timestamp(event); stream << " " << static_cast(event->data)->container->type->id << " " << static_cast(event->data)->container->id; print_row(); } void print_pajeSetVariable(paje_event_t event) { print_paje_debug(__FUNCTION__, event); print_default_pajeVariable_row(event); } void print_pajeAddVariable(paje_event_t event) { print_paje_debug(__FUNCTION__, event); print_default_pajeVariable_row(event); } void print_pajeSubVariable(paje_event_t event) { print_paje_debug(__FUNCTION__, event); print_default_pajeVariable_row(event); } void print_pajeSetState(paje_event_t event) { print_paje_debug(__FUNCTION__, event); print_default_pajeState_row(event); stream << " " << static_cast(event->data)->value->id; #if HAVE_SMPI if (xbt_cfg_get_boolean("smpi/trace-call-location")) { stream << " \"" << static_cast(event->data)->filename << "\" " << static_cast(event->data)->linenumber; } #endif print_row(); } void print_pajePushState(paje_event_t event) { print_paje_debug(__FUNCTION__, event); print_default_pajeState_row(event); stream << " " << static_cast(event->data)->value->id; if (TRACE_display_sizes()) { stream << " "; if (static_cast(event->data)->extra != nullptr) { stream << static_cast(static_cast(event->data)->extra)->send_size; } else { stream << 0; } } #if HAVE_SMPI if (xbt_cfg_get_boolean("smpi/trace-call-location")) { stream << " \"" << static_cast(event->data)->filename << "\" " << static_cast(event->data)->linenumber; } #endif print_row(); if (static_cast(event->data)->extra != nullptr) { if (static_cast(static_cast(event->data)->extra)->sendcounts != nullptr) xbt_free(static_cast(static_cast(event->data)->extra)->sendcounts); if (static_cast(static_cast(event->data)->extra)->recvcounts != nullptr) xbt_free(static_cast(static_cast(event->data)->extra)->recvcounts); xbt_free(static_cast(event->data)->extra); } } void print_pajePopState(paje_event_t event) { print_paje_debug(__FUNCTION__, event); print_default_pajeState_row(event); print_row(); } void print_pajeResetState(paje_event_t event) { print_paje_debug(__FUNCTION__, event); print_default_pajeState_row(event); print_row(); } void print_pajeStartLink(paje_event_t event) { print_paje_debug(__FUNCTION__, event); print_default_pajeLink_row(event); stream << " " << static_cast(event->data)->sourceContainer->id << " " << static_cast(event->data)->key; if (TRACE_display_sizes()) { stream << " " << static_cast(event->data)->size; } print_row(); } void print_pajeEndLink(paje_event_t event) { print_paje_debug(__FUNCTION__, event); print_default_pajeLink_row(event); stream << " " << static_cast(event->data)->destContainer->id << " " << static_cast(event->data)->key; print_row(); } void print_pajeNewEvent (paje_event_t event) { print_paje_debug(__FUNCTION__, event); init_stream(event); print_timestamp(event); stream << " " << static_cast(event->data)->type->id << " " << static_cast(event->data)->container->id << " " << static_cast(event->data)->value->id; print_row(); } SimGrid-3.14.159/src/instr/instr_paje_header.cpp000644 001750 001750 00000023400 13031031272 023167 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010, 2012-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "src/instr/instr_private.h" #include "simgrid/sg_config.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY(instr_paje_header, instr, "Paje tracing event system (header)"); extern FILE *tracing_file; static void TRACE_header_PajeDefineContainerType (int basic, int size) { fprintf(tracing_file, "%%EventDef PajeDefineContainerType %d\n", PAJE_DefineContainerType); fprintf(tracing_file, "%% Alias string\n"); if (basic){ fprintf(tracing_file, "%% ContainerType string\n"); }else{ fprintf(tracing_file, "%% Type string\n"); } fprintf(tracing_file, "%% Name string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeDefineVariableType (int basic, int size) { fprintf(tracing_file, "%%EventDef PajeDefineVariableType %d\n", PAJE_DefineVariableType); fprintf(tracing_file, "%% Alias string\n"); if (basic){ fprintf(tracing_file, "%% ContainerType string\n"); }else{ fprintf(tracing_file, "%% Type string\n"); } fprintf(tracing_file, "%% Name string\n"); fprintf(tracing_file, "%% Color color\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeDefineStateType (int basic, int size) { fprintf(tracing_file, "%%EventDef PajeDefineStateType %d\n", PAJE_DefineStateType); fprintf(tracing_file, "%% Alias string\n"); if (basic){ fprintf(tracing_file, "%% ContainerType string\n"); }else{ fprintf(tracing_file, "%% Type string\n"); } fprintf(tracing_file, "%% Name string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeDefineEventType (int basic, int size) { fprintf(tracing_file, "%%EventDef PajeDefineEventType %d\n", PAJE_DefineEventType); fprintf(tracing_file, "%% Alias string\n"); if (basic){ fprintf(tracing_file, "%% ContainerType string\n"); }else{ fprintf(tracing_file, "%% Type string\n"); } fprintf(tracing_file, "%% Name string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeDefineLinkType (int basic, int size) { fprintf(tracing_file, "%%EventDef PajeDefineLinkType %d\n", PAJE_DefineLinkType); fprintf(tracing_file, "%% Alias string\n"); if (basic){ fprintf(tracing_file, "%% ContainerType string\n"); fprintf(tracing_file, "%% SourceContainerType string\n"); fprintf(tracing_file, "%% DestContainerType string\n"); }else{ fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% StartContainerType string\n"); fprintf(tracing_file, "%% EndContainerType string\n"); } fprintf(tracing_file, "%% Name string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeDefineEntityValue (int basic, int size) { fprintf(tracing_file, "%%EventDef PajeDefineEntityValue %d\n", PAJE_DefineEntityValue); fprintf(tracing_file, "%% Alias string\n"); if (basic){ fprintf(tracing_file, "%% EntityType string\n"); }else{ fprintf(tracing_file, "%% Type string\n"); } fprintf(tracing_file, "%% Name string\n"); fprintf(tracing_file, "%% Color color\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeCreateContainer (int basic, int size) { fprintf(tracing_file, "%%EventDef PajeCreateContainer %d\n", PAJE_CreateContainer); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Alias string\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%% Name string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeDestroyContainer (int basic, int size) { fprintf(tracing_file, "%%EventDef PajeDestroyContainer %d\n", PAJE_DestroyContainer); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Name string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeSetVariable (int basic, int size) { fprintf(tracing_file, "%%EventDef PajeSetVariable %d\n", PAJE_SetVariable); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%% Value double\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeAddVariable (int basic, int size) { fprintf(tracing_file, "%%EventDef PajeAddVariable %d\n", PAJE_AddVariable); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%% Value double\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeSubVariable (int basic, int size) { fprintf(tracing_file, "%%EventDef PajeSubVariable %d\n", PAJE_SubVariable); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%% Value double\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeSetState (int basic, int size) { fprintf(tracing_file, "%%EventDef PajeSetState %d\n", PAJE_SetState); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%% Value string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajePushState (int basic, int size) { fprintf(tracing_file, "%%EventDef PajePushState %d\n", PAJE_PushState); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%% Value string\n"); if (size) fprintf(tracing_file, "%% Size int\n"); #if HAVE_SMPI if (xbt_cfg_get_boolean("smpi/trace-call-location")) { /** * paje currently (May 2016) uses "Filename" and "Linenumber" as * reserved words. We cannot use them... */ fprintf(tracing_file, "%% Fname string\n"); fprintf(tracing_file, "%% Lnumber int\n"); } #endif fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajePopState (int basic, int size) { fprintf(tracing_file, "%%EventDef PajePopState %d\n", PAJE_PopState); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeResetState (int basic, int size) { if (basic) return; fprintf(tracing_file, "%%EventDef PajeResetState %d\n", PAJE_ResetState); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeStartLink (int basic, int size) { fprintf(tracing_file, "%%EventDef PajeStartLink %d\n", PAJE_StartLink); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%% Value string\n"); if (basic){ fprintf(tracing_file, "%% SourceContainer string\n"); }else{ fprintf(tracing_file, "%% StartContainer string\n"); } fprintf(tracing_file, "%% Key string\n"); if (size) fprintf(tracing_file, "%% Size int\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeEndLink (int basic, int size) { fprintf(tracing_file, "%%EventDef PajeEndLink %d\n", PAJE_EndLink); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%% Value string\n"); if (basic){ fprintf(tracing_file, "%% DestContainer string\n"); }else{ fprintf(tracing_file, "%% EndContainer string\n"); } fprintf(tracing_file, "%% Key string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } static void TRACE_header_PajeNewEvent (int basic, int size) { fprintf(tracing_file, "%%EventDef PajeNewEvent %d\n", PAJE_NewEvent); fprintf(tracing_file, "%% Time date\n"); fprintf(tracing_file, "%% Type string\n"); fprintf(tracing_file, "%% Container string\n"); fprintf(tracing_file, "%% Value string\n"); fprintf(tracing_file, "%%EndEventDef\n"); } void TRACE_header(int basic, int size) { XBT_DEBUG ("Define paje header"); TRACE_header_PajeDefineContainerType (basic, size); TRACE_header_PajeDefineVariableType (basic, size); TRACE_header_PajeDefineStateType (basic, size); TRACE_header_PajeDefineEventType (basic, size); TRACE_header_PajeDefineLinkType (basic, size); TRACE_header_PajeDefineEntityValue (basic, size); TRACE_header_PajeCreateContainer (basic, size); TRACE_header_PajeDestroyContainer (basic, size); TRACE_header_PajeSetVariable (basic, size); TRACE_header_PajeAddVariable (basic, size); TRACE_header_PajeSubVariable (basic, size); TRACE_header_PajeSetState (basic, size); TRACE_header_PajePushState (basic, size); TRACE_header_PajePopState (basic, size); TRACE_header_PajeResetState (basic, size); TRACE_header_PajeStartLink (basic, size); TRACE_header_PajeEndLink (basic, size); TRACE_header_PajeNewEvent (basic, size); } SimGrid-3.14.159/src/instr/instr_resource_utilization.cpp000644 001750 001750 00000010773 13031031272 025233 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "src/instr/instr_private.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY (instr_resource, instr, "tracing (un)-categorized resource utilization"); //to check if variables were previously set to 0, otherwise paje won't simulate them static xbt_dict_t platform_variables; //used by all methods static void __TRACE_surf_check_variable_set_to_zero(double now, const char *variable, const char *resource) { /* To trace resource utilization, we use pajeAddVariable and pajeSubVariable only. * The Paje simulator needs a pajeSetVariable in the first place so it knows the initial value of all variables for * subsequent adds/subs. If we don't do so, the first pajeAddVariable is added to a non-determined value within * the Paje simulator, causing analysis problems. */ // create a key considering the resource and variable int n = strlen(variable)+strlen(resource)+1; char *key = (char*)xbt_malloc(n*sizeof(char)); snprintf (key, n, "%s%s", resource, variable); // check if key exists: if it doesn't, set the variable to zero and mark this in the dict if (!xbt_dict_get_or_null(platform_variables, key)) { container_t container = PJ_container_get (resource); type_t type = PJ_type_get (variable, container->type); new_pajeSetVariable (now, container, type, 0); xbt_dict_set(platform_variables, key, (char*)"", nullptr); } xbt_free(key); } static void instr_event (double now, double delta, type_t variable, container_t resource, double value) { __TRACE_surf_check_variable_set_to_zero(now, variable->name, resource->name); new_pajeAddVariable(now, resource, variable, value); new_pajeSubVariable(now + delta, resource, variable, value); } /* TRACE_surf_link_set_utilization: entry point from SimGrid */ void TRACE_surf_link_set_utilization(const char *resource, const char *category, double value, double now, double delta) { //only trace link utilization if link is known by tracing mechanism if (!PJ_container_get_or_null(resource)) return; if (!value) return; //trace uncategorized link utilization if (TRACE_uncategorized()){ XBT_DEBUG("UNCAT LINK [%f - %f] %s bandwidth_used %f", now, now+delta, resource, value); container_t container = PJ_container_get (resource); type_t type = PJ_type_get ("bandwidth_used", container->type); instr_event (now, delta, type, container, value); } //trace categorized utilization if (TRACE_categorized()){ if (!category) return; //variable of this category starts by 'b', because we have a link here char category_type[INSTR_DEFAULT_STR_SIZE]; snprintf (category_type, INSTR_DEFAULT_STR_SIZE, "b%s", category); XBT_DEBUG("CAT LINK [%f - %f] %s %s %f", now, now+delta, resource, category_type, value); container_t container = PJ_container_get (resource); type_t type = PJ_type_get (category_type, container->type); instr_event (now, delta, type, container, value); } return; } /* TRACE_surf_host_set_utilization: entry point from SimGrid */ void TRACE_surf_host_set_utilization(const char *resource, const char *category, double value, double now, double delta) { //only trace host utilization if host is known by tracing mechanism container_t container = PJ_container_get_or_null(resource); if (!container) return; if (!value) return; //trace uncategorized host utilization if (TRACE_uncategorized()){ XBT_DEBUG("UNCAT HOST [%f - %f] %s power_used %f", now, now+delta, resource, value); type_t type = PJ_type_get ("power_used", container->type); instr_event (now, delta, type, container, value); } //trace categorized utilization if (TRACE_categorized()){ if (!category) return; //variable of this category starts by 'p', because we have a host here char category_type[INSTR_DEFAULT_STR_SIZE]; snprintf (category_type, INSTR_DEFAULT_STR_SIZE, "p%s", category); XBT_DEBUG("CAT HOST [%f - %f] %s %s %f", now, now+delta, resource, category_type, value); type_t type = PJ_type_get (category_type, container->type); instr_event (now, delta, type, container, value); } return; } void TRACE_surf_resource_utilization_alloc() { platform_variables = xbt_dict_new_homogeneous(nullptr); } void TRACE_surf_resource_utilization_release() { xbt_dict_free(&platform_variables); } SimGrid-3.14.159/src/instr/instr_private.h000644 001750 001750 00000035535 13031031272 022073 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #ifndef INSTR_PRIVATE_H_ #define INSTR_PRIVATE_H_ #include #include "simgrid/instr.h" #include "instr/instr_interface.h" #include "src/internal_config.h" #include "simgrid_config.h" SG_BEGIN_DECL() /* Need to define function drand48 for Windows */ /* FIXME: use _drand48() defined in src/surf/random_mgr.c instead */ #ifdef _WIN32 # define drand48() (rand()/(RAND_MAX + 1.0)) #endif #define INSTR_DEFAULT_STR_SIZE 500 #include "xbt/graph.h" #include "xbt/dict.h" typedef enum { PAJE_DefineContainerType, PAJE_DefineVariableType, PAJE_DefineStateType, PAJE_DefineEventType, PAJE_DefineLinkType, PAJE_DefineEntityValue, PAJE_CreateContainer, PAJE_DestroyContainer, PAJE_SetVariable, PAJE_AddVariable, PAJE_SubVariable, PAJE_SetState, PAJE_PushState, PAJE_PopState, PAJE_ResetState, PAJE_StartLink, PAJE_EndLink, PAJE_NewEvent } e_event_type; typedef enum { TYPE_VARIABLE, TYPE_LINK, TYPE_CONTAINER, TYPE_STATE, TYPE_EVENT } e_entity_types; typedef struct s_type *type_t; typedef struct s_type { char *id; char *name; char *color; e_entity_types kind; struct s_type *father; xbt_dict_t children; xbt_dict_t values; //valid for all types except variable and container }s_type_t; typedef struct s_val *val_t; typedef struct s_val { char *id; char *name; char *color; type_t father; }s_val_t; typedef enum { INSTR_HOST, INSTR_LINK, INSTR_ROUTER, INSTR_AS, INSTR_SMPI, INSTR_MSG_VM, INSTR_MSG_PROCESS, INSTR_MSG_TASK } e_container_types; typedef struct s_container *container_t; typedef struct s_container { sg_netcard_t netcard; char *name; /* Unique name of this container */ char *id; /* Unique id of this container */ type_t type; /* Type of this container */ int level; /* Level in the hierarchy, root level is 0 */ e_container_types kind; /* This container is of what kind */ struct s_container *father; xbt_dict_t children; }s_container_t; typedef struct paje_event *paje_event_t; typedef struct paje_event { double timestamp; e_event_type event_type; void (*print) (paje_event_t event); void (*free) (paje_event_t event); void *data; } s_paje_event_t; typedef struct s_defineContainerType *defineContainerType_t; typedef struct s_defineContainerType { type_t type; }s_defineContainerType_t; typedef struct s_defineVariableType *defineVariableType_t; typedef struct s_defineVariableType { type_t type; }s_defineVariableType_t; typedef struct s_defineStateType *defineStateType_t; typedef struct s_defineStateType { type_t type; }s_defineStateType_t; typedef struct s_defineEventType *defineEventType_t; typedef struct s_defineEventType { type_t type; }s_defineEventType_t; typedef struct s_defineLinkType *defineLinkType_t; typedef struct s_defineLinkType { type_t type; type_t source; type_t dest; }s_defineLinkType_t; typedef struct s_defineEntityValue *defineEntityValue_t; typedef struct s_defineEntityValue { val_t value; }s_defineEntityValue_t; typedef struct s_createContainer *createContainer_t; typedef struct s_createContainer { container_t container; }s_createContainer_t; typedef struct s_destroyContainer *destroyContainer_t; typedef struct s_destroyContainer { container_t container; }s_destroyContainer_t; typedef struct s_setVariable *setVariable_t; typedef struct s_setVariable { container_t container; type_t type; double value; }s_setVariable_t; typedef struct s_addVariable *addVariable_t; typedef struct s_addVariable { container_t container; type_t type; double value; }s_addVariable_t; typedef struct s_subVariable *subVariable_t; typedef struct s_subVariable { container_t container; type_t type; double value; }s_subVariable_t; typedef struct s_setState *setState_t; typedef struct s_setState { container_t container; type_t type; val_t value; const char* filename; int linenumber; }s_setState_t; typedef struct s_pushState *pushState_t; typedef struct s_pushState { container_t container; type_t type; val_t value; int size; const char* filename; int linenumber; void* extra; }s_pushState_t; typedef struct s_popState *popState_t; typedef struct s_popState { container_t container; type_t type; xbt_dynar_t extra; }s_popState_t; typedef struct s_resetState *resetState_t; typedef struct s_resetState { container_t container; type_t type; }s_resetState_t; typedef struct s_startLink *startLink_t; typedef struct s_startLink { container_t container; type_t type; container_t sourceContainer; char *value; char *key; int size; }s_startLink_t; typedef struct s_endLink *endLink_t; typedef struct s_endLink { container_t container; type_t type; container_t destContainer; char *value; char *key; }s_endLink_t; typedef struct s_newEvent *newEvent_t; typedef struct s_newEvent { container_t container; type_t type; val_t value; }s_newEvent_t; extern XBT_PRIVATE xbt_dict_t created_categories; extern XBT_PRIVATE xbt_dict_t declared_marks; extern XBT_PRIVATE xbt_dict_t user_host_variables; extern XBT_PRIVATE xbt_dict_t user_vm_variables; extern XBT_PRIVATE xbt_dict_t user_link_variables; extern XBT_PRIVATE double TRACE_last_timestamp_to_dump; /* instr_paje_header.c */ XBT_PRIVATE void TRACE_header(int basic, int size); /* from paje.c */ XBT_PRIVATE void TRACE_paje_init(); XBT_PRIVATE void TRACE_paje_start(); XBT_PRIVATE void TRACE_paje_end(); XBT_PRIVATE void TRACE_paje_dump_buffer (int force); XBT_PUBLIC(void) new_pajeDefineContainerType(type_t type); XBT_PUBLIC(void) new_pajeDefineVariableType(type_t type); XBT_PUBLIC(void) new_pajeDefineStateType(type_t type); XBT_PUBLIC(void) new_pajeDefineEventType(type_t type); XBT_PUBLIC(void) new_pajeDefineLinkType(type_t type, type_t source, type_t dest); XBT_PUBLIC(void) new_pajeDefineEntityValue (val_t type); XBT_PUBLIC(void) new_pajeCreateContainer (container_t container); XBT_PUBLIC(void) new_pajeDestroyContainer (container_t container); XBT_PUBLIC(void) new_pajeSetVariable (double timestamp, container_t container, type_t type, double value); XBT_PUBLIC(void) new_pajeAddVariable (double timestamp, container_t container, type_t type, double value); XBT_PUBLIC(void) new_pajeSubVariable (double timestamp, container_t container, type_t type, double value); XBT_PUBLIC(void) new_pajeSetState (double timestamp, container_t container, type_t type, val_t value); XBT_PUBLIC(void) new_pajePushState (double timestamp, container_t container, type_t type, val_t value); XBT_PUBLIC(void) new_pajePushStateWithExtra (double timestamp, container_t container, type_t type, val_t value, void* extra); XBT_PUBLIC(void) new_pajePopState (double timestamp, container_t container, type_t type); XBT_PUBLIC(void) new_pajeResetState (double timestamp, container_t container, type_t type); XBT_PUBLIC(void) new_pajeStartLink (double timestamp, container_t container, type_t type, container_t sourceContainer, const char *value, const char *key); XBT_PUBLIC(void) new_pajeStartLinkWithSize (double timestamp, container_t container, type_t type, container_t sourceContainer, const char *value, const char *key, int size); XBT_PUBLIC(void) new_pajeEndLink (double timestamp, container_t container, type_t type, container_t destContainer, const char *value, const char *key); XBT_PUBLIC(void) new_pajeNewEvent (double timestamp, container_t container, type_t type, val_t value); /* from instr_config.c */ XBT_PRIVATE bool TRACE_needs_platform (); XBT_PRIVATE bool TRACE_is_enabled(); XBT_PRIVATE bool TRACE_platform(); XBT_PRIVATE bool TRACE_platform_topology(); XBT_PRIVATE bool TRACE_is_configured(); XBT_PRIVATE bool TRACE_categorized (); XBT_PRIVATE bool TRACE_uncategorized (); XBT_PRIVATE bool TRACE_msg_process_is_enabled(); XBT_PRIVATE bool TRACE_msg_vm_is_enabled(); XBT_PRIVATE bool TRACE_buffer (); XBT_PRIVATE bool TRACE_disable_link(); XBT_PRIVATE bool TRACE_disable_speed(); XBT_PRIVATE bool TRACE_onelink_only (); XBT_PRIVATE bool TRACE_disable_destroy (); XBT_PRIVATE bool TRACE_basic (); XBT_PRIVATE bool TRACE_display_sizes (); XBT_PRIVATE char *TRACE_get_comment (); XBT_PRIVATE char *TRACE_get_comment_file (); XBT_PRIVATE int TRACE_precision (); XBT_PRIVATE char *TRACE_get_filename(); XBT_PRIVATE char *TRACE_get_viva_uncat_conf (); XBT_PRIVATE char *TRACE_get_viva_cat_conf (); XBT_PRIVATE void TRACE_generate_viva_uncat_conf (); XBT_PRIVATE void TRACE_generate_viva_cat_conf (); XBT_PRIVATE void instr_pause_tracing (); XBT_PRIVATE void instr_resume_tracing (); /* Public functions used in SMPI */ XBT_PUBLIC(bool) TRACE_smpi_is_enabled(); XBT_PUBLIC(bool) TRACE_smpi_is_grouped(); XBT_PUBLIC(bool) TRACE_smpi_is_computing(); XBT_PUBLIC(bool) TRACE_smpi_is_sleeping(); XBT_PUBLIC(bool) TRACE_smpi_view_internals(); /* from resource_utilization.c */ XBT_PRIVATE void TRACE_surf_host_set_utilization(const char *resource, const char *category, double value, double now, double delta); XBT_PRIVATE void TRACE_surf_link_set_utilization(const char *resource,const char *category, double value, double now, double delta); XBT_PUBLIC(void) TRACE_surf_resource_utilization_alloc(); /* instr_paje.c */ extern XBT_PRIVATE xbt_dict_t trivaNodeTypes; extern XBT_PRIVATE xbt_dict_t trivaEdgeTypes; XBT_PRIVATE long long int instr_new_paje_id (); XBT_PRIVATE void PJ_container_alloc (); XBT_PRIVATE void PJ_container_release (); XBT_PUBLIC(container_t) PJ_container_new (const char *name, e_container_types kind, container_t father); XBT_PUBLIC(container_t) PJ_container_get (const char *name); XBT_PUBLIC(container_t) PJ_container_get_or_null (const char *name); XBT_PUBLIC(container_t) PJ_container_get_root (); XBT_PUBLIC(void) PJ_container_set_root (container_t root); XBT_PUBLIC(void) PJ_container_free (container_t container); XBT_PUBLIC(void) PJ_container_free_all (void); XBT_PUBLIC(void) PJ_container_remove_from_parent (container_t container); /* instr_paje_types.c */ XBT_PRIVATE void PJ_type_alloc (); XBT_PRIVATE void PJ_type_release (); XBT_PUBLIC(type_t) PJ_type_get_root (); XBT_PRIVATE type_t PJ_type_container_new (const char *name, type_t father); XBT_PRIVATE type_t PJ_type_event_new (const char *name, type_t father); type_t PJ_type_link_new (const char *name, type_t father, type_t source, type_t dest); XBT_PRIVATE XBT_PRIVATE type_t PJ_type_variable_new (const char *name, const char *color, type_t father); XBT_PRIVATE type_t PJ_type_state_new (const char *name, type_t father); XBT_PUBLIC(type_t) PJ_type_get (const char *name, const type_t father); XBT_PUBLIC(type_t) PJ_type_get_or_null (const char *name, type_t father); void PJ_type_free_all (); XBT_PRIVATE XBT_PRIVATE void PJ_type_free (type_t type); /* instr_paje_values.c */ XBT_PUBLIC(val_t) PJ_value_new (const char *name, const char *color, type_t father); XBT_PUBLIC(val_t) PJ_value_get_or_new (const char *name, const char *color, type_t father); XBT_PUBLIC(val_t) PJ_value_get (const char *name, const type_t father); XBT_PRIVATE void PJ_value_free (val_t value); XBT_PRIVATE void print_pajeDefineContainerType(paje_event_t event); XBT_PRIVATE void print_pajeDefineVariableType(paje_event_t event); XBT_PRIVATE void print_pajeDefineStateType(paje_event_t event); XBT_PRIVATE void print_pajeDefineEventType(paje_event_t event); XBT_PRIVATE void print_pajeDefineLinkType(paje_event_t event); XBT_PRIVATE void print_pajeDefineEntityValue (paje_event_t event); XBT_PRIVATE void print_pajeCreateContainer(paje_event_t event); XBT_PRIVATE void print_pajeDestroyContainer(paje_event_t event); XBT_PRIVATE void print_pajeSetVariable(paje_event_t event); XBT_PRIVATE void print_pajeAddVariable(paje_event_t event); XBT_PRIVATE void print_pajeSubVariable(paje_event_t event); XBT_PRIVATE void print_pajeSetState(paje_event_t event); XBT_PRIVATE void print_pajePushState(paje_event_t event); XBT_PRIVATE void print_pajePopState(paje_event_t event); XBT_PRIVATE void print_pajeResetState(paje_event_t event); XBT_PRIVATE void print_pajeStartLink(paje_event_t event); XBT_PRIVATE void print_pajeEndLink(paje_event_t event); XBT_PRIVATE void print_pajeNewEvent (paje_event_t event); XBT_PRIVATE void print_TIPushState(paje_event_t event); XBT_PRIVATE void print_TICreateContainer(paje_event_t event); XBT_PRIVATE void print_TIDestroyContainer(paje_event_t event); XBT_PRIVATE void TRACE_TI_start(); XBT_PRIVATE void TRACE_TI_end(); XBT_PRIVATE void TRACE_TI_init(); XBT_PRIVATE void print_NULL (paje_event_t event); XBT_PRIVATE void TRACE_paje_dump_buffer (int force); XBT_PRIVATE void dump_comment_file (const char *filename); XBT_PRIVATE void dump_comment (const char *comment); typedef struct instr_trace_writer { void (*print_DefineContainerType) (paje_event_t event); void (*print_DefineVariableType)(paje_event_t event); void (*print_DefineStateType)(paje_event_t event); void (*print_DefineEventType)(paje_event_t event); void (*print_DefineLinkType)(paje_event_t event); void (*print_DefineEntityValue)(paje_event_t event); void (*print_CreateContainer)(paje_event_t event); void (*print_DestroyContainer)(paje_event_t event); void (*print_SetVariable)(paje_event_t event); void (*print_AddVariable)(paje_event_t event); void (*print_SubVariable)(paje_event_t event); void (*print_SetState)(paje_event_t event); void (*print_PushState)(paje_event_t event); void (*print_PopState)(paje_event_t event); void (*print_ResetState)(paje_event_t event); void (*print_StartLink)(paje_event_t event); void (*print_EndLink)(paje_event_t event); void (*print_NewEvent) (paje_event_t event); } s_instr_trace_writer_t; struct s_instr_extra_data; typedef struct s_instr_extra_data *instr_extra_data; typedef enum{ TRACING_INIT, TRACING_FINALIZE, TRACING_COMM_SIZE, TRACING_COMM_SPLIT, TRACING_COMM_DUP, TRACING_SEND, TRACING_ISEND, TRACING_SSEND, TRACING_ISSEND, TRACING_RECV, TRACING_IRECV, TRACING_SENDRECV, TRACING_TEST, TRACING_WAIT, TRACING_WAITALL, TRACING_WAITANY, TRACING_BARRIER, TRACING_BCAST, TRACING_REDUCE, TRACING_ALLREDUCE, TRACING_ALLTOALL, TRACING_ALLTOALLV, TRACING_GATHER, TRACING_GATHERV, TRACING_SCATTER, TRACING_SCATTERV, TRACING_ALLGATHER, TRACING_ALLGATHERV, TRACING_REDUCE_SCATTER, TRACING_COMPUTING, TRACING_SLEEPING, TRACING_SCAN, TRACING_EXSCAN } e_caller_type ; typedef struct s_instr_extra_data { e_caller_type type; int send_size; int recv_size; double comp_size; double sleep_duration; int src; int dst; int root; const char* datatype1; const char* datatype2; int * sendcounts; int * recvcounts; int num_processes; } s_instr_extra_data_t; SG_END_DECL() #endif /* INSTR_PRIVATE_H_ */ SimGrid-3.14.159/src/instr/instr_smpi.h000644 001750 001750 00000001010 13031031272 021346 0ustar00mquinsonmquinson000000 000000 #ifndef INSTR_SMPI_H_ #define INSTR_SMPI_H_ #ifdef __cplusplus #include extern "C" { #endif typedef struct smpi_trace_call_location { const char* filename; int linenumber; const char* previous_filename; int previous_linenumber; #ifdef __cplusplus std::string get_composed_key() { return std::string(previous_filename) + ':' + std::to_string(previous_linenumber) + ':' + filename + ':' + std::to_string(linenumber); } #endif } smpi_trace_call_location_t; #ifdef __cplusplus } #endif #endif SimGrid-3.14.159/src/instr/instr_paje_values.cpp000644 001750 001750 00000004114 13031031272 023237 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2012-2015. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include #include "src/instr/instr_private.h" XBT_LOG_NEW_DEFAULT_SUBCATEGORY (instr_paje_values, instr, "Paje tracing event system (values)"); val_t PJ_value_new (const char *name, const char *color, type_t father) { if (name == nullptr || father == nullptr){ THROWF (tracing_error, 0, "can't create a value with a nullptr name (or a nullptr father)"); } val_t ret = xbt_new0(s_val_t, 1); ret->name = xbt_strdup (name); ret->father = father; ret->color = xbt_strdup (color); char str_id[INSTR_DEFAULT_STR_SIZE]; snprintf (str_id, INSTR_DEFAULT_STR_SIZE, "%lld", instr_new_paje_id()); ret->id = xbt_strdup (str_id); xbt_dict_set (father->values, name, ret, nullptr); XBT_DEBUG("new value %s, child of %s", ret->name, ret->father->name); new_pajeDefineEntityValue(ret); return ret; } val_t PJ_value_get_or_new (const char *name, const char *color, type_t father) { val_t ret = 0; try { ret = PJ_value_get(name, father); } catch(xbt_ex& e) { ret = PJ_value_new(name, color, father); } return ret; } val_t PJ_value_get (const char *name, type_t father) { if (name == nullptr || father == nullptr){ THROWF (tracing_error, 0, "can't get a value with a nullptr name (or a nullptr father)"); } if (father->kind == TYPE_VARIABLE) THROWF(tracing_error, 0, "variables can't have different values (%s)", father->name); val_t ret = (val_t)xbt_dict_get_or_null (father->values, name); if (ret == nullptr) { THROWF(tracing_error, 2, "value with name (%s) not found in father type (%s)", name, father->name); } return ret; } void PJ_value_free (val_t value) { XBT_DEBUG("free value %s, child of %s", value->name, value->father->name); xbt_free(((val_t)value)->name); xbt_free(((val_t)value)->color); xbt_free(((val_t)value)->id); xbt_free(value); } SimGrid-3.14.159/src/instr/jedule/000755 001750 001750 00000000000 13031031273 020267 5ustar00mquinsonmquinson000000 000000 SimGrid-3.14.159/src/instr/jedule/jedule.cpp000644 001750 001750 00000002512 13031031273 022243 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "xbt/asserts.h" #include "simgrid/jedule/jedule.hpp" #if HAVE_JEDULE namespace simgrid{ namespace jedule { Jedule::~Jedule() { delete this->root_container; for (auto evt: this->event_set) delete evt; this->event_set.clear(); } void Jedule::addMetaInfo(char *key, char *value) { xbt_assert(key != nullptr); xbt_assert(value != nullptr); this->meta_info.insert({key, value}); } void Jedule::writeOutput(FILE *file) { if (!this->event_set.empty()){ fprintf(file, "\n"); if (!this->meta_info.empty()){ fprintf(file, " \n"); for (auto elm: this->meta_info) fprintf(file, " \n",elm.first,elm.second); fprintf(file, " \n"); } fprintf(file, " \n"); this->root_container->print(file); fprintf(file, " \n"); fprintf(file, " \n"); for (auto event :this->event_set) event->print(file); fprintf(file, " \n"); fprintf(file, "\n"); } } } } #endif SimGrid-3.14.159/src/instr/jedule/jedule_sd_binding.cpp000644 001750 001750 00000003466 13031031273 024434 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "xbt/asserts.h" #include "simgrid/jedule/jedule_sd_binding.h" #include "simgrid/forward.h" #include "../../simdag/simdag_private.hpp" #include "simgrid/jedule/jedule.hpp" #include "simgrid/s4u/NetZone.hpp" #include "simgrid/s4u/engine.hpp" #if HAVE_JEDULE XBT_LOG_NEW_CATEGORY(jedule, "Logging specific to Jedule"); XBT_LOG_NEW_DEFAULT_SUBCATEGORY(jed_sd, jedule, "Logging specific to Jedule SD binding"); jedule_t my_jedule; void jedule_log_sd_event(SD_task_t task) { xbt_assert(task != nullptr); jed_event_t event = new simgrid::jedule::Event(std::string(SD_task_get_name(task)), SD_task_get_start_time(task), SD_task_get_finish_time(task), "SD"); event->addResources(task->allocation); my_jedule->event_set.push_back(event); } void jedule_sd_init() { NetZone_t root_comp = simgrid::s4u::Engine::instance()->netRoot(); XBT_DEBUG("root name %s\n", root_comp->name()); my_jedule = new simgrid::jedule::Jedule(); jed_container_t root_container = new simgrid::jedule::Container(std::string(root_comp->name())); root_container->createHierarchy(root_comp); my_jedule->root_container = root_container; } void jedule_sd_exit(void) { delete my_jedule; } void jedule_sd_dump(const char * filename) { if (my_jedule) { char *fname; if (!filename) { fname = bprintf("%s.jed", xbt_binary_name); } else { fname = xbt_strdup(filename); } FILE *fh = fopen(fname, "w"); my_jedule->writeOutput(fh); fclose(fh); xbt_free(fname); } } #endif SimGrid-3.14.159/src/instr/jedule/jedule_platform.cpp000644 001750 001750 00000014501 13031031273 024150 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/jedule/jedule.hpp" #include "simgrid/jedule/jedule_platform.hpp" #include "simgrid/s4u/NetZone.hpp" #include "xbt/asserts.h" #include "xbt/dynar.h" #include #if HAVE_JEDULE namespace simgrid { namespace jedule { Subset::Subset(int start_idx, int end_idx, Container* parent) : start_idx(start_idx), parent(parent) { nres=end_idx-start_idx+1; } Container::Container(std::string name): name(name) { container_name2container.insert({this->name, this}); } Container::~Container() { if(!this->children.empty()) for (auto child: this->children) delete child; } void Container::addChild(jed_container_t child) { xbt_assert(this != nullptr); xbt_assert(child != nullptr); this->children.push_back(child); child->parent = this; } void Container::addResources(std::vector hosts) { this->is_lowest = 1; this->children.clear(); this->last_id = 0; //FIXME do we need to sort?: xbt_dynar_sort_strings(host_names); for (auto host : hosts) { const char *host_name = sg_host_get_name(host); this->name2id.insert({host_name, this->last_id}); (this->last_id)++; host2_simgrid_parent_container.insert({host_name, this}); this->resource_list.push_back(host); } } void Container::createHierarchy(NetZone_t from_as) { xbt_dict_cursor_t cursor = nullptr; char *key; NetZone_t elem; xbt_dict_t routing_sons = from_as->children(); if (xbt_dict_is_empty(routing_sons)) { // I am no AS // add hosts to jedule platform xbt_dynar_t table = from_as->hosts(); unsigned int dynar_cursor; sg_host_t host; std::vector hosts; xbt_dynar_foreach(table, dynar_cursor, host) { hosts.push_back(host); } this->addResources(hosts); xbt_dynar_free(&table); } else { xbt_dict_foreach(routing_sons, cursor, key, elem) { jed_container_t child_container = new simgrid::jedule::Container(std::string(elem->name())); this->addChild(child_container); child_container->createHierarchy(elem); } } } std::vector Container::getHierarchy() { xbt_assert( this!= nullptr ); if(this->parent != nullptr ) { if(!this->parent->children.empty()) { // we are in the last level return this->parent->getHierarchy(); } else { unsigned int i =0; int child_nb = -1; for (auto child : this->parent->children) { if( child == this) { child_nb = i; break; } i++; } xbt_assert( child_nb > - 1); std::vector heir_list = this->parent->getHierarchy(); heir_list.insert(heir_list.begin(), child_nb); return heir_list; } } else { int top_level = 0; std::vector heir_list = {top_level}; return heir_list; } } std::string Container::getHierarchyAsString() { std::string output(""); std::vector heir_list = this->getHierarchy(); unsigned int length = heir_list.size(); unsigned int i = 0; for (auto id : heir_list) { output += std::to_string(id); if( i != length-1 ) { output += "."; } } return output; } void Container::printResources(FILE * jed_file) { unsigned int i=0; xbt_assert(!this->resource_list.empty()); unsigned int res_nb = this->resource_list.size(); std::string resid = this->getHierarchyAsString(); fprintf(jed_file, " resource_list) { const char * res_name = sg_host_get_name(res); fprintf(jed_file, "%s", res_name); if( i != res_nb-1 ) { fprintf(jed_file, "|"); } i++; } fprintf(jed_file, "\" />\n"); } void Container::print(FILE* jed_file) { xbt_assert( this != nullptr ); fprintf(jed_file, " \n", this->name.c_str()); if( !this->children.empty()){ for (auto child: this->children) { child->print(jed_file); } } else { this->printResources(jed_file); } fprintf(jed_file, " \n"); } } } static void add_subsets_to(std::vector *subset_list, std::vector hostgroup, jed_container_t parent) { // get ids for each host // sort ids // compact ids // create subset for each id group xbt_assert( parent != nullptr ); std::vector id_list; for (auto host_name : hostgroup) { xbt_assert( host_name != nullptr ); jed_container_t parent = host2_simgrid_parent_container.at(host_name); unsigned int id = parent->name2id.at(host_name); id_list.push_back(id); } unsigned int nb_ids = id_list.size(); std::sort(id_list.begin(), id_list.end()); if( nb_ids > 0 ) { int start = 0; int pos = start; for(unsigned int i=0; i 1 ) { subset_list->push_back(new simgrid::jedule::Subset(id_list[start], id_list[pos], parent)); start = i; if( i == nb_ids-1 ) { subset_list->push_back(new simgrid::jedule::Subset(id_list[i], id_list[i], parent)); } } else { if( i == nb_ids-1 ) { subset_list->push_back(new simgrid::jedule::Subset(id_list[start], id_list[i], parent)); } } pos = i; } } } void get_resource_selection_by_hosts(std::vector *subset_list, std::vector *host_list) { xbt_assert( host_list != nullptr ); // for each host name // find parent container // group by parent container std::unordered_map> parent2hostgroup; for (auto host: *host_list) { const char *host_name = sg_host_get_name(host); jed_container_t parent = host2_simgrid_parent_container.at(host_name); xbt_assert( parent != nullptr ); auto host_group = parent2hostgroup.find(parent->name.c_str()); if (host_group == parent2hostgroup.end()) parent2hostgroup.insert({parent->name.c_str(), std::vector(1,host_name)}); else host_group->second.push_back(host_name); } for (auto elm: parent2hostgroup) { jed_container_t parent = container_name2container.at(elm.first); add_subsets_to(subset_list, elm.second, parent); } } #endif SimGrid-3.14.159/src/instr/jedule/jedule_events.cpp000644 001750 001750 00000005350 13031031273 023632 0ustar00mquinsonmquinson000000 000000 /* Copyright (c) 2010-2016. The SimGrid Team. * All rights reserved. */ /* This program is free software; you can redistribute it and/or modify it * under the terms of the license (GNU LGPL) which comes with this package. */ #include "simgrid/jedule/jedule_events.hpp" #include "simgrid/jedule/jedule.hpp" #include "simgrid/jedule/jedule_platform.hpp" #include "simgrid/s4u/NetZone.hpp" #include "xbt/asserts.h" #if HAVE_JEDULE namespace simgrid{ namespace jedule{ Event::Event(std::string name, double start_time, double end_time, std::string type) : name(name), start_time(start_time), end_time(end_time), type(type) { this->resource_subsets = new std::vector(); } Event::~Event() { if (!this->resource_subsets->empty()){ for (auto subset: *this->resource_subsets) delete subset; delete this->resource_subsets; } } void Event::addResources(std::vector *host_selection) { get_resource_selection_by_hosts(this->resource_subsets, host_selection); } void Event::addCharacteristic(char *characteristic) { xbt_assert( characteristic != nullptr ); this->characteristics_list.push_back(characteristic); } void Event::addInfo(char* key, char *value) { xbt_assert((key != nullptr) && value != nullptr); this->info_map.insert({key, value}); } void Event::print(FILE *jed_file) { fprintf(jed_file, " \n"); fprintf(jed_file, " \n", this->name.c_str()); fprintf(jed_file, " \n", this->start_time); fprintf(jed_file, " \n", this->end_time); fprintf(jed_file, " \n", this->type.c_str()); xbt_assert(!this->resource_subsets-> empty()); fprintf(jed_file, " \n"); for (auto subset: *this->resource_subsets) { fprintf(jed_file, "