Heavily based on Sun Grid Engine package (parallel/sge).
Open Grid Scheduler/Grid Engine is a free and open-source
batch-queuing system for distributed resource management.
OGS/GE is based on Sun Grid Engine, and maintained by the same
group of external (i.e. non-Sun) developers who started
contributing code to Sun Grid Engine since 2001.
The Portable Hardware Locality (hwloc) software package provides
a portable abstraction (across OS, versions, architectures, ...)
of the hierarchical topology of modern architectures, including
NUMA memory nodes, sockets, shared caches, cores and
simultaneous multithreading. It also gathers various system
attributes such as cache and memory information as well as the
locality of I/O devices such as network interfaces, InfiniBand
HCAs or GPUs. It primarily aims at helping applications with
gathering information about modern computing hardware so as to
exploit it accordingly and efficiently.
The Son of Grid Engine is a community project to continue Sun Grid Engine.
Notable changes in Son of Grid Engine
-------------------------------------
Version 8.0.0d
--------------
* Bug fixes
* Man and fixes
* Fix building with older gcc versions
* Provide load average in qstat XML output [#446, #454]
* Partially back out Univa change which broke classic spooling
* Fix -terse in sge_request [#777]
* Other changes (possibly-incompatible)
* Message fixes
Version 8.0.0c
--------------
* Bug fixes
* Man and other documentation fixes
* Build/installation fixes (particularly for Red Hat 6 and Linux 3)
* Fix group ids for submitted jobs [U]
* Fix default value of boolean with JSV [U]
* Windows fixes for helper crashes and Vista GUI jobs [U]
* Ensure parallel jobs are dispatched to the least loaded host [U]
* Correct ownership of qsub -pty output file; was owned by admin user [U]
* Fix format of Windows loadcheck.exe output [U]
* Read from stderr even if stdout is already closed in IJS [U]
* Fix PDC_INTERVAL=NEVER execd parameter [U]
* Fix accounting information for Windows GUI jobs [U]
* Increase default MAX_DYN_EC qmaster param [U]
* Fix qsub -sync y error message and enforce MAX_DYN_EC correctly [U]
* Fix job validation (-w e) behaviour [#716] [U]
* Fix qrsh input redirection [U]
* Avoid warning when submitting a qrsh job [U]
* Print start time in qstat -j -xml output [U]
* Don't raise an error changing resource request on waiting job [#806]
* Don't exit 0 on error with qconf -secl or -sep
* Include string.h in drmaa.h [#712]
* Fix process-scheduler-log with host aliases
* Enhancements
* Base qmake and qtcsh on the current gmake and tcsh source [#289,
#504, #832]
* Support "-binding linear" and "-binding linear:slots"
* Use the hwloc library for all topology information and core
binding, supporting more operating systems (now: AIX, Darwin,
FreeBSD, GNU/Linux, HPUX, MS Windows, OSF/1, Solaris), and more
hardware types (specifically AMD Magny Cours and similar)
* Add task number to execd "exceeds job ... limit"
* Other changes (possibly-incompatible)
* Modify default paths in build files and elsewhere [U]
* Assorted message fixes
* In RPMs, move qsched to qmaster package, and separate drmaa4ruby
* Default to newijs in load_sge_config.sh
* Default to sh, not csh for configured shell
Version 8.0.0b
--------------
* Bug fixes
* Build/installation fixes [including #424, #1349] [(U)]
* Fix execd init script [#1348]
* Man and other documentation fixes [including #614, #764] [(U)]
* Fix contents of admin mail properly [#1307, #1345]
* Fix qalter messages for -tc
* Fix build with -DSGE_PQS_API
* Fix group ids for submitted jobs [U]
* Enhancements
* Update qsched and add man page
* Other changes (possibly-incompatible)
* Avoid the use of /bin/ksh [#1306]
* Change installation defaults to classic spooling, not adding
shadow hosts, and not JMX. [(U)]
Version 8.0.0a
--------------
This is roughly a superset of Univa's 8.0.0 (the V800_TAG from
https://github.com/gridengine/gridengine), with thanks for that.
Changes made there which haven't been included in this version: PLPA
source not removed; some different build/installation defaults
(e.g. for JMX); Univa/UGE "branding" (partly because trademark status
is unknown); authuser not removed (for SDM and testing use),
* Bug fixes
* Many man and other documentation fixes [including #790, #776,
#769, #733, #610, #587, #581, #459, #456, #439, #255, #1288, #797,
#1271, #773] [(U)]
* Some program message fixes [(U)]
* Various build and installation fixes [including #761, #709, #656,
#616, #546, #536, #521, #491, #438, #414, #411, #383, #381, #138,
#455, #344, #438, #1311, #1272, #1273] [(U)]
* Ask for keystore password twice on installation
* Fix qmaster crashes with tightly integrated parallel jobs or
un-discoverable qinstance [#789] [U]
* Report 0 cores and sockets on unsupported Solaris hosts [U]
* Fix malloc hooks which caused crashes, particularly with SuSE 11
[#792, #748, #749] [U]
* Verify the pe task start user in execd in non-CSP mode [U]
* Fix binding parameters parsing [U]
* Fix JSV logging with multiple users submitting jobs on same submit
host [U]
* Fix unresponsive qmaster when modifying the global configuration
in a huge cluster [U]
* Speed up finishing tightly integrated jobs [U]
* Check consistency of JSV binding information properly [U]
* Fix broken project spooling, which caused loss of project when
restarting master when using core binding [U]
* Fix slotwise preemption failure to unsuspend one job per host [#775] [U]
* Fix problems retrieving passwd and group information with large
responses [#1295] [(U)]
* Fix JSV changing default of boolean [U]
* Fix ENABLE_RESCHEDULE_SLAVE=1 [U]
* Allow comma in CMDNAME with Perl JSV scripts [#803]
* Don't put queue into error state when supplementary group id
cannot be set [#185] [U]
* Don't convert LF to CRLF with qrsh -pty [U]
* Fix qconf segfault on bad subordination string [U]
* Fix group ids of submitted jobs [U]
* Disallow -masterq with serial jobs [#155] [U]
* Fix 100% CPU use by shepherd of qsh [U]
* Removed unnecessary binding warning on job starts [U]
* Fix qconf error reports when tmp directory has 755 permissions [U]
* Fix suspending of remote process on qrsh -pty yes <cmd> on Solaris [U]
* Fix starting jobs after global host changed [U]
* Reject invalid load_formula value [U]
* Fix handling of implicitly-requested exclusive resources [U]
* Fix execd vmem reporting on 64-bit Linux [U]
* Fix startup of execd on Windows Vista [U]
* Set xterm's path more appropriately on GNU/Linux [#557]
* Fix generation of admin email from failed jobs [#1307]
* Fix some ill-formed output from qstat -xml [#314]
* Fix handling of multi-line environment variables propagated to
shepherd [#395]
* Fix example MPI PE templates
* Fix bad quoting in JSV sh library
* Fix checking of consumables for parallel jobs across multiple hosts [U]
* Enhancements
* Additional and clarified documentation
* PAM modules for ssh tight integration and access control for
interactive jobs
* Initial core binding support for Solaris/SPARC64 [U]
* Some efficiency improvements and memory leaks fixed [U]
* Ports to S/390 and PARISC GNU/Linux [U]
* New complex m_thread [U]
* Show topology by default in qhost [U]
* qsub -pty switch [#704] [U]
* Improved qmon graphics [#530] [(U)]
* Include bash in default shell list [U]
* A JSV that rejects all jobs [U]
* Files for Scali-MPI
* Ruby DRMAA implementation
* Enable easy building against shared system libraries and use
system openssl and bdb binaries
* New scripts: "qsched" reports resource reservations; "status"
wraps qstat; enable/disable submission; node-selection (idle etc.)
* Restart argument for daemon init scripts
* Improved efficiency of shell JSV if used with bash
* Core dumps from crashing daemons enabled under Linux [U]
* Example host_aliases file [#154]
* Spec file for RPM packaging [#820]
* Other changes (possibly-incompatible)
* Show core binding by default in qstat, qhost (use -ncb for
compatibility) [U]
* Removed Berkeley DB RPC support (recently dropped by BDB) [U]
* Changed position in pending job list for user-rescheduled jobs
(exit99, qmod -rj) and OLD_RESCHEDULE_BEHAVIOR,
OLD_RESCHEDULE_BEHAVIOR_ARRAY_JOB parameters [U]
* Unified GNU/Linux arch strings (lx-*, from lx24-* and lx26-*) [U]
* Default to enabling core binding on GNU/Linux [U]
* Removed Sun service tags support [U]
* Removed obsolete SunHPCT5 files
SLURM is an open-source resource manager designed for Linux
clusters of all sizes. It provides three key functions. First it
allocates exclusive and/or non-exclusive access to resources
(computer nodes) to users for some duration of time so they can
perform work. Second, it provides a framework for starting,
executing, and monitoring work (typically a parallel job) on a
set of allocated nodes. Finally, it arbitrates contention for
resources by managing a queue of pending work.
===============================================================================
Changes in 1.4.1
===============================================================================
# OVERALL: Several improvements to the ARMCI API implementation
within MPICH2.
# Build system: Added beta support for DESTDIR while installing
MPICH2.
# PM/PMI: Upgrade hwloc to 1.2.1rc2.
# PM/PMI: Initial support for the PBS launcher.
# Several other minor bug fixes, memory leak fixes, and code cleanup.
A full list of changes is available using:
svn log -r8675:HEAD https://svn.mcs.anl.gov/repos/mpi/mpich2/tags/release/mpich2-1.4.1
... or at the following link:
https://trac.mcs.anl.gov/projects/mpich2/log/mpich2/tags/release/mpich2-1.4.1?action=follow_copy&rev=HEAD&stop_rev=8675&mode=follow_copy
===============================================================================
Changes in 1.4
===============================================================================
# OVERALL: Improvements to fault tolerance for collective
operations. Thanks to Rui Wang @ ICT for reporting several of these
issues.
# OVERALL: Improvements to the universe size detection. Thanks to
Yauheni Zelenko for reporting this issue.
# OVERALL: Bug fixes for Fortran attributes on some systems. Thanks
to Nicolai Stange for reporting this issue.
# OVERALL: Added new ARMCI API implementation (experimental).
# OVERALL: Added new MPIX_Group_comm_create function to allow
non-collective creation of sub-communicators.
# FORTRAN: Bug fixes in the MPI_DIST_GRAPH_ Fortran bindings.
# PM/PMI: Support for a manual "none" launcher in Hydra to allow for
higher-level tools to be built on top of Hydra. Thanks to Justin
Wozniak for reporting this issue, for providing several patches for
the fix, and testing it.
# PM/PMI: Bug fixes in Hydra to handle non-uniform layouts of hosts
better. Thanks to the MVAPICH group at OSU for reporting this issue
and testing it.
# PM/PMI: Bug fixes in Hydra to handle cases where only a subset of
the available launchers or resource managers are compiled
in. Thanks to Satish Balay @ Argonne for reporting this issue.
# PM/PMI: Support for a different username to be provided for each
host; this only works for launchers that support this (such as
SSH).
# PM/PMI: Bug fixes for using Hydra on AIX machines. Thanks to
Kitrick Sheets @ NCSA for reporting this issue and providing the
first draft of the patch.
# PM/PMI: Bug fixes in memory allocation/management for environment
variables that was showing up on older platforms. Thanks to Steven
Sutphen for reporting the issue and providing detailed analysis to
track down the bug.
# PM/PMI: Added support for providing a configuration file to pick
the default options for Hydra. Thanks to Saurabh T. for reporting
the issues with the current implementation and working with us to
improve this option.
# PM/PMI: Improvements to the error code returned by Hydra.
# PM/PMI: Bug fixes for handling "=" in environment variable values in
hydra.
# PM/PMI: Upgrade the hwloc version to 1.2.
# COLLECTIVES: Performance and memory usage improvements for MPI_Bcast
in certain cases.
# VALGRIND: Fix incorrect Valgrind client request usage when MPICH2 is
built for memory debugging.
# BUILD SYSTEM: "--enable-fast" and "--disable-error-checking" are once
again valid simultaneous options to configure.
# TEST SUITE: Several new tests for MPI RMA operations.
# Several other minor bug fixes, memory leak fixes, and code cleanup.
A full list of changes is available using:
svn log -r7838:HEAD https://svn.mcs.anl.gov/repos/mpi/mpich2/tags/release/mpich2-1.4
... or at the following link:
https://trac.mcs.anl.gov/projects/mpich2/log/mpich2/tags/release/mpich2-1.4?action=follow_copy&rev=HEAD&stop_rev=7838&mode=follow_copy
===============================================================================
Changes in 1.3.2
===============================================================================
# OVERALL: MPICH2 now recognizes the OSX mach_absolute_time as a
native timer type.
# OVERALL: Performance improvements to MPI_Comm_split on large
systems.
# OVERALL: Several improvements to error returns capabilities in the
presence of faults.
# PM/PMI: Several fixes and improvements to Hydra's process binding
capability.
# PM/PMI: Upgrade the hwloc version to 1.1.1.
# PM/PMI: Allow users to sort node lists allocated by resource
managers in Hydra.
# PM/PMI: Improvements to signal handling. Now Hydra respects Ctrl-Z
signals and passes on the signal to the application.
# PM/PMI: Improvements to STDOUT/STDERR handling including improved
support for rank prepending on output. Improvements to STDIN
handling for applications being run in the background.
# PM/PMI: Split the bootstrap servers into "launchers" and "resource
managers", allowing the user to pick a different resource manager
from the launcher. For example, the user can now pick the "SLURM"
resource manager and "SSH" as the launcher.
# PM/PMI: The MPD process manager is deprecated.
# PM/PMI: The PLPA process binding library support is deprecated.
# WINDOWS: Adding support for gfortran and 64-bit gcc libs.
# Several other minor bug fixes, memory leak fixes, and code cleanup.
A full list of changes is available using:
svn log -r7457:HEAD https://svn.mcs.anl.gov/repos/mpi/mpich2/tags/release/mpich2-1.3.2
... or at the following link:
https://trac.mcs.anl.gov/projects/mpich2/log/mpich2/tags/release/mpich2-1.3.2?action=follow_copy&rev=HEAD&stop_rev=7457&mode=follow_copy
Intel(r) Threading Building Blocks (Intel TBB) offers a rich and
complete approach to expressing parallelism in a C++ program.
It is a library that helps you take advantage of multi-core
processor performance without having to be a threading expert.
Intel TBB is not just a threads-replacement library. It represents
a higher-level, task-based parallelism that abstracts platform
details and threading mechanisms for scalability and performance.
Changes in v1.5.4 as compared to v1.5.3:
- Add support for the (as yet unreleased) Mellanox MXM transport.
- Add support for dynamic service levels (SLs) in the openib BTL.
- Fixed C++ bindings cosmetic/warnings issue with
MPI::Comm::NULL_COPY_FN and MPI::Comm::NULL_DELETE_FN. Thanks to
Julio Hoffimann for identifying the issues.
- Also allow the word "slots" in rankfiles (i.e., not just "slot").
(** also to appear in 1.4.4)
- Add Mellanox ConnectX 3 device IDs to the openib BTL defaults.
(** also to appear in 1.4.4)
- Various FCA updates.
- Fix 32 bit SIGBUS errors on Solaris SPARC platforms.
- Add missing ARM assembly code files.
- Update to allow more than 128 entries in an appfile.
(** also to appear in 1.4.4)
- Various VT updates and bug fixes.
- Update description of btl_openib_cq_size to be more accurate.
(** also to appear in 1.4.4)
- Various assembly "clobber" fixes.
- Fix a hang in carto selection in obscure situations.
- Guard the inclusion of execinfo.h since not all platforms have it. Thanks
to Aleksej Saushev for identifying this issue.
(** also to appear in 1.4.4)
- Support Solaris legacy munmap prototype changes.
(** also to appear in 1.4.4)
- Updated to Automake 1.11.1 per
http://www.open-mpi.org/community/lists/devel/2011/07/9492.php.
- Fix compilation of LSF support.
- Update MPI_Comm_spawn_multiple.3 man page to reflect what it
actually does.
- Fix for possible corruption of the environment. Thanks to Peter
Thompson for the suggestion. (** also to appear in 1.4.4)
- Enable use of PSM on direct-launch SLURM jobs.
- Update paffinity hwloc to v1.2, and to fix minor bugs affinity
assignment bugs on PPC64/Linux platforms.
- Let the openib BTL auto-detect its bandwidth.
- Support new MPI-2.2 datatypes.
- Updates to support more datatypes in MPI one-sided communication.
- Fix recursive locking bug when MPI-IO was used with
MPI_THREAD_MULTIPLE. (** also to appear in 1.4.4)
- Fix mpirun handling of prefix conflicts.
- Ensure mpirun's --xterm options leaves sessions attached.
(** also to appear in 1.4.4)
- Fixed type of sendcounts and displs in the "use mpi" F90 module.
ABI is preserved, but applications may well be broken. See the
README for more details. Thanks to Stanislav Sazykin for
identifying the issue. (** also to appear in 1.4.4)
- Fix indexed datatype leaks. Thanks to Pascal Deveze for supplying
the initial patch. (** also to appear in 1.4.4)
- Fix debugger mapping when mpirun's -npernode option is used.
- Fixed support for configure's --disable-dlopen option when used with
"make distclean".
- Fix segv associated with MPI_Comm_create with MPI_GROUP_EMPTY.
Thanks to Dominik Goeddeke for finding this.
(** also to appear in 1.4.4)
- Improved LoadLeveler ORTE support.
- Add new WindVerbs BTL plugin, supporting native OpenFabrics verbs on
Windows (the "wv" BTL).
- Add new btl_openib_gid_index MCA parameter to allow selecting which
GID to use on an OpenFabrics device's GID table.
- Add support for PCI relaxed ordering in the OpenFabrics BTL (when
available).
- Update rsh logic to allow correct SGE operation.
- Ensure that the mca_paffinity_alone MCA parameter only appears once
in the ompi_info output. Thanks to Gus Correa for identifying the
issue.
- Fixed return codes from MPI_PROBE and MPI_IPROBE.
(** also to appear in 1.4.4)
- Remove --enable-progress-thread configure option; it doesn't work on
the v1.5 branch. Rename --enable-mpi-threads to
--enable-mpi-thread-multiple. Add new --enable-opal-multi-threads
option.
- Updates for Intel Fortran compiler version 12.
- Remove bproc support. Farewell bproc!
- If something goes wrong during MPI_INIT, fix the error
message to say that it's illegal to invoke MPI_INIT before
MPI_INIT.
more machines. A job is typically a single command or a small script that
has to be run for each of the lines in the input. The typical input is a
list of files, a list of hosts, a list of users, or a list of tables.
1.5.3
-----
- Add missing "affinity" MPI extension (i.e., the OMPI_Affinity_str()
API) that was accidentally left out of the 1.5.2 release.
1.5.2
-----
- Replaced all custom topology / affinity code with initial support
for hwloc v1.1.1 (PLPA has been removed -- long live hwloc!). Note
that hwloc is bundled with Open MPI, but an external hwloc can be
used, if desired. See README for more details.
- Many CMake updates for Windows builds.
- Updated opal_cr_thread_sleep_wait MCA param default value to make it
less aggressive.
- Updated debugger support to allow Totalview attaching from jobs
launched directly via srun (not mpirun). Thanks to Nikolay Piskun
for the patch.
- Added more FTB/CIFTS support.
- Fixed compile error with the PGI compiler.
- Portability fixes to allow the openib BTL to run on the Solaris
verbs stack.
- Fixed multi-token command-line issues when using the mpirun
--debug switch. For example:
mpirun --debug -np 2 a.out "foo bar"
Thanks to Gabriele Fatigati for reporting the issue.
- Added ARM support.
- Added the MPI_ROOT environment variable in the Open MPI Linux SRPM
for customers who use the BPS and LSF batch managers.
- Updated ROMIO from MPICH v1.3.1 (plus one additional patch).
- Fixed some deprecated MPI API function notification messages.
- Added new "bfo" PML that provides failover on OpenFabrics networks.
- Fixed some buffer memcheck issues in MPI_*_init.
- Added Solaris-specific chip detection and performance improvements.
- Fix some compile errors on Solaris.
- Updated the "rmcast" framework with bug fixes, new functionality.
- Updated the Voltaire FCA component with bug fixes, new
functionality. Support for FCA version 2.1.
- Fix gcc 4.4.x and 4.5.x over-aggressive warning notifications on
possibly freeing stack variables. Thanks to the Gentoo packagers
for reporting the issue.
- Make the openib component be verbose when it disqualifies itself due
to MPI_THREAD_MULTIPLE.
- Minor man page fixes.
- Various checkpoint / restart fixes.
- Fix race condition in the one-sided unlock code. Thanks to
Guillaume Thouvenin for finding the issue.
- Improve help message aggregation.
- Add OMPI_Affinity_str() optional user-level API function (i.e., the
"affinity" MPI extension). See README for more details.
- Added btl_tcp_if_seq MCA parameter to select a different ethernet
interface for each MPI process on a node. This parameter is only
useful when used with virtual ethernet interfaces on a single
network card (e.g., when using virtual interfaces give dedicated
hardware resources on the NIC to each process).
- Changed behavior of mpirun to terminate if it receives 10 (or more)
SIGPIPEs.
- Fixed oversubscription detection.
- Added new mtl_mx_board and mtl_mx_endpoint MCA parameters.
- Added ummunotify support for OpenFabrics-based transports. See the
README for more details.
Changes in 1.3.1
# OVERALL: MPICH2 is now fully compliant with the CIFTS FTB standard
MPI events (based on the draft standard).
# OVERALL: Major improvements to RMA performance for long lists of
RMA operations.
# OVERALL: Performance improvements for Group_translate_ranks.
# COLLECTIVES: Collective algorithm selection thresholds can now be controlled
at runtime via environment variables.
# ROMIO: PVFS error codes are now mapped to MPI error codes.
# Several other minor bug fixes, memory leak fixes, and code cleanup.
A full list of changes is available using:
svn log -r7350:HEAD https://svn.mcs.anl.gov/repos/mpi/mpich2/tags/release/mpich2-1.3.1
... or at the following link:
https://trac.mcs.anl.gov/projects/mpich2/log/mpich2/tags/release/mpich2-1.3.1?action=follow_copy&rev=HEAD&stop_rev=7350&mode=follow_copy
Changes in 1.3
# OVERALL: Initial support for fine-grained threading in
ch3:nemesis:tcp.
# OVERALL: Support for Asynchronous Communication Progress.
# OVERALL: The ssm and shm channels have been removed.
# OVERALL: Checkpoint/restart support using BLCR.
# OVERALL: Improved tolerance to process and communication failures
when error handler is set to MPI_ERRORS_RETURN. If a communication
operation fails (e.g., due to a process failure) MPICH2 will return
an error, and further communication to that process is not
possible. However, communication with other processes will still
proceed normally. Note, however, that the behavior collective
operations on communicators containing the failed process is
undefined, and may give incorrect results or hang some processes.
# OVERALL: Experimental support for inter-library dependencies.
# PM/PMI: Hydra is now the default process management framework
replacing MPD.
# PM/PMI: Added dynamic process support for Hydra.
# PM/PMI: Added support for LSF, SGE and POE in Hydra.
# PM/PMI: Added support for CPU and memory/cache topology aware
process-core binding.
# DEBUGGER: Improved support and bug fixes in the Totalview support.
# Build system: Replaced F90/F90FLAGS by FC/FCFLAGS. F90/F90FLAGS are
not longer supported in the configure.
# Multi-compiler support: On systems where C compiler that is used to
build mpich2 libraries supports multiple weak symbols and multiple aliases,
the Fortran binding built in the mpich2 libraries can handle different
Fortran compilers (than the one used to build mpich2). Details in README.
# Several other minor bug fixes, memory leak fixes, and code cleanup.
A full list of changes is available using:
svn log -r5762:HEAD https://svn.mcs.anl.gov/repos/mpi/mpich2/tags/release/mpich2-1.3
... or at the following link:
https://trac.mcs.anl.gov/projects/mpich2/log/mpich2/tags/release/mpich2-1.3?action=follow_copy&rev=HEAD&stop_rev=5762&mode=follow_copy
