This release adds several performance features for the Hydra process manager, support for communicator-specific tuning of eager/rendezvous thresholds, and fixes several bugs present in 3.0.3. Please use this release instead of 3.0.3.
A new stable release of MPICH, 3.0.3, is now available for download. This release adds several performance features for MPI-RMA and fixes several bugs present in 3.0.2. Please use this release instead of 3.0.2.
-------------
* New operating system backends
+ Add BlueGene/Q compute node kernel (CNK) support. See the FAQ in the
documentation for details. Thanks to Jeff Hammond, Christopher Samuel
and Erik Schnetter for their help.
+ Add NetBSD support, thanks to Aleksej Saushev.
* New I/O device discovery
+ Add co-processor OS devices such as "mic0" for Intel Xeon Phi (MIC)
on Linux. Thanks to Jerome Vienne for helping.
+ Add co-processor OS devices such as "cuda0" for NVIDIA CUDA-capable GPUs.
+ Add co-processor OS devices such as "opencl0d0" for OpenCL GPU devices
on the AMD OpenCL implementation.
+ Add GPU OS devices such as ":0.0" for NVIDIA X11 displays.
+ Add GPU OS devices such as "nvml0" for NVIDIA GPUs.
Thanks to Marwan Abdellah and Stefan Eilemann for helping.
These new OS devices have some string info attributes such as CoProcType,
GPUModel, etc. to better identify them.
See the I/O Devices and Attributes documentation sections for details.
* New components
+ Add the "opencl", "cuda", "nvml" and "gl" components for I/O device
discovery.
+ "nvml" also improves the discovery of NVIDIA GPU PCIe link speed.
All of these new components may be built as plugins. They may also be
disabled entirely by passing --disable-opencl/cuda/nvml/gl to configure.
See the I/O Devices, Components and Plugins, and FAQ documentation
sections for details.
* API
+ Add hwloc_topology_get_flags().
+ Add hwloc/plugins.h for building external plugins.
See the Adding new discovery components and plugins section.
* Interoperability
+ Add hwloc/opencl.h, hwloc/nvml.h, hwloc/gl.h and hwloc/intel-mic.h
to retrieve the locality of OS devices that correspond to AMD OpenCL
GPU devices or indexes, to NVML devices or indexes, to NVIDIA X11
displays, or to Intel Xeon Phi (MIC) device indexes.
+ Add new helpers in hwloc/cuda.h and hwloc/cudart.h to convert
between CUDA devices or indexes and hwloc OS devices.
+ Add hwloc_ibv_get_device_osdev() and clarify the requirements
of the OpenFabrics Verbs helpers in hwloc/openfabrics-verbs.h.
* Tools
+ hwloc-info is not only a synonym of lstopo -s anymore, it also
dumps information about objects given on the command-line.
* Documentation
+ Add a section "Existing components and plugins".
+ Add a list of common OS devices in section "Software devices".
+ Add a new FAQ entry "Why is lstopo slow?" about lstopo slowness
issues because of GPUs.
+ Clarify the documentation of inline helpers in hwloc/myriexpress.h
and hwloc/openfabrics-verbs.h.
* Misc
+ Improve cache detection on AIX.
+ The HWLOC_COMPONENTS variable now excludes the components whose
names are prefixed with '-'.
+ lstopo --ignore PU now works when displaying the topology in
graphical and textual mode (not when exporting to XML).
+ The `libpci' component is now called `pci' but the old name is still
accepted in the HWLOC_COMPONENTS variable for backward compatibility.
File too long (should be no more than 24 lines).
Line too long (should be no more than 80 characters).
Trailing empty lines.
Trailing white-space.
Trucated the long files as best as possible while preserving the most info
contained in them.
===============================================================================
Changes in 3.0
===============================================================================
# MPI-3: All MPI-3 features are now implemented and the MPI_VERSION
bumped up to 3.0.
# OVERALL: Added support for ARM-v7 native atomics
# MPE: MPE is now separated out of MPICH and can be downloaded/used
as a separate package.
# PM/PMI: Upgraded to hwloc-1.6
# Several other minor bug fixes, memory leak fixes, and code cleanup.
A full list of changes is available using:
svn log -r10344:HEAD https://svn.mcs.anl.gov/repos/mpi/mpich2/tags/release/mpich-3.0
... or at the following link:
https://trac.mcs.anl.gov/projects/mpich2/log/mpich2/tags/release/mpich-3.0?action=follow_copy&rev=HEAD&stop_rev=10344&mode=follow_copy
===============================================================================
Changes in 1.5
===============================================================================
# OVERALL: Nemesis now supports an "--enable-yield=..." configure
option for better performance/behavior when oversubscribing
processes to cores. Some form of this option is enabled by default
on Linux, Darwin, and systems that support sched_yield().
# OVERALL: Added support for Intel Many Integrated Core (MIC)
architecture: shared memory, TCP/IP, and SCIF based communication.
# OVERALL: Added support for IBM BG/Q architecture. Thanks to IBM
for the contribution.
# MPI-3: const support has been added to mpi.h, although it is
disabled by default. It can be enabled on a per-translation unit
basis with "#define MPICH2_CONST const".
# MPI-3: Added support for MPIX_Type_create_hindexed_block.
# MPI-3: The new MPI-3 nonblocking collective functions are now
available as "MPIX_" functions (e.g., "MPIX_Ibcast").
# MPI-3: The new MPI-3 neighborhood collective routines are now available as
"MPIX_" functions (e.g., "MPIX_Neighbor_allgather").
# MPI-3: The new MPI-3 MPI_Comm_split_type function is now available
as an "MPIX_" function.
# MPI-3: The new MPI-3 tools interface is now available as "MPIX_T_"
functions. This is a beta implementation right now with several
limitations, including no support for multithreading. Several
performance variables related to CH3's message matching are exposed
through this interface.
# MPI-3: The new MPI-3 matched probe functionality is supported via
the new routines MPIX_Mprobe, MPIX_Improbe, MPIX_Mrecv, and
MPIX_Imrecv.
# MPI-3: The new MPI-3 nonblocking communicator duplication routine,
MPIX_Comm_idup, is now supported. It will only work for
single-threaded programs at this time.
# MPI-3: MPIX_Comm_reenable_anysource support
# MPI-3: Native MPIX_Comm_create_group support (updated version of
the prior MPIX_Group_comm_create routine).
# MPI-3: MPI_Intercomm_create's internal communication no longer interferes
with point-to-point communication, even if point-to-point operations on the
parent communicator use the same tag or MPI_ANY_TAG.
# MPI-3: Eliminated the possibility of interference between
MPI_Intercomm_create and point-to-point messaging operations.
# Build system: Completely revamped build system to rely fully on
autotools. Parallel builds ("make -j8" and similar) are now supported.
# Build system: rename "./maint/updatefiles" --> "./autogen.sh" and
"configure.in" --> "configure.ac"
# JUMPSHOT: Improvements to Jumpshot to handle thousands of
timelines, including performance improvements to slog2 in such
cases.
# JUMPSHOT: Added navigation support to locate chosen drawable's ends
when viewport has been scrolled far from the drawable.
# PM/PMI: Added support for memory binding policies.
# PM/PMI: Various improvements to the process binding support in
Hydra. Several new pre-defined binding options are provided.
# PM/PMI: Upgraded to hwloc-1.5
# PM/PMI: Several improvements to PBS support to natively use the PBS
launcher.
# Several other minor bug fixes, memory leak fixes, and code cleanup.
A full list of changes is available using:
svn log -r8478:HEAD https://svn.mcs.anl.gov/repos/mpi/mpich2/tags/release/mpich2-1.5
... or at the following link:
https://trac.mcs.anl.gov/projects/mpich2/log/mpich2/tags/release/mpich2-1.5?action=follow_copy&rev=HEAD&stop_rev=8478&mode=follow_copy
Inter-process locking may be broken on NetBSD under some conditions.
(This didn't get worse since previous package.)
Field tests (benchmarks/skampi) don't reveal that the relevant
code path is invoked.
Major Changes in OpenPA v1.0.4
* native ARM (v7 and higher) support
* numerous memory barrier placement improvements in queue and test code
* x86 memory barrier improvements, including for Intel MIC
* numerous build system improvements
Major Changes in OpenPA v1.0.3
* Libtool shared library support for OPA, avoiding shared/static mixed
linking problems on some platforms.
* The build system should now work more portably with broken/exotic shells.
* pkg-config support
* Emulated atomics can now be detected by the presence of an OPA_EXPLICIT_EMULATION
#define in opa_config.h.
* PPC types are now correctly aligned to 8 bytes instead of 16 bytes.
* many more tests for "make check", fixed missing memory barriers in one test
paexec:
- fixed: NULL dereference when the first line given on input is
empty. Thanks to Sergey Revyako for bug report!
- fixed: entire command passed tp ssh-like transport should be
shquoted. In particular this fixes -x that didn't work with -t.
Thanks to Sergey Revyako for bug report!
- 'paexec -g' accepts empty strings as tasks.
- More regression tests were added
paexec_reorder.1:
- Mistype fix
Here is a list of changes in v1.6.4 as compared to v1.6.3:
- Fix Cygwin shared memory and debugger plugin support. Thanks to
Marco Atzeri for reporting the issue and providing initial patches.
- Fix to obtaining the correct available nodes when a rankfile is
providing the allocation. Thanks to Siegmar Gross for reporting the
problem.
- Fix process binding issue on Solaris. Thanks to Siegmar Gross for
reporting the problem.
- Updates for MXM 2.0.
- Major VT update to 5.14.2.3.
- Fixed F77 constants for Cygwin/Cmake build.
- Fix a linker error when configuring --without-hwloc.
- Automatically provide compiler flags that compile properly on some
types of ARM systems.
- Fix slot_list behavior when multiple sockets are specified. Thanks
to Siegmar Gross for reporting the problem.
- Fixed memory leak in one-sided operations. Thanks to Victor
Vysotskiy for letting us know about this one.
- Added performance improvements to the OpenIB (OpenFabrics) BTL.
- Improved error message when process affinity fails.
- Fixed MPI_MINLOC on man pages for MPI_REDUCE(_LOCAL). Thanks to Jed
Brown for noticing the problem and supplying a fix.
- Made malloc hooks more friendly to IO interprosers. Thanks to the
bug report and suggested fix from Darshan maintainer Phil Carns.
- Restored ability to direct launch under SLURM without PMI support.
- Fixed MPI datatype issues on OpenBSD.
- Major VT update to 5.14.2.3.
- Support FCA v3.0+.
- Fixed header file problems on OpenBSD.
- Fixed issue with MPI_TYPE_CREATE_F90_REAL.
- Fix an issue with using external libltdl installations. Thanks to
opolawski for identifying the problem.
- Fixed MPI_IN_PLACE case for MPI_ALLGATHER for FCA.
- Allow SLURM PMI support to look in lib64 directories. Thanks to
Guillaume Papaure for the patch.
- Restore "use mpi" ABI compatibility with the rest of the 1.5/1.6
series (except for v1.6.3, where it was accidentally broken).
- Fix a very old error in opal_path_access(). Thanks to Marco Atzeri
for chasing it down.
Version 1.6.3 is a minor bugfix release.
All users are encouraged to upgrade to v1.6.3 when possible.
Note that v1.6.3 is ABI compatible with the entire v1.5.x and
v1.6.x series, but is not ABI compatible with the v1.4.x series.
See http://www.open-mpi.org/software/ompi/versions/ for a
description of Open MPI's release methodology.
Here is a list of changes in v1.6.3 as compared to v1.6.2:
- Fix mpirun --launch-agent behavior when a prefix is specified.
Thanks to Reuti for identifying the issue.
- Fixed memchecker configury.
- Brought over some compiler warning squashes from the development trunk.
- Fix spawning from a singleton to multiple hosts when the "add-host"
MPI_Info key is used. Thanks to Brian Budge for pointing out the
problem.
- Add Mellanox ConnextIB IDs and max inline value.
- Fix rankfile when no -np is given.
- FreeBSD detection improvement. Thanks to Brooks Davis for the
patch.
- Removed TCP warnings on Windows.
- Improved collective algorithm selection for very large messages.
- Fix PSM MTL affinity settings.
- Fix issue with MPI_OP_COMMUTATIVE in the mpif.h bindings. Thanks to
Ake Sandgren for providing a patch to fix the issue.
- Fix issue with MPI_SIZEOF when using CHARACTER and LOGICAL types in
the mpi module. Thanks to Ake Sandgren for providing a patch to fix
the issue.
Don't build VampirTrace anymore, it will be introduced as separate package.
Changes in v1.6.2 as compared to v1.6.1:
- Fix issue with MX MTL. Thanks to Doug Eadline for raising the issue.
- Fix singleton MPI_COMM_SPAWN when the result job spans multiple nodes.
- Fix MXM hang, and update for latest version of MXM.
- Update to support Mellanox FCA 2.5.
- Fix startup hang for large jobs.
- Ensure MPI_TESTANY / MPI_WAITANY properly set the empty status when
count==0.
- Fix MPI_CART_SUB behavior of not copying periods to the new
communicator properly. Thanks to John Craske for the bug report.
- Add btl_openib_abort_not_enough_reg_mem MCA parameter to cause Open
MPI to abort MPI jobs if there is not enough registered memory
available on the system (vs. just printing a warning). Thanks to
Brock Palen for raising the issue.
- Minor fix to Fortran MPI_INFO_GET: only copy a value back to the
user's buffer if the flag is .TRUE.
- Fix VampirTrace compilation issue with the PGI compiler suite.
v1.5 is a major new release series. It includes many new features and
changes over the v1.4.x series. The most noticeable changes are the
addition of the lstopo-no-graphics program (which does require any
heavy external library such as Cairo) and the discovery of instruction
caches. There are also many small improvements to all backends, and
some deprecated features have been removed.
Version 1.5.0
-------------
* Backends
+ Do not limit the number of processors to 1024 on Solaris anymore.
+ Gather total machine memory on FreeBSD.
+ XML topology files do not depend on the locale anymore. Float numbers
such as NUMA distances or PCI link speeds now always use a dot as a
decimal separator.
+ Add instruction caches detection on Linux, AIX, Windows and Darwin.
+ Add get_last_cpu_location() support for the current thread on AIX.
+ Support binding on AIX when threads or processes were bound with
bindprocessor(). Thanks to Hendryk Bockelmann for reporting the issue
and testing patches, and to Farid Parpia for explaining the binding
interfaces.
+ Improve AMD topology detection in the x86 backend (for FreeBSD) using
the topoext feature.
* API
+ Increase HWLOC_API_VERSION to 0x00010500 so that API changes may be
detected at build-time.
+ Add a cache type attribute describind Data, Instruction and Unified
caches. Caches with different types but same depth (for instance L1d
and L1i) are placed on different levels.
+ Add hwloc_get_cache_type_depth() to retrieve the hwloc level depth of
of the given cache depth and type, for instance L1i or L2.
It helps disambiguating the case where hwloc_get_type_depth() returns
HWLOC_TYPE_DEPTH_MULTIPLE.
+ Instruction caches are ignored unless HWLOC_TOPOLOGY_FLAG_ICACHES is
passed to hwloc_topology_set_flags() before load.
+ Add hwloc_ibv_get_device_osdev_by_name() OpenFabrics helper in
openfabrics-verbs.h to find the hwloc OS device object corresponding to
an OpenFabrics device.
* Tools
+ Add lstopo-no-graphics, a lstopo built without graphical support to
avoid dependencies on external libraries such as Cairo and X11. When
supported, graphical outputs are only available in the original lstopo
program.
- Packagers splitting lstopo and lstopo-no-graphics into different
packages are advised to use the alternatives system so that lstopo
points to the best available binary.
+ Instruction caches are enabled in lstopo by default. User --no-icaches
to disable them.
+ Add -t/--threads to show threads in hwloc-ps.
* Removal of obsolete components
+ Remove the old cpuset interface (hwloc/cpuset.h) which is deprecated and
superseded by the bitmap API (hwloc/bitmap.h) since v1.1.
hwloc_cpuset and nodeset types are still defined, but all hwloc_cpuset_*
compatibility wrappers are now gone.
+ Remove Linux libnuma conversion helpers for the deprecated and
broken nodemask_t interface.
+ Remove support for "Proc" type name, it was superseded by "PU" in v1.0.
+ Remove hwloc-mask symlinks, it was replaced by hwloc-calc in v1.0.
* Misc
+ Fix PCIe 3.0 link speed computation.
+ Non-printable characters are dropped from strings during XML export.
+ Fix importing of escaped characters with the minimalistic XML backend.
+ Assert hwloc_is_thissystem() in several I/O related helpers.
+ Fix some memory leaks in the x86 backend for FreeBSD.
+ Minor fixes to ease native builds on Windows.
+ Limit the number of retries when operating on all threads within a
process on Linux if the list of threads is heavily getting modified.
paexec:
- Option -x was added. With its help paexec can run one command
per task. If -g is also specified, command's exit status is
analysed. Appropriate task and dependants are marked as "failed"
if it is non-zero.
- First character of -n argument must be alphanumeric, `+', `_',
`:' or `/'. Other symbols are reserved for future extentions.
- With '-n :filename' paexec reads a list of nodes from the
specified file.
- With a help of new option '-m t=<eot>' end of task string
may be specified, which is an empty line by default.
- Option -md=<delim> was added that overrides the default
delimiter (space character) between tasks in graph mode (-g).
- Output line that contains failed dependants no longer ends with
unnecessary space.
- Long options were completely removed.
paexec_reorder:
- Fix. "paexec_reorder -g" now handles correctly failed tasks'
output. One extra line after "fatal" is expected.
- Options -m was added. It does the same things as paexec's -m.
More examples of use and regression tests.
Documentation update, clean-ups and improvements.
Regression tests:
- Signals handling was fixed in.
- LC_ALL is always set to C in regression tests, this fixes some
problems in internationalized environment.
mk-configure>=0.23.0 is required at build time
* Changes in SLURM 2.4.1
========================
-- Fix bug for job state change from 2.3 -> 2.4 job state can now be preserved
correctly when transitioning. This also applies for 2.4.0 -> 2.4.1, no
state will be lost. (Thanks to Carles Fenoy)
* Changes in SLURM 2.4.0
========================
-- Cray - Improve support for zero compute note resource allocations.
Partition used can now be configured with no nodes nodes.
-- BGQ - make it so srun -i<taskid> works correctly.
-- Fix parse_uint32/16 to complain if a non-digit is given.
-- Add SUBMITHOST to job state passed to Moab vial sched/wiki2. Patch by Jon
Bringhurst (LANL).
-- BGQ - Fix issue when running with AllowSubBlockAllocations=Yes without
compiling with --enable-debug
-- Modify scontrol to require "-dd" option to report batch job's script. Patch
from Don Albert, Bull.
-- Modify SchedulerParamters option to match documentation: "bf_res="
changed to "bf_resolution=". Patch from Rod Schultz, Bull.
-- Fix bug that clears job pending reason field. Patch fron Don Lipari, LLNL.
-- In etc/init.d/slurm move check for scontrol after sourcing
/etc/sysconfig/slurm. Patch from Andy Wettstein, University of Chicago.
-- Fix in scheduling logic that can delay jobs with min/max node counts.
-- BGQ - fix issue where if a step uses the entire allocation and then
the next step in the allocation only uses part of the allocation it gets
the correct cnodes.
-- BGQ - Fix checking for IO on a block with new IBM driver V1R1M1 previous
function didn't always work correctly.
-- BGQ - Fix issue when a nodeboard goes down and you want to combine blocks
to make a larger small block and are running with sub-blocks.
-- BLUEGENE - Better logic for making small blocks around bad nodeboard/card.
-- BGQ - When using an old IBM driver cnodes that go into error because of
a job kill timeout aren't always reported to the system. This is now
handled by the runjob_mux plugin.
-- BGQ - Added information on how to setup the runjob_mux to run as SlurmUser.
-- Improve memory consumption on step layouts with high task count.
-- BGQ - quiter debug when the real time server comes back but there are
still messages we find when we poll but haven't given it back to the real
time yet.
-- BGQ - fix for if a request comes in smaller than the smallest block and
we must use a small block instead of a shared midplane block.
-- Fix issues on large jobs (>64k tasks) to have the correct counter type when
packing the step layout structure.
-- BGQ - fix issue where if a user was asking for tasks and ntasks-per-node
but not node count the node count is correctly figured out.
-- Move logic to always use the 1st alphanumeric node as the batch host for
batch jobs.
-- BLUEGENE - fix race condition where if a nodeboard/card goes down at the
same time a block is destroyed and that block just happens to be the
smallest overlapping block over the bad hardware.
-- Fix bug when querying accounting looking for a job node size.
-- BLUEGENE - fix possible race condition if cleaning up a block and the
removal of the job on the block failed.
-- BLUEGENE - fix issue if a cable was in an error state make it so we can
check if a block is still makable if the cable wasn't in error.
-- Put nodes names in alphabetic order in node table.
-- If preempted job should have a grace time and preempt mode is not cancel
but job is going to be canceled because it is interactive or other reason
it now receives the grace time.
-- BGQ - Modified documents to explain new plugin_flags needed in bg.properties
in order for the runjob_mux to run correctly.
-- BGQ - change linking from libslurm.o to libslurmhelper.la to avoid warning.
* Changes in SLURM 2.4.0.rc1
=============================
-- Improve task binding logic by making fuller use of HWLOC library,
especially with respect to Opteron 6000 series processors. Work contributed
by Komoto Masahiro.
-- Add new configuration parameter PriorityFlags, based upon work by
Carles Fenoy (Barcelona Supercomputer Center).
-- Modify the step completion RPC between slurmd and slurmstepd in order to
eliminate a possible deadlock. Based on work by Matthieu Hautreux, CEA.
-- Change the owner of slurmctld and slurmdbd log files to the appropriate
user. Without this change the files will be created by and owned by the
user starting the daemons (likely user root).
-- Reorganize the slurmstepd logic in order to better support NFS and
Kerberos credentials via the AUKS plugin. Work by Matthieu Hautreux, CEA.
-- Fix bug in allocating GRES that are associated with specific CPUs. In some
cases the code allocated first available GRES to job instead of allocating
GRES accessible to the specific CPUs allocated to the job.
-- spank: Add callbacks in slurmd: slurm_spank_slurmd_{init,exit}
and job epilog/prolog: slurm_spank_job_{prolog,epilog}
-- spank: Add spank_option_getopt() function to api
-- Change resolution of switch wait time from minutes to seconds.
-- Added CrpCPUMins to the output of sshare -l for those using hard limit
accounting. Work contributed by Mark Nelson.
-- Added mpi/pmi2 plugin for complete support of pmi2 including acquiring
additional resources for newly launched tasks. Contributed by Hongjia Cao,
NUDT.
-- BGQ - fixed issue where if a user asked for a specific node count and more
tasks than possible without overcommit the request would be allowed on more
nodes than requested.
-- Add support for new SchedulerParameters of bf_max_job_user, maximum number
of jobs to attempt backfilling per user. Work by BjæËrn-Helge Mevik,
University of Oslo.
-- BLUEGENE - fixed issue where MaxNodes limit on a partition only limited
larger than midplane jobs.
-- Added cpu_run_min to the output of sshare --long. Work contributed by
Mark Nelson.
-- BGQ - allow regular users to resolve Rack-Midplane to AXYZ coords.
-- Add sinfo output format option of "%R" for partition name without "*"
appended for default partition.
-- Cray - Add support for zero compute note resource allocation to run batch
script on front-end node with no ALPS reservation. Useful for pre- or post-
processing.
-- Support for cyclic distribution of cpus in task/cgroup plugin from Martin
Perry, Bull.
-- GrpMEM limit for QOSes and associations added Patch from BjæËrn-Helge Mevik,
University of Oslo.
-- Various performance improvements for up to 500% higher throughput depending
upon configuration. Work supported by the Oak Ridge National Laboratory
Extreme Scale Systems Center.
-- Added jobacct_gather/cgroup plugin. It is not advised to use this in
production as it isn't currently complete and doesn't provide an equivalent
substitution for jobacct_gather/linux yet. Work by Martin Perry, Bull.
