Updated ABI generation code and new libraries #13280

hppritcha · 2025-05-27T20:32:21Z

Two external MPI libraries are now created: libmpi.so and libmpi_abi.so.
Backend code that was originally in libmpi.la has been extracted into
libopen-mpi.la to be linked into both libraries.

Parts of the Open MPI C interface are now being generated by a python
script (abi.py) from modified source files (named with *.in). This
script generates files for both the ompi ABI and the standard ABI from
the same source file, also including new bigcount interfaces.

To compile standard ABI code, there's a new mpicc_abi compiler wrapper.
The standard ABI does not yet include all functions or symbols, so more
complicated source files will not compile. ROMIO must be disabled for
the code to link, since it's relying on the external MPI interface.

Many todos left:

switch over to using the canonical mpi.h header file
fix some remaining issues with types in the bindings framework
implement method for wrapping user callbacks so that they are passed abi versions of MPI handles
Add binding generation for MPI T functions
Fix enable-mca-dso
other

This PR supercedes #12033

Two external MPI libraries are now created: libmpi.la and libmpi_abi.la. Backend code that was originally in libmpi.la has been extracted into libopen-mpi.la to be linked into both libraries. Parts of the Open MPI C interface are now being generated by a python script (abi.py) from modified source files (named with *.in). This script generates files for both the ompi ABI and the standard ABI from the same source file, also including new bigcount interfaces. To compile standard ABI code, there's a new mpicc_abi compiler wrapper. The standard ABI does not yet include all functions or symbols, so more complicated source files will not compile. ROMIO must be disabled for the code to link, since it's relying on the external MPI interface. Signed-off-by: Jake Tronge <jtronge@lanl.gov>

Implement lots of missing functionality in the original api.py script. The functionality is now part of the bindings generation framework used for Big Count. Number of todos still to do, in particular provide support for wrapping user supplied callback functions. Also, the sendrecv_replace, etc. code needs to be refactored to work with the bindings framework. Signed-off-by: Howard Pritchard <howardp@lanl.gov>

to heed the --enable-abi-standard config option Signed-off-by: Howard Pritchard <howardp@lanl.gov>

Signed-off-by: Howard Pritchard <howardp@lanl.gov>

for comm/type/win attributes. Not clear if this is the way to handle attr copy/del call backs compiled againt the mpi.h ABI header file. Signed-off-by: Howard Pritchard <howardp@lanl.gov>

This approach works for attributes but other approaches will be needed for error handlers and datatype conversion related functions. Signed-off-by: Howard Pritchard <howardp@lanl.gov>

dalcinl · 2025-07-11T19:17:08Z

Maybe you should somehow vendor the mpi.h header from https://github.com/mpi-forum/mpi-abi-stubs and use it as the baseline to extract values for handles and constants? That's what MPICH is doing.
Alternatively, the mpi.h header the mpi-abi-stubs repo could be downloaded on the fly by some Python script and the values in that header used to update stuff to be committed in the ompi repo.

In short, I think it would be in everyone's convenience to use https://github.com/mpi-forum/mpi-abi-stubs as the "source of truth" for ABI-related stuff, avoiding manual synchronization of handle/constant values.

Signed-off-by: Howard Pritchard <howardp@lanl.gov>

github-actions · 2025-08-14T20:52:33Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

f7d94fa: WIP: explain issue with pympistandard for callback...

check_signed_off: does not contain a valid Signed-off-by line

4d79937: WIP: fix JSONs

check_signed_off: does not contain a valid Signed-off-by line

ec9c45a: WIP: fix typo in pympistd arg

check_signed_off: does not contain a valid Signed-off-by line

b047b19: WIP: add JSONs for ABI and API

check_signed_off: does not contain a valid Signed-off-by line

c9d0c7a: WIP: bump pympistandard commit for profiling embig...

check_signed_off: does not contain a valid Signed-off-by line

a1255ce: WIP: move Aint helper macros under ifndef OMPI_NO_...

check_signed_off: does not contain a valid Signed-off-by line

342b072: WIP: add some workarounds for MPI_Fint and MPI_Inf...

check_signed_off: does not contain a valid Signed-off-by line

53aeac5: WIP: mangle some more functions

check_signed_off: does not contain a valid Signed-off-by line

93c0a39: WIP: avoid double inclusion of abi.h

check_signed_off: does not contain a valid Signed-off-by line

afd9eb2: WIP: use pympistandard by editing PYTHONPATH (inst...

check_signed_off: does not contain a valid Signed-off-by line

d397bfc: WIP: fix some bugs in mangling names

check_signed_off: does not contain a valid Signed-off-by line

da2630f: WIP: fix typo for MPI_internal

check_signed_off: does not contain a valid Signed-off-by line

956dded: WIP: add additional types and functions to be mang...

check_signed_off: does not contain a valid Signed-off-by line

0b10ce6: WIP: temp fix for Aint problems

check_signed_off: does not contain a valid Signed-off-by line

9830327: WIP: add input for abi.h.in

check_signed_off: does not contain a valid Signed-off-by line

68c69df: WIP: move abi.h.in

check_signed_off: does not contain a valid Signed-off-by line

702cb23: WIP: add in 5.0 apis.json

check_signed_off: does not contain a valid Signed-off-by line

6b2784f: WIP: move code out of consts.py

check_signed_off: does not contain a valid Signed-off-by line

22d4cb2: WIP: call c_header from Makefile

check_signed_off: does not contain a valid Signed-off-by line

b0d0ff2: WIP: mangle names for internal usage

check_signed_off: does not contain a valid Signed-off-by line

96a33c3: WIP: generate callback function prototypes

check_signed_off: does not contain a valid Signed-off-by line

d2dd7ed: WIP: remove comment function

check_signed_off: does not contain a valid Signed-off-by line

45d8789: WIP: print out embiggened versions of functions

check_signed_off: does not contain a valid Signed-off-by line

b081aa2: WIP: add MPI and ABI versions

check_signed_off: does not contain a valid Signed-off-by line

80ebfe7: WIP: generate API prototypes

check_signed_off: does not contain a valid Signed-off-by line

93e6375: WIP: Comment out a Fortran-only category

check_signed_off: does not contain a valid Signed-off-by line

db8a2b5: WIP: add comment pointing back to MPI standard

check_signed_off: does not contain a valid Signed-off-by line

08dfb42: WIP: use enums for most int values

check_signed_off: does not contain a valid Signed-off-by line

aab2023: WIP: create ABI header file from template with cat...

check_signed_off: does not contain a valid Signed-off-by line

46fae8e: WIP: generate header with ABI values for #defines

check_signed_off: does not contain a valid Signed-off-by line

c15a05d: WIP: remove abi.py

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2025-08-14T21:07:44Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

f7d94fa: WIP: explain issue with pympistandard for callback...

check_signed_off: does not contain a valid Signed-off-by line

4d79937: WIP: fix JSONs

check_signed_off: does not contain a valid Signed-off-by line

ec9c45a: WIP: fix typo in pympistd arg

check_signed_off: does not contain a valid Signed-off-by line

b047b19: WIP: add JSONs for ABI and API

check_signed_off: does not contain a valid Signed-off-by line

c9d0c7a: WIP: bump pympistandard commit for profiling embig...

check_signed_off: does not contain a valid Signed-off-by line

a1255ce: WIP: move Aint helper macros under ifndef OMPI_NO_...

check_signed_off: does not contain a valid Signed-off-by line

342b072: WIP: add some workarounds for MPI_Fint and MPI_Inf...

check_signed_off: does not contain a valid Signed-off-by line

53aeac5: WIP: mangle some more functions

check_signed_off: does not contain a valid Signed-off-by line

93c0a39: WIP: avoid double inclusion of abi.h

check_signed_off: does not contain a valid Signed-off-by line

afd9eb2: WIP: use pympistandard by editing PYTHONPATH (inst...

check_signed_off: does not contain a valid Signed-off-by line

d397bfc: WIP: fix some bugs in mangling names

check_signed_off: does not contain a valid Signed-off-by line

da2630f: WIP: fix typo for MPI_internal

check_signed_off: does not contain a valid Signed-off-by line

956dded: WIP: add additional types and functions to be mang...

check_signed_off: does not contain a valid Signed-off-by line

0b10ce6: WIP: temp fix for Aint problems

check_signed_off: does not contain a valid Signed-off-by line

9830327: WIP: add input for abi.h.in

check_signed_off: does not contain a valid Signed-off-by line

68c69df: WIP: move abi.h.in

check_signed_off: does not contain a valid Signed-off-by line

702cb23: WIP: add in 5.0 apis.json

check_signed_off: does not contain a valid Signed-off-by line

6b2784f: WIP: move code out of consts.py

check_signed_off: does not contain a valid Signed-off-by line

22d4cb2: WIP: call c_header from Makefile

check_signed_off: does not contain a valid Signed-off-by line

b0d0ff2: WIP: mangle names for internal usage

check_signed_off: does not contain a valid Signed-off-by line

96a33c3: WIP: generate callback function prototypes

check_signed_off: does not contain a valid Signed-off-by line

d2dd7ed: WIP: remove comment function

check_signed_off: does not contain a valid Signed-off-by line

45d8789: WIP: print out embiggened versions of functions

check_signed_off: does not contain a valid Signed-off-by line

b081aa2: WIP: add MPI and ABI versions

check_signed_off: does not contain a valid Signed-off-by line

80ebfe7: WIP: generate API prototypes

check_signed_off: does not contain a valid Signed-off-by line

93e6375: WIP: Comment out a Fortran-only category

check_signed_off: does not contain a valid Signed-off-by line

db8a2b5: WIP: add comment pointing back to MPI standard

check_signed_off: does not contain a valid Signed-off-by line

08dfb42: WIP: use enums for most int values

check_signed_off: does not contain a valid Signed-off-by line

aab2023: WIP: create ABI header file from template with cat...

check_signed_off: does not contain a valid Signed-off-by line

46fae8e: WIP: generate header with ABI values for #defines

check_signed_off: does not contain a valid Signed-off-by line

c15a05d: WIP: remove abi.py

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2025-08-14T22:36:22Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

e86186b: WIP: add JSONs for ABI and API

check_signed_off: does not contain a valid Signed-off-by line

24417ec: WIP: bump pympistandard commit for profiling embig...

check_signed_off: does not contain a valid Signed-off-by line

718b1d0: WIP: move Aint helper macros under ifndef OMPI_NO_...

check_signed_off: does not contain a valid Signed-off-by line

3431c8d: WIP: add some workarounds for MPI_Fint and MPI_Inf...

check_signed_off: does not contain a valid Signed-off-by line

566fdaa: WIP: mangle some more functions

check_signed_off: does not contain a valid Signed-off-by line

8396bef: WIP: avoid double inclusion of abi.h

check_signed_off: does not contain a valid Signed-off-by line

39c20b7: WIP: use pympistandard by editing PYTHONPATH (inst...

check_signed_off: does not contain a valid Signed-off-by line

d1aece4: WIP: fix some bugs in mangling names

check_signed_off: does not contain a valid Signed-off-by line

c7c1809: WIP: fix typo for MPI_internal

check_signed_off: does not contain a valid Signed-off-by line

2bbf9eb: WIP: add additional types and functions to be mang...

check_signed_off: does not contain a valid Signed-off-by line

1db8082: WIP: temp fix for Aint problems

check_signed_off: does not contain a valid Signed-off-by line

870e925: WIP: add input for abi.h.in

check_signed_off: does not contain a valid Signed-off-by line

a56da85: WIP: move abi.h.in

check_signed_off: does not contain a valid Signed-off-by line

4c2aee1: WIP: add in 5.0 apis.json

check_signed_off: does not contain a valid Signed-off-by line

3d85943: WIP: move code out of consts.py

check_signed_off: does not contain a valid Signed-off-by line

4e85726: WIP: call c_header from Makefile

check_signed_off: does not contain a valid Signed-off-by line

8d8f554: WIP: mangle names for internal usage

check_signed_off: does not contain a valid Signed-off-by line

5f29a48: WIP: generate callback function prototypes

check_signed_off: does not contain a valid Signed-off-by line

c5d5f30: WIP: remove comment function

check_signed_off: does not contain a valid Signed-off-by line

1613149: WIP: print out embiggened versions of functions

check_signed_off: does not contain a valid Signed-off-by line

3f229b3: WIP: add MPI and ABI versions

check_signed_off: does not contain a valid Signed-off-by line

584aeb7: WIP: generate API prototypes

check_signed_off: does not contain a valid Signed-off-by line

7c73091: WIP: Comment out a Fortran-only category

check_signed_off: does not contain a valid Signed-off-by line

72d2bff: WIP: add comment pointing back to MPI standard

check_signed_off: does not contain a valid Signed-off-by line

c303345: WIP: use enums for most int values

check_signed_off: does not contain a valid Signed-off-by line

7d79681: WIP: create ABI header file from template with cat...

check_signed_off: does not contain a valid Signed-off-by line

99e9992: WIP: generate header with ABI values for #defines

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

jsquyres · 2025-08-15T14:43:24Z

I wonder if we should make the bot not complain about unsigned commits on draft PRs. That would reduce some of the noise on PR's like this.

This mod switches from using "synthetic" defined values and handles for the ones specified in the MPI 5.1 standard. The python infrastructure included in this PR for generating both a "canonical" abi standard compatible MPI 5.1 mpi.h using two json files: 1) mpi-standard-abi.json 2) mpi-standard-apis.json 2 is generated as part of building the MPI standard. We import that into our project for use in generating both the mpi.h as well as interface definitions in the man pages. 1 is generated using a separate script that processes the tables in Appendix A of standard. Ideally this script will be merged into the MPI standard code base at some point. This script is currently at https://github.com/Joe-Downs/mpi-standard/tree/pr/handle-constant-tool/const-tool . It is used to generate the portion of mpi.h where defined values and handles are specified. The converter functions that had been generated as part of the build out of the abi variants of the 'c' MPI interfaces are no define in persistent file. The methods defined in this file will be optimized to make use of the Huffman code characteristics of the predefined values in a subsequent PR. This commit also enables generation of the abi interfaces and header files by default. Signed-off-by: Joseph Downs <joe.downs@lanl.gov>

Signed-off-by: Howard Pritchard <howardp@lanl.gov>

in the library as they are not part of the ABI. Signed-off-by: Howard Pritchard <howardp@lanl.gov>

jtronge and others added 2 commits May 27, 2025 09:43

hppritcha marked this pull request as draft May 27, 2025 20:32

github-actions bot added the Target: main label May 27, 2025

jsquyres mentioned this pull request May 28, 2025

Standard ABI generation code and library refactor. #12033

Closed

bindings: fix up makefile for c interfaces

49f042c

to heed the --enable-abi-standard config option Signed-off-by: Howard Pritchard <howardp@lanl.gov>

hppritcha force-pushed the abi-generate-ver2 branch from d127224 to 8fbf968 Compare May 29, 2025 18:36

makefile fixes

29b1a2f

Signed-off-by: Howard Pritchard <howardp@lanl.gov>

hppritcha force-pushed the abi-generate-ver2 branch from 8fbf968 to 29b1a2f Compare May 29, 2025 21:16

hppritcha added 2 commits July 1, 2025 10:55

checkpoint

b0e9261

Signed-off-by: Howard Pritchard <howardp@lanl.gov>

some fixes and start of a wrapper method

340e7ad

for comm/type/win attributes. Not clear if this is the way to handle attr copy/del call backs compiled againt the mpi.h ABI header file. Signed-off-by: Howard Pritchard <howardp@lanl.gov>

hppritcha force-pushed the abi-generate-ver2 branch from da515d4 to 340e7ad Compare July 2, 2025 18:52

jsquyres mentioned this pull request Jul 4, 2025

bugfix: Setting OMPI_MPI_THREAD_LEVEL to a value different than requested crashes #13211

Merged

attributes: add wrapper infrastructure

4661c68

This approach works for attributes but other approaches will be needed for error handlers and datatype conversion related functions. Signed-off-by: Howard Pritchard <howardp@lanl.gov>

hppritcha added 3 commits July 29, 2025 11:35

abi: move converter functions to a persistent file

50afcac

Signed-off-by: Howard Pritchard <howardp@lanl.gov>

minor compiler complaint fix

47c60bc

Signed-off-by: Howard Pritchard <howardp@lanl.gov>

temp commit with comments notes

3d2b84c

Signed-off-by: Howard Pritchard <howardp@lanl.gov>

hppritcha force-pushed the abi-generate-ver2 branch from a9b105c to cefbde8 Compare August 14, 2025 22:35

hppritcha force-pushed the abi-generate-ver2 branch from cefbde8 to d5663f7 Compare August 20, 2025 17:30

hppritcha added 3 commits August 21, 2025 13:02

distcheck fix and more

b8819d4

Signed-off-by: Howard Pritchard <howardp@lanl.gov>

git ignore additions

fedeb73

Signed-off-by: Howard Pritchard <howardp@lanl.gov>

ABI library: don't include functions in 19.3.4/5

efe6b00

in the library as they are not part of the ABI. Signed-off-by: Howard Pritchard <howardp@lanl.gov>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Updated ABI generation code and new libraries #13280

Updated ABI generation code and new libraries #13280

hppritcha commented May 27, 2025 •

edited

Loading

Uh oh!

dalcinl commented Jul 11, 2025

Uh oh!

github-actions bot commented Aug 14, 2025

Uh oh!

github-actions bot commented Aug 14, 2025

Uh oh!

github-actions bot commented Aug 14, 2025

Uh oh!

jsquyres commented Aug 15, 2025

Uh oh!

Uh oh!

Updated ABI generation code and new libraries #13280

Are you sure you want to change the base?

Updated ABI generation code and new libraries #13280

Conversation

hppritcha commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dalcinl commented Jul 11, 2025

Uh oh!

github-actions bot commented Aug 14, 2025

Uh oh!

github-actions bot commented Aug 14, 2025

Uh oh!

github-actions bot commented Aug 14, 2025

Uh oh!

jsquyres commented Aug 15, 2025

Uh oh!

Uh oh!

hppritcha commented May 27, 2025 •

edited

Loading