Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compatibility with netcdf-c version 4.7.4 #1363

Open
awikner opened this issue Sep 17, 2024 · 1 comment
Open

Compatibility with netcdf-c version 4.7.4 #1363

awikner opened this issue Sep 17, 2024 · 1 comment

Comments

@awikner
Copy link

awikner commented Sep 17, 2024

I am trying to install the developer version of netcdf4-python version 1.7.1 with parallel hdf5 and parallel netcdf enabled.

I am using the following software versions:

OS: Red Hat Enterprise Linux version 8.4
C Compiler: impi/2019.7.217/intel64/bin/mpicc
HDF5: hdf5-1.12.0-el8-x86_64+intelmpi-2019.up7+intel-19.1.1
Netcdf-c: netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1
Anaconda: 33.16.6.7

I have built mpi4py and h5py in a conda environment with MPI enabled via the following commands:

export MPICC=$(which mpicc)
pip install mpi4py --no-cache-dir
export CC=$(which mpicc)
export HDF5_MPI="ON"
pip install --no-binary=h5py h5py

I additionally had to install a newer version of openssl with conda install --channel=conda-forge openssl.

My final environment has the following packages installed:

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
asttokens                 2.4.1                    pypi_0    pypi
bzip2                     1.0.8                h5eee18b_6
ca-certificates           2024.8.30            hbcca054_0    conda-forge
certifi                   2024.8.30                pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
executing                 2.1.0                    pypi_0    pypi
h5py                      3.11.0                   pypi_0    pypi
ipython                   8.27.0                   pypi_0    pypi
jedi                      0.19.1                   pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1
libffi                    3.4.4                h6a678d5_1
libgcc                    14.1.0               h77fa898_1    conda-forge
libgcc-ng                 14.1.0               h69a702a_1    conda-forge
libgomp                   14.1.0               h77fa898_1    conda-forge
libstdcxx-ng              11.2.0               h1234567_1
libuuid                   1.41.5               h5eee18b_0
matplotlib-inline         0.1.7                    pypi_0    pypi
mpi4py                    4.0.0                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0
numpy                     2.1.1                    pypi_0    pypi
openssl                   3.3.2                hb9d3cd8_0    conda-forge
packaging                 24.1                     pypi_0    pypi
parso                     0.8.4                    pypi_0    pypi
pexpect                   4.9.0                    pypi_0    pypi
pip                       24.2            py311h06a4308_0
prompt-toolkit            3.0.47                   pypi_0    pypi
ptyprocess                0.7.0                    pypi_0    pypi
pure-eval                 0.2.3                    pypi_0    pypi
pygments                  2.18.0                   pypi_0    pypi
python                    3.11.9               h955ad1f_0
readline                  8.2                  h5eee18b_0
setuptools                72.1.0          py311h06a4308_0
six                       1.16.0                   pypi_0    pypi
sqlite                    3.45.3               h5eee18b_0
stack-data                0.6.3                    pypi_0    pypi
tk                        8.6.14               h39e8969_0
traitlets                 5.14.3                   pypi_0    pypi
typing-extensions         4.12.2                   pypi_0    pypi
tzdata                    2024a                h04d1e81_0
urllib3                   2.2.3                    pypi_0    pypi
wcwidth                   0.2.13                   pypi_0    pypi
wheel                     0.44.0          py311h06a4308_0
xz                        5.4.6                h5eee18b_1
zlib                      1.2.13               h5eee18b_1

When I attempt to install netcdf4-python from source with pip install --no-build-isolation -v ., I receive the following output with an error:

Using pip 24.2 from /home/awikner/.conda/envs/netcdf4-test/lib/python3.11/site-packages/pip (python 3.11)
Processing /home/awikner/netcdf4-python
  Running command Preparing metadata (pyproject.toml)
  -L/software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/lib -lnetcdf
  Package hdf5 was not found in the pkg-config search path.
  Perhaps you should add the directory containing `hdf5.pc'
  to the PKG_CONFIG_PATH environment variable
  Package 'hdf5', required by 'virtual:world', not found
  reading from setup.cfg...
  using /software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/bin/nc-config...
  lib_dirs: ['/software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/lib']
  checking /software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/include...
  hdf5 headers not found in /software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/include
  nc-config did provide path to HDF5 headers, search standard locations...
  checking /software/hdf5-1.12.0-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/include...
  HDF5 library version: 1.12.0 headers found in /software/hdf5-1.12.0-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/include
  HDF5 library version: 1.12.0 found in /software/hdf5-1.12.0-el8-x86_64+intelmpi-2019.up7+intel-19.1.1
  lib_dirs: ['/software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/lib', '/software/hdf5-1.12.0-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/lib']
  using netcdf library version 4.7.4
  using Cython to compile netCDF4.pyx...
  netcdf lib has parallel functions
  NETCDF_PLUGIN_DIR not set, no netcdf compression plugins installed
  running dist_info
  creating /home/awikner/pip-modern-metadata-zaxkixg_/netCDF4.egg-info
  writing /home/awikner/pip-modern-metadata-zaxkixg_/netCDF4.egg-info/PKG-INFO
  writing dependency_links to /home/awikner/pip-modern-metadata-zaxkixg_/netCDF4.egg-info/dependency_links.txt
  writing entry points to /home/awikner/pip-modern-metadata-zaxkixg_/netCDF4.egg-info/entry_points.txt
  writing requirements to /home/awikner/pip-modern-metadata-zaxkixg_/netCDF4.egg-info/requires.txt
  writing top-level names to /home/awikner/pip-modern-metadata-zaxkixg_/netCDF4.egg-info/top_level.txt
  writing manifest file '/home/awikner/pip-modern-metadata-zaxkixg_/netCDF4.egg-info/SOURCES.txt'
  reading manifest file '/home/awikner/pip-modern-metadata-zaxkixg_/netCDF4.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no previously-included files found matching 'examples/data'
  warning: no previously-included files found matching 'src/netCDF4/_netCDF4.c'
  adding license file 'LICENSE'
  writing manifest file '/home/awikner/pip-modern-metadata-zaxkixg_/netCDF4.egg-info/SOURCES.txt'
  creating '/home/awikner/pip-modern-metadata-zaxkixg_/netCDF4-1.7.2.dist-info'
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: cftime in /home/awikner/.local/lib/python3.11/site-packages (from netCDF4==1.7.2) (1.6.4)
Requirement already satisfied: certifi in /home/awikner/.conda/envs/netcdf4-test/lib/python3.11/site-packages (from netCDF4==1.7.2) (2024.8.30)
Requirement already satisfied: numpy in /home/awikner/.conda/envs/netcdf4-test/lib/python3.11/site-packages (from netCDF4==1.7.2) (2.1.1)
Building wheels for collected packages: netCDF4
  Running command Building wheel for netCDF4 (pyproject.toml)
  -L/software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/lib -lnetcdf
  Package hdf5 was not found in the pkg-config search path.
  Perhaps you should add the directory containing `hdf5.pc'
  to the PKG_CONFIG_PATH environment variable
  Package 'hdf5', required by 'virtual:world', not found
  reading from setup.cfg...
  using /software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/bin/nc-config...
  lib_dirs: ['/software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/lib']
  checking /software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/include...
  hdf5 headers not found in /software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/include
  nc-config did provide path to HDF5 headers, search standard locations...
  checking /software/hdf5-1.12.0-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/include...
  HDF5 library version: 1.12.0 headers found in /software/hdf5-1.12.0-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/include
  HDF5 library version: 1.12.0 found in /software/hdf5-1.12.0-el8-x86_64+intelmpi-2019.up7+intel-19.1.1
  lib_dirs: ['/software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/lib', '/software/hdf5-1.12.0-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/lib']
  using netcdf library version 4.7.4
  using Cython to compile netCDF4.pyx...
  netcdf lib has parallel functions
  NETCDF_PLUGIN_DIR not set, no netcdf compression plugins installed
  running bdist_wheel
  running build
  running build_py
  copying src/netCDF4/__init__.py -> build/lib.linux-x86_64-cpython-311/netCDF4
  copying src/netCDF4/utils.py -> build/lib.linux-x86_64-cpython-311/netCDF4
  running egg_info
  writing src/netCDF4.egg-info/PKG-INFO
  writing dependency_links to src/netCDF4.egg-info/dependency_links.txt
  writing entry points to src/netCDF4.egg-info/entry_points.txt
  writing requirements to src/netCDF4.egg-info/requires.txt
  writing top-level names to src/netCDF4.egg-info/top_level.txt
  reading manifest file 'src/netCDF4.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no previously-included files found matching 'examples/data'
  warning: no previously-included files found matching 'src/netCDF4/_netCDF4.c'
  adding license file 'LICENSE'
  writing manifest file 'src/netCDF4.egg-info/SOURCES.txt'
  copying src/netCDF4/__init__.pyi -> build/lib.linux-x86_64-cpython-311/netCDF4
  copying src/netCDF4/_netCDF4.pyi -> build/lib.linux-x86_64-cpython-311/netCDF4
  copying src/netCDF4/_netCDF4.pyx -> build/lib.linux-x86_64-cpython-311/netCDF4
  copying src/netCDF4/py.typed -> build/lib.linux-x86_64-cpython-311/netCDF4
  copying src/netCDF4/plugins/empty.txt -> build/lib.linux-x86_64-cpython-311/netCDF4/plugins
  running build_ext
  building 'netCDF4._netCDF4' extension
  gcc -pthread -B /home/awikner/.conda/envs/netcdf4-test/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/awikner/.conda/envs/netcdf4-test/include -fPIC -O2 -isystem /home/awikner/.conda/envs/netcdf4-test/include -fPIC -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -DNC_COMPLEX_NO_EXPORT=1 -I/software/netcdf-4.7-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/include -I/software/hdf5-1.12.0-el8-x86_64+intelmpi-2019.up7+intel-19.1.1/include -I/home/awikner/.conda/envs/netcdf4-test/lib/python3.11/site-packages/numpy/_core/include -I/home/awikner/.conda/envs/netcdf4-test/lib/python3.11/site-packages/mpi4py/include -Iinclude -Iexternal/nc_complex/include -Iexternal/nc_complex/include/generated_fallbacks -I/home/awikner/.conda/envs/netcdf4-test/include/python3.11 -c external/nc_complex/src/nc_complex.c -o build/temp.linux-x86_64-cpython-311/external/nc_complex/src/nc_complex.o
  external/nc_complex/src/nc_complex.c: In function ‘pfnc_def_var’:
  external/nc_complex/src/nc_complex.c:637:21: error: ‘NC_FORMATX_NCZARR’ undeclared (first use in this function); did you mean ‘NC_FORMATX_ZARR’?
           || (mode == NC_FORMATX_NCZARR)) {
                       ^~~~~~~~~~~~~~~~~
                       NC_FORMATX_ZARR
  external/nc_complex/src/nc_complex.c:637:21: note: each undeclared identifier is reported only once for each function it appears in
  error: command '/usr/bin/gcc' failed with exit code 1
  error: subprocess-exited-with-error

  × Building wheel for netCDF4 (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/awikner/.conda/envs/netcdf4-test/bin/python /home/awikner/.conda/envs/netcdf4-test/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /home/awikner/tmpyco_yekk
  cwd: /home/awikner/netcdf4-python
  Building wheel for netCDF4 (pyproject.toml) ... error
  ERROR: Failed building wheel for netCDF4
Failed to build netCDF4
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (netCDF4)

Looking at the netcdf4.h file in the netcdf-c 4.7.4 source code, it indeed does not have a reference to NC_FORMATX_NCZARR. I assume this was added in a more recent version of netcdf-c.

Is there a way to build this version of netcdf4-python to be compatible with this version of netcdf-c? And if not, what was the most recent version that would support this version of netcdf-c?
@jswhit
Copy link
Collaborator

jswhit commented Sep 21, 2024 •
edited
Loading

Could you try this mod to nc_complex.c? If it works, I'll create a PR

--- external/nc_complex/src/nc_complex.c
+++ external/nc_complex/src/nc_complex.c
@@ -9,6 +9,11 @@

 #include "nc_complex_version.h"

+// to enable compilation with older versions of netcdf-c
+#ifndef NC_FORMATX_NCZARR
+#define NC_FORMATX_NCZARR    (10)
+#endif
+
 // NOLINTBEGIN(bugprone-assignment-in-if-condition)
 #define CHECK(func)           \
     do {

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jswhit @awikner