Incompatibility with Shapely >2 Due to NumPy 2.0 Update

[mjanez]

We have identified a compatibility issue with ckanext-spatial when using Shapely versions higher than 2 (see Error logs), due to the recent release of NumPy 2.0. This issue arises because NumPy 2.0 introduces changes to the C ABI that require rebuilding packages that use the NumPy C API, such as Shapely, to import it, so it is recommended to add this upper pin now for releases.

Refs

• NumPy v2.0.0 has just been released.
• Compatibility with NumPy 2.0 shapely/shapely#1972
• Ecosystem compatibility with numpy 2.0 numpy/numpy#26191

Todo

To address the compatibility issues between ckanext-spatial and the recent release of NumPy 2.0, we propose the following modifications to the requirements file:

• We added a pin for NumPy to ensure compatibility:
  • numpy>=1.26,<2: This ensures that the version of NumPy used is compatible with both current and upcoming changes, but avoids the breaking changes introduced in NumPy 2.0.
• Shapely version is set to 2.0.4, which is confirmed to be compatible with the specified NumPy range.

ckantoolkit
lxml>=2.3
argparse
pyparsing>=2.1.10
requests>=1.1.0
cython==0.29.36; python_version < '3.9'
pyproj==2.6.1; python_version < '3.9'
pyproj==3.6.1; python_version >= '3.9'

# Compatibility with NumPy 2.0: https://github.com/shapely/shapely/issues/1972
numpy>=1.26,<2
Shapely==2.0.4

OWSLib==0.28.1
geojson==3.0.1

Error logs

2024-06-17 07:49:32,637 INFO  [ckan.cli] Using configuration file /srv/app/ckan.ini
2024-06-17 07:49:32,638 INFO  [ckan.config.environment] Loading static files from public
2024-06-17 07:49:32,808 WARNI [ckan.config.environment] Setting CKAN_DB as an env var is deprecated and will be removed in a future release. Use CKAN_SQLALCHEMY_URL instead.
2024-06-17 07:49:32,843 INFO  [ckan.config.environment] Loading templates from /srv/app/src/ckan/ckan/templates

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last):  File "/usr/bin/ckan", line 8, in <module>
    sys.exit(ckan())
  File "/usr/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 781, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/usr/lib/python3.9/site-packages/click/core.py", line 700, in make_context
    self.parse_args(ctx, args)
  File "/srv/app/src/ckan/ckan/cli/cli.py", line 116, in parse_args
    result = super(ExtendableGroup, self).parse_args(ctx, args)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1212, in parse_args
    rest = Command.parse_args(self, ctx, args)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1048, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1630, in handle_parse_result
    value = invoke_param_callback(self.callback, ctx, self, value)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 123, in invoke_param_callback
    return callback(ctx, param, value)
  File "/srv/app/src/ckan/ckan/cli/cli.py", line 126, in _init_ckan_config
    _add_ctx_object(ctx, value)
  File "/srv/app/src/ckan/ckan/cli/cli.py", line 135, in _add_ctx_object
    ctx.obj = CtxObject(path)
  File "/srv/app/src/ckan/ckan/cli/cli.py", line 57, in __init__
    self.app = make_app(self.config)
  File "/srv/app/src/ckan/ckan/config/middleware/__init__.py", line 56, in make_app
    load_environment(conf)
  File "/srv/app/src/ckan/ckan/config/environment.py", line 123, in load_environment
    p.load_all()
  File "/srv/app/src/ckan/ckan/plugins/core.py", line 165, in load_all
    load(*plugins)
  File "/srv/app/src/ckan/ckan/plugins/core.py", line 179, in load
    service = _get_service(plugin)
  File "/srv/app/src/ckan/ckan/plugins/core.py", line 281, in _get_service
    return plugin.load()(name=plugin_name)
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2450, in load
    return self.resolve()
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2456, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/srv/app/src/ckanext-spatial/ckanext/spatial/plugin/__init__.py", line 7, in <module>
    import shapely.geometry
  File "/usr/lib/python3.9/site-packages/shapely/__init__.py", line 1, in <module>
    from .lib import GEOSException  # NOQA
AttributeError: _ARRAY_API not found
Traceback (most recent call last):
  File "/usr/bin/ckan", line 8, in <module>
    sys.exit(ckan())
  File "/usr/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 781, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/usr/lib/python3.9/site-packages/click/core.py", line 700, in make_context
    self.parse_args(ctx, args)
  File "/srv/app/src/ckan/ckan/cli/cli.py", line 116, in parse_args
    result = super(ExtendableGroup, self).parse_args(ctx, args)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1212, in parse_args
    rest = Command.parse_args(self, ctx, args)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1048, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1630, in handle_parse_result
    value = invoke_param_callback(self.callback, ctx, self, value)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 123, in invoke_param_callback
    return callback(ctx, param, value)
  File "/srv/app/src/ckan/ckan/cli/cli.py", line 126, in _init_ckan_config
    _add_ctx_object(ctx, value)
  File "/srv/app/src/ckan/ckan/cli/cli.py", line 135, in _add_ctx_object
    ctx.obj = CtxObject(path)
  File "/srv/app/src/ckan/ckan/cli/cli.py", line 57, in __init__
    self.app = make_app(self.config)
  File "/srv/app/src/ckan/ckan/config/middleware/__init__.py", line 56, in make_app
    load_environment(conf)
  File "/srv/app/src/ckan/ckan/config/environment.py", line 123, in load_environment
    p.load_all()
  File "/srv/app/src/ckan/ckan/plugins/core.py", line 165, in load_all
    load(*plugins)
  File "/srv/app/src/ckan/ckan/plugins/core.py", line 179, in load
    service = _get_service(plugin)
  File "/srv/app/src/ckan/ckan/plugins/core.py", line 281, in _get_service
    return plugin.load()(name=plugin_name)
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2450, in load
    return self.resolve()
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2456, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/srv/app/src/ckanext-spatial/ckanext/spatial/plugin/__init__.py", line 7, in <module>
    import shapely.geometry
  File "/usr/lib/python3.9/site-packages/shapely/__init__.py", line 1, in <module>
    from .lib import GEOSException  # NOQA
ImportError: numpy.core.multiarray failed to import