Conda-Forge

[benjaminrwilson]

Hi @rossant,

Thanks for adding the experimental .whl support: 4f9d13f. Have you considered adding a recipe to https://conda-forge.org? I've had good experiences with their CI/CD and they support building against a large matrix of different python versions + operating systems. Curious on your thoughts.

Ben

[rossant]

Great idea! Definitely on my to-do list, and I'd love to get some help :)
Do you know if conda-forge can build Python wheels as well? It would be great to have a CI/CD service building automatically Python wheels and conda packages for all Python versions and operating systems.

[djhoese]

@rossant Typically what conda-forge does is downloads a source tarball and compiles it from scratch on their CI services. The resulting conda package contains all of the binaries and directory structure (and metadata) to be placed in the final conda environment. They do not currently generate wheels, but if the process is repeatable and semi-automated (the dependencies can be easily installed) then it shouldn't be too bad to do with github actions. We've got this working with vispy if you want to discuss it some time.

The building of conda packages must be done on conda-forge's systems (or at least with their docker images if the process is resource intensive) so that it is binary compatible with their specific library versions.

[rossant]

If I understand correctly, you mean that:

• GitHub Actions would build and upload pip wheels
• conda-forge would build conda packages

?

[djhoese]

Yes.

1. @rossant pushes tag and makes a github release ->
2. GitHub Actions (CI) build sdist tarball and binary wheels -> CI Uploads to PyPI
3. Conda-forge bots see the new version on PyPI (takes a couple hours) -> Bots create new version PR on conda-forge feedstock repository ->
4. conda-forge PR builds conda package for all supported python/OS versions ->
5. @rossant merges conda-forge PR -> conda-forge package is available to download a few hours later ->
6. User does conda install -c conda-forge datoviz

Example conda-forge feedstock PR created by bot for vispy: conda-forge/vispy-feedstock#26

[djhoese]

To clarify, it is common to have github actions (or any CI) deploy packages based on tags and/or github releases. I have some projects that upload packages to test.pypi.org when I make a git tag and only upload to the "real" PyPI when I make a GitHub release.

[rossant]

Looks great! So the first step is to have a GitHub action building sdist tarball and binary wheels and uploading them to PyPI. Any help will be much appreciated! I can give directions on how to build the wheels: it's tricky.

• manage command for creating the wheel on Unix systems
• for the manylinux wheel: wheel creation script to run within a manylinux docker container