The first Scientific Python Developer Summit (May 22-26, 2023) brought together 34 developers at the eScience Institute at the University of Washington to develop shared infrastructure, documentation, tools, and recommendations for libraries in the Scientific Python ecosystem.
Prior to the summit we held several hour-long planning meetings:
- General (2023-02-27)
- May 15, Package metrics, DevStats
- May 15, SPECs
- May 18, Community & Documentation
- May 19, Build Systems & CI Infrastructure
- May 19, PyTest plugins & Sphinx extensions
At the summit, we had a brief check-in and then split into several groups based on each developers time and interests. Raw work progress and log have been collected in a document, we highlight just a few of things we accomplished below:
Almost a quarter of the group worked on sparse arrays for the entire week.
This work is part of a larger, multi-year effort to improve and expand SciPy’s
sparse array API, which will eventually
involve removing the sparse matrix API and eventually
More details can be found in the Developer Summit 1: Sparse blog post.
Scientific Python Ecosystem Coordination documents
We made significant progress on several SPECs, which had been drafted during previous sprints.
SPEC 2—API Dispatch was discussed (in a follow-up video meeting just after the summit)
and is in the process of being marked as
withdrawn or something similar.
SPEC 3—Accessibility was discussed and updated. We hope to see it endorsed by several core projects in the near future.
SPEC 4—Using a creating nightly wheels was rewritten, a helper GitHub action upload-nightly-action was created, and PRs to update the various projects to use the new nightly wheels location were made. The updates are now complete and the SPEC was endorsed by two core projects.
We anticipate several more core projects to endorse the existing SPECs over the coming months and we are now holding regular SPEC steering committee meetings to continue developing and expanding the SPECs.
We created a comprehensive community guide to empower projects in fostering their communities. This guide includes essential information on the role of community managers, along with practical strategies for community meetings, outreach, onboarding, and project management.
One of the fun things that happens at summits like these are the chance encounters of people from different projects. For example, a couple of attendees worked on creating a co-collaboration network across the broader scientific python ecosystem. This gave us the opportunity to look at how contributors collaborate across projects. We could see how the bigger projects were all clustered together as there are multiple contributors who share maintenance duties for multiple projects. We could also, for example, see how the Scikit-HEP cluster was a bit further away from the usual scientific Python cluster. An action item for us :) We need more collaboration!!
Pytest plugins/Sphinx extensions
Several attendees worked on pytest plugins and Sphinx extensions:
pytest-regex was created to support selecting tests with regular expressions.
pytest-doctestplus was moved upstream into the Scientific Python organization. The summit provided new momentum to develop new features (e.g. produce updated docstring), and to use it for the NumPy documentation testing.
sphinx-scientific-python, a new extension as a home for various features from the ecosystem, e.g., we agreed on bringing existing extensions from MNE tools to this extension.
SciPy release management progress
The first release candidate of SciPy 1.11.0 was published on PyPI
on May 31, 2023, five days after the conclusion of the summit. The
summit facilitated high-bandwidth decision making on several proposed
SciPy code changes by allowing the current SciPy release manager (Tyler Reddy,
Los Alamos National Laboratory) to consult with other SciPy core developers
in person. Specific code changes were discussed with the following SciPy
maintainers: Stefan van der Walt (
scipy.ndimage), CJ Carey (
Matt Haberland (
scipy.stats), and Pamphile Roy (
scipy.stats). When SciPy
releases are performed out of band from the summit, the release manager
will often have to delay incorporation of useful code changes to the
next release six months later, due to lack of availability of the
pertinent domain experts.