If you’re interested in getting involved in the development of Modin, but aren’t sure where start, take a look at the issues tagged Good first issue or Documentation. These are issues that would be good for getting familiar with the codebase and better understanding some of the more complex components of the architecture. There is documentation here about the architecture that you will want to review in order to get started.
Also, feel free to join the discussions on the developer mailing list.
Certificate of Origin¶
To keep a clear track of who did what, we use a sign-off procedure (same requirements for using the signed-off-by process as the Linux kernel has https://www.kernel.org/doc/html/v4.17/process/submitting-patches.html) on patches or pull requests that are being sent. The sign-off is a simple line at the end of the explanation for the patch, which certifies that you wrote it or otherwise have the right to pass it on as an open-source patch. The rules are pretty simple: if you can certify the below:
CERTIFICATE OF ORIGIN V 1.1¶
“By making a contribution to this project, I certify that:
1.) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or 2.) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or 3.) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it. 4.) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved.”
This is my commit message Signed-off-by: Awesome Developer <email@example.com>
Code without a proper signoff cannot be merged into the master branch. Note: You must use your real name (sorry, no pseudonyms or anonymous contributions.)
The text can either be manually added to your commit body, or you can add either
--signoff to your usual
git commit commands:
git commit --signoff git commit -s
This will use your default git configuration which is found in .git/config. To change this, you can use the following commands:
git config --global user.name "Awesome Developer" git config --global user.email "awesome.developer.@example.org"
If you have authored a commit that is missing the signed-off-by line, you can amend your commits and push them to GitHub.
git commit --amend --signoff
If you’ve pushed your changes to GitHub already you’ll need to force push your branch
after this with
git push -f.
Commit Message formatting¶
To ensure that all commit messages in the master branch follow a specific format, we enforce that all commit messages must follow the following format:
FEAT-#9999: Add `DataFrame.rolling` functionality, to enable rolling window operations
FEAT component represents the type of commit. This component of the commit
message can be one of the following:
- FEAT: A new feature that is added
- DOCS: Documentation improvements or updates
- FIX: A bugfix contribution
- REFACTOR: Moving or removing code without change in functionality
- TEST: Test updates or improvements
#9999 component of the commit message should be the issue number in the Modin
GitHub issue tracker: https://github.com/modin-project/modin/issues. This is important
because it links commits to their issues.
The commit message should follow a colon (:) and be descriptive and succinct.
We recommend doing development in a virtualenv or conda environment, though this decision is ultimately yours. You will want to run the following in order to install all of the required dependencies for running the tests and formatting the code:
conda env create --file environment-dev.yml # or pip install -r requirements-dev.txt
Code Formatting and Lint¶
We use black for code formatting. Before you submit a pull request, please make sure that you run the following from the project root:
We also use flake8 to check linting errors. Running the following from the project root will ensure that it passes the lint checks on Github Actions:
We test that this has been run on our Github Actions test suite. If you do this and find that the tests are still failing, try updating your version of black and flake8.
Adding a test¶
If you find yourself fixing a bug or adding a new feature, don’t forget to add a test to the test suite to verify its correctness! More on testing and the layout of the tests can be found in our testing documentation. We ask that you follow the existing structure of the tests for ease of maintenance.
Running the tests¶
To run the entire test suite, run the following from the project root:
The test suite is very large, and may take a long time if you run every test. If you’ve only modified a small amount of code, it may be sufficient to run a single test or some subset of the test suite. In order to run a specific test run:
The entire test suite is automatically run for each pull request.
We use Asv tool for performance tracking of various Modin functionality.
Here are some scenarios in which Asv can be used:
- It is necessary to check the impact of the new patch on the performance of a certain set of operations:
asv continuous -f 1.05 src/master HEAD -b TimeGroupBy --launch-method=spawn
- It is necessary to check presence of errors inside of benchmarks after making changes or writing new ones:
asv run --quick --show-stderr --python=same --launch-method=spawn
- You just need to run the entire test suite to get the current time numbers:
asv run --launch-method=spawn
- It is necessary to check the range of commits for performance degradation:
asv run [start_hash]..[end_hash] --launch-method=spawn asv publish asv preview
For more consistent results, you may need to use the following parameters:
Some details about using Modin on Ray with Asv:
--launch-method=forkserveris not working
- Each set of parameters for each test is launched in its own process, which brings a large overhead, since for each process redis server and other necessary binaries from ray initialization are started and destroyed.
Some details for maintenance:
modin/asv_bench/asv.conf.jsoncontains the modin dependencies, with which testing takes place.
We need to keep them up to date according to the dependencies in
To build the documentation, please follow the steps below from the project root:
cd docs pip install -r requirements-doc.txt sphinx-build -b html . build
To visualize the documentation locally, run the following from build folder:
python -m http.server <port> # python -m http.server 1234
then open the browser at 0.0.0.0:<port> (e.g. 0.0.0.0:1234).
Contributing a new execution framework or in-memory format¶
If you are interested in contributing support for a new execution framework or in-memory format, please make sure you understand the architecture of Modin.
The best place to start the discussion for adding a new execution framework or in-memory format is the developer mailing list.
More docs on this coming soon…