Testing Suite#

NeuroConv verifies the integrity of all code changes by running a full test suite on short examples of real data from the formats we support. The testing suite is broken up into sub-folders based on the scope of functionalities and dependencies you wish to test. We recommend always running tests in a fresh environment to ensure errors are not the result of contaminated dependencies.

There are several categories of tests in the NeuroConv codebase:

  1. Minimal Tests: Core functionality tests. These tests should work only with base dependencies.

  2. Modality Tests: Tests for machinery of the different data modalities (ecephys, ophys, etc.).

  3. Example Data Tests: Tests that run on real data examples. This needs the full dependencies and data downloaded from gin.

  4. Remote Transfer Services: Tests for external cloud service integrations

  5. Import Structure Tests: Tests that verify the import structure of the package and ensure that top level packages can be imported with minimal installation.

Run all tests#

To run all tests, first clone the repo and cd into it.

git clone https://github.com/catalystneuro/neuroconv.git
cd neuroconv

Then install all required and optional dependencies in a fresh environment.

pip install --editable ".[full]"
pip install --group test

Then simply run all tests with pytest

pytest

Note

You will likely observe many failed tests if the test data is not available. See the section ‘Testing on Example Data’ for instructions on how to download the test data.

Minimal Tests#

These test internal functionality using only minimal dependencies or pre-downloaded data.

Sub-folders: tests/test_minimal

These can be run using only pip install --editable ".", pip install --group test and calling pytest tests/test_minimal

Modality Tests#

These test the functionality of our write tools tailored to specific modalities such as electrophysiology, optical physiology, behavior, etc. The tests are broken up into sub-folders based on the modality being tested.

Modalities:

These can be run in isolation using pip install --editable ".[<modality>]", pip install --group test and calling pytest tests/test_modalities/test_<modality> where <modality> can be any of ophys, fiber_photometry, ecephys, image, text, or behavior.

Ideally, these tests should not require data and run in mock testing interfaces but there are exceptions.

Testing On Example Data#

For proprietary formats, we regularly test our conversions against small snippets of real data, stored somewhere on your local system. These can each by downloaded using Datalad

For electrophysiology data#

datalad install -rg https://gin.g-node.org/NeuralEnsemble/ephy_testing_data

For optical physiology data#

datalad install -rg https://gin.g-node.org/CatalystNeuro/ophys_testing_data

For behavioral data#

datalad install -rg https://gin.g-node.org/CatalystNeuro/behavior_testing_data

Running the data tests#

Once the data is downloaded to your system, you must manually modify the testing config file (example). This file should be located and named as tests/test_on_data/gin_test_config.json whenever neuroconv is installed in editable -e mode). The LOCAL_PATH field points to the folder on your system that contains the dataset folder (e.g., ephy_testing_data for testing ecephys). The code will automatically detect that the tests are being run locally, so all you need to do ensure the path is correct in your specific system.

The output of these tests is, by default, stored in a temporary directory that is then cleaned after the tests finish running. To examine these files for quality assessment purposes, set the flag SAVE_OUTPUTS=true in the gin_test_config.json file and modify the variable OUTPUT_PATH in the respective test if necessary.

Sub-folders: tests/test_on_data

These can be run in total using pip install --editable ".[full]", pip install --group test and calling pytest tests/test_on_data or in isolation by installing the required <modality> as in the previous section and calling pytest tests/test_on_data/<modality>.

Update existing test data#

If you have downloaded these data repositories previously and want to update them, cd into the directory you want to update and run

datalad update --how=ff-only --reobtain-data

To update GIN data, run the command above within the repository you would like to update.

Remote Transfer Services#

These tests verify the functionality of tools that interact with external cloud services for data transfer and storage operations. They require actual credentials and API keys to communicate with live services such as AWS, DANDI, and Globus.

Important: These tests are not automatically collected by pytest’s default collection mechanism because they don’t follow the “test_” naming convention in their filenames. This is intentional to prevent them from running during regular test runs, as they require specific credentials and can take longer to execute.

Sub-folders: tests/remote_transfer_services

Required credentials#

To run these tests, you need to set up the following environment variables:

  • For DANDI tests: DANDI_API_KEY

  • For AWS tests: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

  • For Globus tests: Appropriate credentials as documented in the test files

Running remote transfer tests#

Since these tests are not automatically collected, you need to run them explicitly:

# Install required dependencies
pip install --editable ".[aws]"
pip install --group test

# Run specific service tests
pytest tests/remote_transfer_services/dandi_transfer_tools.py
pytest tests/remote_transfer_services/aws_tools_tests.py
pytest tests/remote_transfer_services/globus_transfer_tools.py
pytest tests/remote_transfer_services/yaml_dandi_transfer_tools.py

Import Structure Tests#

The tests/imports.py file contains tests that verify the import structure of the NeuroConv package. These tests ensure that the package can be imported correctly and that all expected modules and attributes are available in the correct namespaces.

These tests are particularly important for ensuring that the package’s public API remains stable and that dependencies are correctly managed. They verify that:

  1. The top-level package imports expose the expected classes and functions

  2. The tools submodule contains all required utilities

  3. The datainterfaces submodule correctly exposes all interface classes

To run these tests specifically:

pytest tests/imports.py::TestImportStructure::test_top_level
pytest tests/imports.py::TestImportStructure::test_tools
pytest tests/imports.py::TestImportStructure::test_datainterfaces