2

I'm working on a Python project with the following structure:

/
├── src/
│   ├── __init__.py
│   ├── afrr/
│   │   ├── __init__.py
│   │   ├── dumper.py
│   │   └── cleaner.py
│   ├── config.py
│   ├── tests/
│   │   ├── afrr/
│   │   │   ├── __init__.py
│   │   │   └── test_afrr_dumper.py

The test_afrr_dumper.py file is intended to test the dump_afrr_data function in afrr/dumper.py. The relevant code snippets are as follows:

src/afrr/dumper.py

import os
import pandas as pd
from datetime import datetime
from src.config import PROCESSED_DIR  # Ensure PROCESSED_DIR is defined in src/config.py

src/tests/afrr/test_afrr_dumper.py

from afrr.dumper import dump_afrr_data
from src.config import PROCESSED_DIR

When I attempt to run the test using either pytest or directly with Python, I encounter the following error:

ModuleNotFoundError: No module named 'src'

  1. Setting PYTHONPATH:

    set PYTHONPATH=src

    Verified with echo %PYTHONPATH%, but the error persists when running the test:

    pytest src/tests/afrr/test_afrr_dumper.py

  2. Adding __init__.py:

    • Added __init__.py files in the following directories to ensure they are treated as packages: 
      src/
src/afrr/
src/tests/afrr/

  3. Installing the Project as a Package:

    • Created a setup.py file: 
      from setuptools import setup, find_packages

setup(
    name="p",
    version="0.1",
    packages=find_packages(where="src"),
    package_dir={"": "src"},
    install_requires=[
        "pandas>=1.0.0",
    ],
)
    • Installed the project using: 
      pip install -e .

  4. Moving Test Files:

    • Moved test_afrr_dumper.py to src/ and ran: 
      pytest src/test_afrr_dumper.py
      Still encountered the same ModuleNotFoundError.

Expected Behavior

The src directory should be recognized as the root package, and the imports in test_afrr_dumper.py should resolve without issues. For example, from src.config import PROCESSED_DIR should work seamlessly.

Actual Behavior

Despite trying all of the above, Python consistently fails to recognize src as a module and raises `ModuleNotFoundError: No module named 'src'.

Environment Details

  • OS: Windows 10
  • Python Version: 3.12.8
  • Virtual Environment: venv created with python -m venv
  • Installed Dependencies:
    • pandas
    • pytest
    • Other standard libraries.

Questions

  1. How can I properly configure my project so that src is treated as the root module for imports?
  2. What am I missing in terms of Python package/module resolution?
  3. Is there a more structural, permanent fix for this issue without relying on setting PYTHONPATH manually?
Improve this question
3
  • Assuming hyperMVP is the intended name of your package, hyperMVP should be under src, and imports should be from hyperMVP..., and then you'd need to make your package installable with a suitable pyproject.toml, and install it as editable. Commented Jan 5 at 18:53
  • Anyway, it's all a little convoluted to explain, but I did open a PR that fixes these things up for you. github.com/hilbertp/hypermvp/pull/1 Commented Jan 5 at 19:05
  • For better explanations, though, maybe look at my previous answers mentioning sys.path and PYTHONPATH... Commented Jan 5 at 19:08

2 Answers 2

Reset to default
1

The src directory should be recognized as the root package,

This is not what you want to do. If all of the packages in the world did this, no one would have a good time.

How can I properly configure my project so that src is treated as the root module for imports?

You shouldn't. No Python import statement should ever contain the word src, unless you're maybe working on something that actually transforms source code. src should be above the package directory, a plain directory that contains all of the project's packages. This is called src layout.

What am I missing in terms of Python package/module resolution?

You haven't set up your package to be correctly installable, and the package's name really isn't your project's name (hypermvp).

Is there a more structural, permanent fix for this issue without relying on setting PYTHONPATH manually?

Yes - rejiggling your directory layout so src/ is the top level, then your project's package(s), and then subpackages. Following that, you can set up packaging (modernly with pyproject.toml and e.g. Hatch), so your package becomes installable. When your package is installable, you can install it as editable, with pip install -e ., which essentially links your working copy's src/hypermvp to your Python environment's import path, so edits you do are reflected in your running environment immediately.

I took the liberty of opening a PR that does this for you at https://github.com/hilbertp/hypermvp/pull/1.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

0

The way you have setup up your project means you can simply remove src. from your imports eg.

from src.afrr.dumper import ...

becomes:

from afrr.dumper import ...

Explanation

When you say packages=find_packages(where="src") in setup.py, what you are saying is that instead of treating your project root (path/to/hyperMVP) as the place to find packages, setuptools should instead treat path/to/hyperMVP/src as the place to find packages. And the path to your code from the src directory is stuff like ./afrr/__init__.py and so forth. That is, you were asking for a module called src.afrr, but that module does not exist. Instead what has been installed is simply afrr

Your test code will only work if you have installed your package into your virtual environment. An editable install is best/ This is so that changes to your source code are picked up when you rerun tests.

Without installing your package into the virtual environment, you would instead need to tell pytest where to find your source code. This is simple as putting a file called pytest.ini in the root of your project which contains the folllowing:

[pytest]
pythonpath = src

pyproject.toml

Both your setup.py and the pytest.ini configuration values can be put in a file called pyproject.toml. The principal advantage of this is that tools do not need to run your code to be able extract metadata about your project. A sample pyproject.toml that would replace both your setup.py and pytest.ini would be:

[build-system]
requires = ["setuptools >= 75"]
build-backend = "setuptools.build_meta"

[project]
name = "hypermvp"
version = "0.1"
dependencies = [
    "pandas >= 1.0.0",
]

[tool.setuptools.packages.find]
where = ["src"]
# files under src/tests are not to be distributed with your package
# You can still be discovered by pytest though
exclude = ["tests"]

[tool.pytest.ini_options]
pythonpath = ["src"]
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.