Deep Learning
Managing Python Dependencies with Poetry vs Conda & Pip
June 23, 2023
8 min read
Managing Dependencies for Your Python Projects
At the start of your deep learning and data science projects, manage the dependencies inside your project handling external libraries and packages required for a project to function correctly.
We got the classic Pip and Conda that many developers default use. For starters, Pip and Conda are great tools but as projects expand, so do the number of dependencies, making it challenging to keep up. Any programmer knows the headaches of wrongfully installed packages and incompatible dependencies can tirelessly frustrating. We are here to talk about Poetry, a more modern dependency library built on top of Pip, that expands on the capabilites and meaning of dependency management and provides a powerful tool for creating and maintaining Python projects.
Package Management
Poetry uses a pyproject.toml file to specify the configuration for your project accompanied by an automatically generated lockfile. The pyproject.toml file looks like this:
[tool.poetry.dependencies]
python = "^3.8"
pandas = "^1.5"
numpy = "^1.24.3"
[tool.poetry.dev.dependencies]
pytest = "^7.3.2"
precomit = "^3.3.3"
Like other dependency managers, it keeps track of the versions of each package in the current environment with an accompanied lockfile with project metadata, package version parameters, and more. Developers can separate the dependencies by dev-based and prod-based to keep deployment environment toml files more compact while reducing the risk of conflicts on, let's say, operating systems. Poetry’s pyproject.toml file addresses some shortcomings that Pip’s requirement.txt and Conda’s environment.yaml file.
Pip and Conda’s compacted list the dependencies is just a long list without metadata which is housed in a different file. They do not have a lock feature by default so if another user were to pip install a library needed to run your environment, it would install the latest version instead of your intended version found in your requirements.txt file. You can use the freeze function for Pip requirements.txt file but it makes the dependencies in your environment less flexible and too rigid, a counterintuitive approach for maintenance. Changes would need to be performed by manually editing the txt file.
Updating, Installing, and Removing Dependencies
With Poetry, updating libraries is simple and accounts for other dependencies to ensure they are up to date accordingly. Poetry has a mass update command that will update your dependencies (according to your toml file) while keeping all dependencies still compatible with one another and maintaining package version parameters inside found in the lock file. This will simultaneously update your lock file.
As for installation, it could not get any simpler. To install dependencies with Poetry you can use the poetry add function that you can either specify the version, use logic to specify version parameters (greater than less than), or use flags like @latest which will install the most recenter version of the package from PyPI. You can even group multiple packages in the same add function. Any newly installed package is automatically resolved to maintain the correct dependencies.
$poetry add requests pandas@latest
As for the classic dependency managers, let's test to see what happens when we try to install an older incompatible version. Pip-installed packages will output errors and conflicts but will ultimately still install the package which can lead to development that is not ideal. Conda does have a solver for errors in compatibility but immediately goes into search mode to solve the compatibility issue and only outputs an error when it cannot find a solution, instead of notifying the user directly.
(test-env) user:~$ pip install "numpy<1.18.5"
Collecting numpy<1.18.5
Downloading numpy-1.18.4-cp38-cp38-manylinux1_x86_64.whl (20.7 MB)
|████████████████████████████████| 20.7 MB 10.9 MB/s
Installing collected packages: numpy
Attempting uninstall: numpy
Found existing installation: numpy 1.22.3
Uninstalling numpy-1.22.3:
Successfully uninstalled numpy-1.22.3
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
pandas 1.4.2 requires numpy>=1.18.5; platform_machine != "aarch64" and platform_machine != "arm64" and python_version < "3.10", but you have numpy 1.18.4 which is incompatible.
Successfully installed numpy-1.18.4
(test-env) user:~$ pip list
Package Version
--------------- -------
numpy 1.18.4
pandas 1.4.2
pip 21.1.1
python-dateutil 2.8.2
pytz 2022.1
six 1.16.0
Poetry has an immediate response to dependency compatibility errors for prompt and early notice of conflicts. It refuses to continue the installation, so the user is now in charge of either finding a different version of the new package or the existing package.
user:~$ poetry add "numpy<1.18.5"
Updating dependencies
Resolving dependencies... (53.1s)
SolverProblemError
Because pandas (1.4.2) depends on numpy (>=1.18.5)
and no versions of pandas match >1.4.2,<2.0.0, pandas (>=1.4.2,<2.0.0) requires numpy (>=1.18.5).
So, because dependency-manager-test depends on both pandas (^1.4.2) and numpy (<1.18.5), version solving failed.
...
user:~$ poetry show
numpy 1.22.3 NumPy is the fundamental package for array computing with Python.
pandas 1.4.2 Powerful data structures for data analysis, time series, and statistics
python-dateutil 2.8.2 Extensions to the standard Python datetime module
pytz 2022.1 World timezone definitions, modern and historical
six 1.16.0 Python 2 and 3 compatibility utilities
Last but not least is Poetry’s uninstallation of packages. Some packages require more dependencies that are installed. For Pip, it removal of a package will only uninstall the defined package and nothing else. Conda will remove some packages but not all dependencies. Poetry on the other hand will remove the package and all its dependencies to keep your list of dependencies clutter free.
Is Poetry Compatible with Existing Pip or Conda Projects?
Yes, Poetry is compatible with existing projects managed by Pip or Conda. Just initialize your code using Poetry's Poetry.toml format and run it to grab the library of packages and its dependencies, allowing for a seamless transition.
If you have an existing project that uses Pip or Conda, you can migrate it to Poetry without much difficulty. Poetry uses its own pyproject.toml file to manage project dependencies and settings. To start using Poetry in your project, you can follow these steps:
1. Install Poetry either by curling and piping or using Pip
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
2. Navigate to the root directory of your existing project.
3. Initialize Poetry in your project directory:
poetry init
This command will guide you through a series of prompts to set up the initial configuration for your project.
4. Once the initialization is complete, Poetry will create the pyproject.toml in your project director. Open the toml file to add or modify your project's dependencies
5. To install the existing dependencies in your project to
poetry install
This will create a virtual environment and install the project dependencies within it.
6. You can now use the Poetry run command to execute your project's scripts, similar to how you would use Python or Conda commands.
poetry run python my_script.py
Poetry manages the virtual environment and dependency resolution for your project, making it compatible with existing Pip or Conda projects. It simplifies the management of dependencies and allows for consistent package installations across different environments.
Note: It's always a good practice to back up your project before making any significant changes to its configuration or dependency management tools.
Closing Thoughts
Making sure the right versions of the packages are in your code environment is essential for getting the right results every time. Slight changes to the backend of your code can alter the outcome. But also, keeping those packages and libraries up to date is just as important, to leverage the innovations each patch provides the next.
To manage these dependencies in your code, Poetry is a great tool for those working with more complex and diverse projects with a higher number of dependencies. While Pip and Conda are still viable options, they are more suited for smaller environments that are less complex. Not everyone might use Poetry, but since Pip has been around forever, it may be worth the ease of use to just use Pip.
But if your project and your workload value the importance of organization and are willing to explore new tools to improve your process, Poetry is a tool you should consider. The extended functionality from Pip to Poetry really makes a difference. We encourage you to try Poetry out for yourself.
Data Science and Deep Learning are making waves in the tech industry with new AI tools popping up left and right. Develop, train, and deploy your own AI models accelerated by an Exxact Deep Learning GPU solution, and boost your computing to achieve more.
Have any questions?
Contact Us Today!
Deep Learning
Managing Python Dependencies with Poetry vs Conda & Pip
June 23, 20238 min read
Managing Dependencies for Your Python Projects
At the start of your deep learning and data science projects, manage the dependencies inside your project handling external libraries and packages required for a project to function correctly.
We got the classic Pip and Conda that many developers default use. For starters, Pip and Conda are great tools but as projects expand, so do the number of dependencies, making it challenging to keep up. Any programmer knows the headaches of wrongfully installed packages and incompatible dependencies can tirelessly frustrating. We are here to talk about Poetry, a more modern dependency library built on top of Pip, that expands on the capabilites and meaning of dependency management and provides a powerful tool for creating and maintaining Python projects.
Package Management
Poetry uses a pyproject.toml file to specify the configuration for your project accompanied by an automatically generated lockfile. The pyproject.toml file looks like this:
[tool.poetry.dependencies]
python = "^3.8"
pandas = "^1.5"
numpy = "^1.24.3"
[tool.poetry.dev.dependencies]
pytest = "^7.3.2"
precomit = "^3.3.3"
Like other dependency managers, it keeps track of the versions of each package in the current environment with an accompanied lockfile with project metadata, package version parameters, and more. Developers can separate the dependencies by dev-based and prod-based to keep deployment environment toml files more compact while reducing the risk of conflicts on, let's say, operating systems. Poetry’s pyproject.toml file addresses some shortcomings that Pip’s requirement.txt and Conda’s environment.yaml file.
Pip and Conda’s compacted list the dependencies is just a long list without metadata which is housed in a different file. They do not have a lock feature by default so if another user were to pip install a library needed to run your environment, it would install the latest version instead of your intended version found in your requirements.txt file. You can use the freeze function for Pip requirements.txt file but it makes the dependencies in your environment less flexible and too rigid, a counterintuitive approach for maintenance. Changes would need to be performed by manually editing the txt file.
Updating, Installing, and Removing Dependencies
With Poetry, updating libraries is simple and accounts for other dependencies to ensure they are up to date accordingly. Poetry has a mass update command that will update your dependencies (according to your toml file) while keeping all dependencies still compatible with one another and maintaining package version parameters inside found in the lock file. This will simultaneously update your lock file.
As for installation, it could not get any simpler. To install dependencies with Poetry you can use the poetry add function that you can either specify the version, use logic to specify version parameters (greater than less than), or use flags like @latest which will install the most recenter version of the package from PyPI. You can even group multiple packages in the same add function. Any newly installed package is automatically resolved to maintain the correct dependencies.
$poetry add requests pandas@latest
As for the classic dependency managers, let's test to see what happens when we try to install an older incompatible version. Pip-installed packages will output errors and conflicts but will ultimately still install the package which can lead to development that is not ideal. Conda does have a solver for errors in compatibility but immediately goes into search mode to solve the compatibility issue and only outputs an error when it cannot find a solution, instead of notifying the user directly.
(test-env) user:~$ pip install "numpy<1.18.5"
Collecting numpy<1.18.5
Downloading numpy-1.18.4-cp38-cp38-manylinux1_x86_64.whl (20.7 MB)
|████████████████████████████████| 20.7 MB 10.9 MB/s
Installing collected packages: numpy
Attempting uninstall: numpy
Found existing installation: numpy 1.22.3
Uninstalling numpy-1.22.3:
Successfully uninstalled numpy-1.22.3
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
pandas 1.4.2 requires numpy>=1.18.5; platform_machine != "aarch64" and platform_machine != "arm64" and python_version < "3.10", but you have numpy 1.18.4 which is incompatible.
Successfully installed numpy-1.18.4
(test-env) user:~$ pip list
Package Version
--------------- -------
numpy 1.18.4
pandas 1.4.2
pip 21.1.1
python-dateutil 2.8.2
pytz 2022.1
six 1.16.0
Poetry has an immediate response to dependency compatibility errors for prompt and early notice of conflicts. It refuses to continue the installation, so the user is now in charge of either finding a different version of the new package or the existing package.
user:~$ poetry add "numpy<1.18.5"
Updating dependencies
Resolving dependencies... (53.1s)
SolverProblemError
Because pandas (1.4.2) depends on numpy (>=1.18.5)
and no versions of pandas match >1.4.2,<2.0.0, pandas (>=1.4.2,<2.0.0) requires numpy (>=1.18.5).
So, because dependency-manager-test depends on both pandas (^1.4.2) and numpy (<1.18.5), version solving failed.
...
user:~$ poetry show
numpy 1.22.3 NumPy is the fundamental package for array computing with Python.
pandas 1.4.2 Powerful data structures for data analysis, time series, and statistics
python-dateutil 2.8.2 Extensions to the standard Python datetime module
pytz 2022.1 World timezone definitions, modern and historical
six 1.16.0 Python 2 and 3 compatibility utilities
Last but not least is Poetry’s uninstallation of packages. Some packages require more dependencies that are installed. For Pip, it removal of a package will only uninstall the defined package and nothing else. Conda will remove some packages but not all dependencies. Poetry on the other hand will remove the package and all its dependencies to keep your list of dependencies clutter free.
Is Poetry Compatible with Existing Pip or Conda Projects?
Yes, Poetry is compatible with existing projects managed by Pip or Conda. Just initialize your code using Poetry's Poetry.toml format and run it to grab the library of packages and its dependencies, allowing for a seamless transition.
If you have an existing project that uses Pip or Conda, you can migrate it to Poetry without much difficulty. Poetry uses its own pyproject.toml file to manage project dependencies and settings. To start using Poetry in your project, you can follow these steps:
1. Install Poetry either by curling and piping or using Pip
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
2. Navigate to the root directory of your existing project.
3. Initialize Poetry in your project directory:
poetry init
This command will guide you through a series of prompts to set up the initial configuration for your project.
4. Once the initialization is complete, Poetry will create the pyproject.toml in your project director. Open the toml file to add or modify your project's dependencies
5. To install the existing dependencies in your project to
poetry install
This will create a virtual environment and install the project dependencies within it.
6. You can now use the Poetry run command to execute your project's scripts, similar to how you would use Python or Conda commands.
poetry run python my_script.py
Poetry manages the virtual environment and dependency resolution for your project, making it compatible with existing Pip or Conda projects. It simplifies the management of dependencies and allows for consistent package installations across different environments.
Note: It's always a good practice to back up your project before making any significant changes to its configuration or dependency management tools.
Closing Thoughts
Making sure the right versions of the packages are in your code environment is essential for getting the right results every time. Slight changes to the backend of your code can alter the outcome. But also, keeping those packages and libraries up to date is just as important, to leverage the innovations each patch provides the next.
To manage these dependencies in your code, Poetry is a great tool for those working with more complex and diverse projects with a higher number of dependencies. While Pip and Conda are still viable options, they are more suited for smaller environments that are less complex. Not everyone might use Poetry, but since Pip has been around forever, it may be worth the ease of use to just use Pip.
But if your project and your workload value the importance of organization and are willing to explore new tools to improve your process, Poetry is a tool you should consider. The extended functionality from Pip to Poetry really makes a difference. We encourage you to try Poetry out for yourself.
Data Science and Deep Learning are making waves in the tech industry with new AI tools popping up left and right. Develop, train, and deploy your own AI models accelerated by an Exxact Deep Learning GPU solution, and boost your computing to achieve more.
Have any questions?
Contact Us Today!
