Introduction
Resolving conflicting dependencies can be a real headache in python applications. Different parts of the codebase might require different versions of the same package, creating a seemingly impossible situation. While virtual environments help with project-level isolation, they don't solve the problem of conflicting dependencies within a single application. In this article, we'll explore a proof-of-concept approach that uses Python's subprocess mechanism to run specific functions with isolated dependencies. This is purely for exploration and is not at all recommended to be productionized, unless you are ready for late night pagers!
Glossary
Subprocess: A separate process spawned by a parent process, with its own memory space and environment
Runtime Environment: The set of dependencies available to a function when it executes
Dependency Isolation: Running code with specific dependencies regardless of what's installed in the parent process
The Challenge
Consider this scenario: one part of your application needs requests==2.22.0 while another requires requests==2.32.2. How do you satisfy both requirements within the same Python process? The initial thoughts are:
Compromise: Choose one version and hope everything works
Refactoring: Rewrite code to work with a single version
Let us explore a third option: run specific functions in isolated subprocesses with their own dependencies.
The Subprocess Solution
The core idea is: when a function needs specific dependencies, we:
Spawn a new Python subprocess
Install the required dependencies in a temporary directory
Execute the function code in this subprocess
Capture the result and return it to the parent process
This approach leverages the fact that subprocesses have their own memory space and Python environment, allowing for true isolation without affecting the parent process.
Key Components
Our solution consists of two main parts:
A decorator (runtime_env) that wraps functions to execute in isolated subprocesses
A subprocess runner that handles dependency installation and function execution
Here's how the decorator is used:
from decorator import runtime_env
@runtime_env("requirements.txt")
def process_data(data, factor=1):
import requests
result = {
"original": data,
"processed": {"name": data["name"].upper(), "value": data["value"] * factor},
"metadata": {"requests_version": requests.__version__}
}
return result
How It Works
When the decorated function is called, the following sequence happens:
Function identification: The decorator captures the source code of the module containing the function
Argument serialization: Function arguments are serialized using Python's pickle module
Subprocess creation: A new Python process is started, running our subprocess runner
Dependency installation: The subprocess installs required packages in a temporary directory
Function execution: The function runs in the isolated environment with its own dependencies
Result communication: The result is serialized and passed back to the parent process
Cleanup: Temporary directories and files are removed
Example in Action
Let's explore a concrete example. The parent process might have requests==2.26.0 installed, while we want a function to run with requests==2.28.0:
def get_data():
try:
import requests
print(f"Parent process using requests version: {requests.__version__}")
except ImportError:
print("Requests not available in parent")
return {"name": "Test Data", "value": 42}
@runtime_env("requirements.txt") # Contains requests==2.28.0
def process_data(data):
import requests
print(f"Subprocess using requests version: {requests.__version__}")
return {"result": data, "version_used": requests.__version__}
input_data = get_data()
result = process_data(input_data)
print(f"Process completed with requests version: {result['version_used']}")
When executed, we see different versions of the requests library being used in each process, demonstrating successful isolation. Here’s a screenshot from the PoC that we did:
As you can see in the example above, the parent process PID is 1, where as the subprocess PID is 7, and for both the parent process and subprocess, the requests library version is different.
Behind the Scenes
I won’t be delving in the exact approach here, I would let you take a stab at it, and yes, use LLMs to help you solution! However, here’s a summary of the approach at a very high level:
The decorator first identifies the module to which the function belongs, and get the module via the sys package, and uses the inspect module to fetch the source file. We need the module in order to ensure that we can also use the imports used for the decorated function.
Once we have the code information, we use the subprocess module to create a new process, and we create a new temporary directory in which we install the required dependencies
Now we execute the function and return the output back to the caller process.
Only for fun, not for production!
This approach is only for educational purposes, the primary reason being, this approach is very naive, and there are a lot of aspects to cover even to make it dev ready, for example, the protocol or mechanism used to communicate the results back to the parent process. What about concurrent processes creating new isolated envs? What about common dependencies between the parent and the child process? And what not!
Explore something production ready?
To checkout an implementation that deals with conflicting dependencies you can read about Ray's Runtime Environments feature, which provides a more robust, optimized approach to dependency isolation with additional features like package caching and environment reuse.
Conclusion
Python’s subprocess is a powerful module. The example above reveals the power of process isolation as a solution to conflicting dependencies, a common challenge in modern Python development. While this is a proof-of-concept, nevertheless, understanding the underlying principles of process isolation provides valuable insight into how dependency conflicts can be managed in complex applications.
If you want to explore the source code, feel free to get in touch! However, I strongly recommend doing it yourself, with LLMs at your disposal, should be an hour of effort!
