feat: dev-environments - python

This commit is contained in:
Ryan Yin 2023-07-18 13:13:24 +08:00
parent 8a517a6b36
commit 8d4710b132
2 changed files with 113 additions and 0 deletions

View file

@ -9,3 +9,65 @@ Luckily, some people in the community have done this for us. The following repos
If you think the structure of `flake.nix` is still too complicated and want a simpler way, you can also consider using the following project, which encapsulates Nix more thoroughly and provides users with a simpler definition:
- [cachix/devenv](https://github.com/cachix/devenv)
## Dev Environment for Python
The development environment for Python is much more cumbersome compared to languages like Java or Go because it defaults to installing software in the global environment.
To install software for the current project, you must create a virtual environment first (unlike in languages such as JavaScript or Go,
where virtual environments are not necessary). This behavior is very unfriendly for Nix.
By default, when using pip in Python, it installs software globally. On NixOS, running `pip install` directly will result in an error:
```bash
pip install -r requirements.txt
error: externally-managed-environment
× This environment is externally managed
╰─> This command has been disabled as it tries to modify the immutable
`/nix/store` filesystem.
To use Python with Nix and nixpkgs, have a look at the online documentation:
<https://nixos.org/manual/nixpkgs/stable/#python>.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
```
Based on the error message, `pip install` is directly disabled by NixOS. Even when attempting `pip install --user`, it is similarly disabled.
To improve the reproducibility of the environment, Nix eliminates these commands altogether.
Even if we create a new environment using methods like `mkShell`,
these commands still result in errors (presumably because the pip command in Nixpkgs itself has
been modified to prevent any modification instructions like `install` from running).
However, many project installation scripts are based on pip, which means these scripts cannot be used directly.
Additionally, the content in nixpkgs is limited, and many packages from PyPI are missing.
This requires users to package them themselves, adding a lot of complexity and mental burden.
One solution is to use the `venv` virtual environment. Within a virtual environment, you can use commands like pip normally:
```shell
python -m venv ./env
source ./env/bin/activate
```
Alternatively, you can use a third-party tool called `virtualenv`, but this requires additional installation.
For those who still lack confidence in the venv created directly with Python, they may prefer to include the virtual environment in `/nix/store` to make it immutable.
This can be achieved by directly installing the dependencies from `requirements.txt` or `poetry.toml` using Nix.
There are existing Nix packaging tools available to assist with this:
> Note that even in these environments, running commands like `pip install` directly will still fail.
Python dependencies must be installed through `flake.nix` because the data is located in the `/nix/store` directory,
and these modification commands can only be executed during the Nix build phase.
- [DavHau/mach-nix](https://github.com/DavHau/mach-nix)
- [poetry2nix](https://github.com/nix-community/poetry2nix)
The advantage of these tools is that they utilize the lock mechanism of Nix Flakes to improve reproducibility.
However, the downside is that they add an extra layer of abstraction, making the underlying system more complex.
Finally, in some more complex projects, neither of the above solutions may be feasible.
In such cases, the best solution is to use containers such as Docker or Podman. Containers have fewer restrictions compared to Nix and can provide the best compatibility.

View file

@ -9,3 +9,54 @@
如果你觉得 `flake.nix` 的结构还是太复杂了,希望能有更简单的方法,也可以考虑使用下面这个项目,它对 Nix 做了更彻底的封装,对用户提供了更简单的定义:
- [cachix/devenv](https://github.com/cachix/devenv)
## Python 开发环境
Python 的开发环境比 Java/Go 等语言要麻烦许多因为它默认就往全局环境装软件要往当前项目装还必须得先创建虚拟环境JS/Go 等语言里可没虚拟环境这种幺蛾子)。
这对 Nix 而言是非常不友好的行为。
Python 的 pip 默认会将软件安装到全局,在 NixOS 中 `pip install` 会直接报错:
```shell
pip install -r requirements.txt
error: externally-managed-environment
× This environment is externally managed
╰─> This command has been disabled as it tries to modify the immutable
`/nix/store` filesystem.
To use Python with Nix and nixpkgs, have a look at the online documentation:
<https://nixos.org/manual/nixpkgs/stable/#python>.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
```
根据错误信息,`pip install` 直接被 NixOS 禁用掉了,测试了 `pip install --user` 也同样被禁用。为了提升环境的可复现能力Nix 把它们全部废掉了。
即使我们通过 `mkShell` 等方式创建一个新环境,这些命令照样会报错(猜测是 Nixpkgs 中的 pip 命令本身就被魔改了,只要是跑 `install` 等修改指令就直接嘎掉)。
但是很多项目的安装脚本都是基于 pip 的,这导致这些脚本都不能直接使用,而且另一方面 nixpkgs 中的内容有限,很多 pypi 中的包里边都没有,还得自己打包,相对麻烦很多,也加重了用户的心智负担。
解决方案之一是改用 `venv` 虚拟环境,在虚拟环境里当然就能正常使用 pip 等命令了:
```shell
python -m venv ./env
source ./env/bin/activate
```
或者使用第三方工具 `virtualenv`,缺点是这个需要额外安装。
这样用 python 直接创建的 venv对一些人而言可能还是没有安全感仍然希望将这个虚拟环境也弄进 `/nix/store` 里使其不可变,通过 nix 直接安装 `requirements.txt` 或者 `poetry.toml` 中的依赖项。
这当然是可行的,有现成的 Nix 封装工具帮我们干这个活:
> 注意即使是在这俩环境中,直接跑 `pip install` 之类的安装命令仍然是会失败的,必须通过 `flake.nix` 来安装 Python 依赖!
因为数据还是在 `/nix/store` 中,这类修改命令必须在 Nix 的构建阶段才能执行...
- [DavHau/mach-nix](https://github.com/DavHau/mach-nix)
- [poetry2nix](https://github.com/nix-community/poetry2nix)
这俩工具的好处是,能利用上 Nix Flakes 的锁机制来提升可复现能力,缺点是多了一层封装,底层变得更复杂了。
最后,在一些更复杂的项目上,上述两种方案可能都行不通,这时候最佳的解决方案,就是改用容器了,比如 Docker、Podman 等,容器的限制没 Nix 这么严格,能提供最佳的兼容性。