mirror of
https://github.com/ryan4yin/nixos-and-flakes-book
synced 2024-11-23 04:33:08 +00:00
feat: dev-environments - python
This commit is contained in:
parent
8a517a6b36
commit
8d4710b132
2 changed files with 113 additions and 0 deletions
|
@ -9,3 +9,65 @@ Luckily, some people in the community have done this for us. The following repos
|
|||
If you think the structure of `flake.nix` is still too complicated and want a simpler way, you can also consider using the following project, which encapsulates Nix more thoroughly and provides users with a simpler definition:
|
||||
|
||||
- [cachix/devenv](https://github.com/cachix/devenv)
|
||||
|
||||
## Dev Environment for Python
|
||||
|
||||
The development environment for Python is much more cumbersome compared to languages like Java or Go because it defaults to installing software in the global environment.
|
||||
To install software for the current project, you must create a virtual environment first (unlike in languages such as JavaScript or Go,
|
||||
where virtual environments are not necessary). This behavior is very unfriendly for Nix.
|
||||
|
||||
By default, when using pip in Python, it installs software globally. On NixOS, running `pip install` directly will result in an error:
|
||||
|
||||
```bash
|
||||
› pip install -r requirements.txt
|
||||
error: externally-managed-environment
|
||||
|
||||
× This environment is externally managed
|
||||
╰─> This command has been disabled as it tries to modify the immutable
|
||||
`/nix/store` filesystem.
|
||||
|
||||
To use Python with Nix and nixpkgs, have a look at the online documentation:
|
||||
<https://nixos.org/manual/nixpkgs/stable/#python>.
|
||||
|
||||
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
|
||||
hint: See PEP 668 for the detailed specification.
|
||||
```
|
||||
|
||||
Based on the error message, `pip install` is directly disabled by NixOS. Even when attempting `pip install --user`, it is similarly disabled.
|
||||
To improve the reproducibility of the environment, Nix eliminates these commands altogether.
|
||||
Even if we create a new environment using methods like `mkShell`,
|
||||
these commands still result in errors (presumably because the pip command in Nixpkgs itself has
|
||||
been modified to prevent any modification instructions like `install` from running).
|
||||
|
||||
However, many project installation scripts are based on pip, which means these scripts cannot be used directly.
|
||||
Additionally, the content in nixpkgs is limited, and many packages from PyPI are missing.
|
||||
This requires users to package them themselves, adding a lot of complexity and mental burden.
|
||||
|
||||
One solution is to use the `venv` virtual environment. Within a virtual environment, you can use commands like pip normally:
|
||||
|
||||
```shell
|
||||
python -m venv ./env
|
||||
source ./env/bin/activate
|
||||
```
|
||||
|
||||
Alternatively, you can use a third-party tool called `virtualenv`, but this requires additional installation.
|
||||
|
||||
For those who still lack confidence in the venv created directly with Python, they may prefer to include the virtual environment in `/nix/store` to make it immutable.
|
||||
This can be achieved by directly installing the dependencies from `requirements.txt` or `poetry.toml` using Nix.
|
||||
There are existing Nix packaging tools available to assist with this:
|
||||
|
||||
> Note that even in these environments, running commands like `pip install` directly will still fail.
|
||||
Python dependencies must be installed through `flake.nix` because the data is located in the `/nix/store` directory,
|
||||
and these modification commands can only be executed during the Nix build phase.
|
||||
|
||||
- [DavHau/mach-nix](https://github.com/DavHau/mach-nix)
|
||||
- [poetry2nix](https://github.com/nix-community/poetry2nix)
|
||||
|
||||
The advantage of these tools is that they utilize the lock mechanism of Nix Flakes to improve reproducibility.
|
||||
However, the downside is that they add an extra layer of abstraction, making the underlying system more complex.
|
||||
|
||||
|
||||
Finally, in some more complex projects, neither of the above solutions may be feasible.
|
||||
In such cases, the best solution is to use containers such as Docker or Podman. Containers have fewer restrictions compared to Nix and can provide the best compatibility.
|
||||
|
||||
|
||||
|
|
|
@ -9,3 +9,54 @@
|
|||
如果你觉得 `flake.nix` 的结构还是太复杂了,希望能有更简单的方法,也可以考虑使用下面这个项目,它对 Nix 做了更彻底的封装,对用户提供了更简单的定义:
|
||||
|
||||
- [cachix/devenv](https://github.com/cachix/devenv)
|
||||
|
||||
|
||||
## Python 开发环境
|
||||
|
||||
Python 的开发环境比 Java/Go 等语言要麻烦许多,因为它默认就往全局环境装软件,要往当前项目装,还必须得先创建虚拟环境(JS/Go 等语言里可没虚拟环境这种幺蛾子)。
|
||||
这对 Nix 而言是非常不友好的行为。
|
||||
|
||||
Python 的 pip 默认会将软件安装到全局,在 NixOS 中 `pip install` 会直接报错:
|
||||
|
||||
```shell
|
||||
› pip install -r requirements.txt
|
||||
error: externally-managed-environment
|
||||
|
||||
× This environment is externally managed
|
||||
╰─> This command has been disabled as it tries to modify the immutable
|
||||
`/nix/store` filesystem.
|
||||
|
||||
To use Python with Nix and nixpkgs, have a look at the online documentation:
|
||||
<https://nixos.org/manual/nixpkgs/stable/#python>.
|
||||
|
||||
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
|
||||
hint: See PEP 668 for the detailed specification.
|
||||
```
|
||||
|
||||
根据错误信息,`pip install` 直接被 NixOS 禁用掉了,测试了 `pip install --user` 也同样被禁用。为了提升环境的可复现能力,Nix 把它们全部废掉了。
|
||||
即使我们通过 `mkShell` 等方式创建一个新环境,这些命令照样会报错(猜测是 Nixpkgs 中的 pip 命令本身就被魔改了,只要是跑 `install` 等修改指令就直接嘎掉)。
|
||||
|
||||
但是很多项目的安装脚本都是基于 pip 的,这导致这些脚本都不能直接使用,而且另一方面 nixpkgs 中的内容有限,很多 pypi 中的包里边都没有,还得自己打包,相对麻烦很多,也加重了用户的心智负担。
|
||||
|
||||
解决方案之一是改用 `venv` 虚拟环境,在虚拟环境里当然就能正常使用 pip 等命令了:
|
||||
|
||||
```shell
|
||||
python -m venv ./env
|
||||
source ./env/bin/activate
|
||||
```
|
||||
|
||||
或者使用第三方工具 `virtualenv`,缺点是这个需要额外安装。
|
||||
|
||||
这样用 python 直接创建的 venv,对一些人而言可能还是没有安全感,仍然希望将这个虚拟环境也弄进 `/nix/store` 里使其不可变,通过 nix 直接安装 `requirements.txt` 或者 `poetry.toml` 中的依赖项。
|
||||
这当然是可行的,有现成的 Nix 封装工具帮我们干这个活:
|
||||
|
||||
> 注意即使是在这俩环境中,直接跑 `pip install` 之类的安装命令仍然是会失败的,必须通过 `flake.nix` 来安装 Python 依赖!
|
||||
因为数据还是在 `/nix/store` 中,这类修改命令必须在 Nix 的构建阶段才能执行...
|
||||
|
||||
- [DavHau/mach-nix](https://github.com/DavHau/mach-nix)
|
||||
- [poetry2nix](https://github.com/nix-community/poetry2nix)
|
||||
|
||||
这俩工具的好处是,能利用上 Nix Flakes 的锁机制来提升可复现能力,缺点是多了一层封装,底层变得更复杂了。
|
||||
|
||||
|
||||
最后,在一些更复杂的项目上,上述两种方案可能都行不通,这时候最佳的解决方案,就是改用容器了,比如 Docker、Podman 等,容器的限制没 Nix 这么严格,能提供最佳的兼容性。
|
||||
|
|
Loading…
Reference in a new issue