From f256d6f78a22335a2bcabd9ef7ec67bac5dfefa3 Mon Sep 17 00:00:00 2001 From: Ryan Yin Date: Wed, 4 Oct 2023 14:13:49 +0800 Subject: [PATCH] fix: pkgs.callPackage --- docs/nixpkgs/callpackage.md | 94 ++++++++++++++++++++----- docs/zh/nixpkgs/callpackage.md | 122 +++++++++++++++++++++++++++++---- 2 files changed, 187 insertions(+), 29 deletions(-) diff --git a/docs/nixpkgs/callpackage.md b/docs/nixpkgs/callpackage.md index fb5b3e7..0a0534d 100644 --- a/docs/nixpkgs/callpackage.md +++ b/docs/nixpkgs/callpackage.md @@ -1,24 +1,73 @@ # `pkgs.callPackage` -In the previous section, we used the `import xxx.nix` syntax to import Nix files. This syntax simply returns the execution result of the file without any further processing. +`pkgs.callPackage` is used to parameterize the construction of Nix Derivation. To understand its purpose, let's first consider how we would define a Nix package (also known as a Derivation) without using `pkgs.callPackage`. -Another method to import Nix files is `pkgs.callPackage`. Its syntax is `pkgs.callPackage xxx.nix { ... }`. However, unlike `import`, the Nix file imported by `pkgs.callPackage` must be a Derivation or a function that returns a Derivation. The result of `pkgs.callPackage` is also a Derivation, which represents a software package. +## 1. Without `pkgs.callPackage` -So, what does a Nix file that can be used as a parameter for `pkgs.callPackage` look like? You can refer to the `hello.nix`, `fcitx5-rime.nix`, `vscode/with-extensions.nix`, and `firefox/common.nix` files mentioned earlier. All of these files can be imported using `pkgs.callPackage`. +We can define a Nix package using code like this: -When the `xxx.nix` file used in `pkgs.callPackage xxx.nix {...}` is a function (most Nix packages follow this pattern), the execution flow is as follows: +```nix +pkgs.writeShellScriptBin "hello" ''echo "hello, ryan!"'' +``` -1. `pkgs.callPackage xxx.nix {...}` first imports `xxx.nix` to obtain the function defined within it. This function usually has parameters like `lib`, `stdenv`, `fetchurl`, and sometimes additional custom parameters with default values. +To verify this, you can use `nix repl`, and you'll see that the result is indeed a Derivation: -2. Then, `pkgs.callPackage` searches for a value matching the parameter names from the current environment. Parameters like `lib`, `stdenv`, and `fetchurl` are defined in nixpkgs and will be found in this step. +```shell +› nix repl -f '' +Welcome to Nix 2.13.5. Type :? for help. -3. Next, `pkgs.callPackage` merges its second parameter, `{...}`, with the attribute set obtained in the previous step. It then passes this merged set as the parameter to the function imported from `xxx.nix` for execution. +Loading installable ''... +Added 19203 variables. -4. Finally, the result of the function execution is a Derivation. +nix-repl> pkgs.writeShellScriptBin "hello" '' echo "hello, xxx!" '' +«derivation /nix/store/zhgar12vfhbajbchj36vbbl3mg6762s8-hello.drv» +``` -The common use case for `pkgs.callPackage` is to import customized Nix packages and use them in Nix Modules. +While the definition of this Derivation is quite concise, most Derivations in nixpkgs are much more complex. In previous sections, we introduced and extensively used the `import xxx.nix` method to import Nix expressions from other Nix files, which can enhance code maintainability. -For example, let's say we have a customized NixOS kernel configuration file named `kernel.nix`, which uses the SBC's name and kernel source as its variable parameters: +1. To enhance maintainability, you can store the definition of the Derivation in a separate file, e.g., `hello.nix`. + 1. However, the context within `hello.nix` itself doesn't include the `pkgs` variable, so you'll need to modify its content to pass `pkgs` as a parameter to `hello.nix`. +2. In places where you need to use this Derivation, you can import it using `import ./hello.nix pkgs` and use `pkgs` as a parameter to execute the function defined within. + +Let's continue to verify this using `nix repl`, and you'll see that the result is still a Derivation: + +```shell +› cat hello.nix +pkgs: + pkgs.writeShellScriptBin "hello" '' echo "hello, xxx!" '' + +› nix repl -f '' +Welcome to Nix 2.13.5. Type :? for help. + +warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring +Loading installable ''... +Added 19203 variables. + +nix-repl> import ./hello.nix pkgs +«derivation /nix/store/zhgar12vfhbajbchj36vbbl3mg6762s8-hello.drv» +``` + +## 2. Using `pkgs.callPackage` + +In the previous example without `pkgs.callPackage`, we directly passed `pkgs` as a parameter to `hello.nix`. However, this approach has some drawbacks: + +1. All other dependencies of the `hello` Derivation are tightly coupled with `pkgs`. + 1. If we need custom dependencies, we have to modify either `pkgs` or the content of `hello.nix`, which can be cumbersome. +2. In cases where `hello.nix` becomes complex, it's challenging to determine which Derivations from `pkgs` it relies on, making it difficult to analyze the dependencies between Derivations. + +`pkgs.callPackage`, as a tool for parameterizing the construction of Derivations, addresses these issues. Let's take a look at its source code and comments [nixpkgs/lib/customisation.nix#L101-L121](https://github.com/NixOS/nixpkgs/blob/fe138d3/lib/customisation.nix#L101-L121): + +In essence, `pkgs.callPackage` is used as `pkgs.callPackage fn args`, where `fn` is a Nix file or function, and `args` is an attribute set. Here's how it works: + +1. `pkgs.callPackge fn args` first checks if `fn` is a function or a file. If it's a file, it imports the function defined within. + 1. After this step, you have a function, typically with parameters like `lib`, `stdenv`, `fetchurl`, and possibly some custom parameters. +2. Next, `pkgs.callPackge fn args` merges `args` with the `pkgs` attribute set. If there are conflicts, the parameters in `args` will override those in `pkgs`. +3. Then, `pkgs.callPackge fn args` extracts the parameters of the `fn` function from the merged attribute set and uses them to execute the function. +4. The result of the function execution is a Derivation, which is a Nix package. + +What can a Nix file or function, used as an argument to `pkgs.callPackge`, look like? You can examine examples we've used before: `hello.nix`, `fcitx5-rime.nix`, `vscode/with-extensions.nix`, and `firefox/common.nix`. All of them can be imported using `pkgs.callPackage`. + +For instance, if you've defined a custom NixOS kernel configuration in `kernel.nix` and made the development branch name and kernel source code configurable: ```nix { @@ -36,16 +85,16 @@ For example, let's say we have a customized NixOS kernel configuration file name inherit src lib stdenv; - # File path to the generated kernel config file (`.config` generated by make menuconfig) + # file path to the generated kernel config file(the `.config` generated by make menuconfig) # - # Here, we use a special usage to generate a file path from a string. + # here is a special usage to generate a file path from a string configfile = ./. + "${boardName}_config"; allowImportFromDerivation = true; }) ``` -We can use `pkgs.callPackage ./kernel.nix {}` in any Nix Module to import and replace any of its parameters: +You can use `pkgs.callPackage ./hello.nix {}` in any Nix module to import and use it, replacing any of its parameters as needed: ```nix { lib, pkgs, pkgsKernel, kernel-src, ... }: @@ -56,16 +105,27 @@ We can use `pkgs.callPackage ./kernel.nix {}` in any Nix Module to import and re boot = { # ...... kernelPackages = pkgs.linuxPackagesFor (pkgs.callPackage ./pkgs/kernel { - src = kernel-src; # The kernel source is passed as a `specialArgs` and injected into this module. - boardName = "licheepi4a"; # The board name, used to generate the kernel config file path. + src = kernel-src; # kernel source is passed as a `specialArgs` and injected into this module. + boardName = "licheepi4a"; # the board name, used to generate the kernel config file path. }); # ...... } ``` -In the example above, we use `pkgs.callPackage` to pass different `src` and `boardName` parameters to the function defined in `kernel.nix`. This allows us to generate different kernel packages. By changing the parameters passed to `pkgs.callPackage`, `kernel.nix` can adapt to different kernel sources and development boards. +As shown above, by using `pkgs.callPackage`, you can pass different `src` and `boardName` values to the `kernel.nix` function to generate different kernel packages. This allows you to adapt the same `kernel.nix` to different kernel source code and development boards. + +The advantages of `pkgs.callPackage` are: + +1. Derivation definitions are parameterized, and all dependencies of the Derivation are the function parameters in its definition. This makes it easy to analyze dependencies between Derivations. +2. All dependencies and other custom parameters of the Derivation can be easily replaced by using the second parameter of `pkgs.callPackage`, greatly enhancing Derivation reusability. +3. While achieving the above two functionalities, it does not increase code complexity, as all dependencies in `pkgs` can be automatically injected. + +So it's always recommended to use `pkgs.callPackage` to define Derivations. ## References -- [Chapter 13. Callpackage Design Pattern - Nix Pills](https://nixos.org/guides/nix-pills/callpackage-design-pattern.html) \ No newline at end of file +- [Chapter 13. Callpackage Design Pattern - Nix Pills](https://nixos.org/guides/nix-pills/callpackage-design-pattern.html) +- [callPackage, a tool for the lazy - The Summer of Nix](https://summer.nixos.org/blog/callpackage-a-tool-for-the-lazy/) +- [Document what callPackage does and its preconditions - Nixpkgs Issues](https://github.com/NixOS/nixpkgs/issues/36354) + diff --git a/docs/zh/nixpkgs/callpackage.md b/docs/zh/nixpkgs/callpackage.md index 16fbe0a..4b6be4f 100644 --- a/docs/zh/nixpkgs/callpackage.md +++ b/docs/zh/nixpkgs/callpackage.md @@ -1,21 +1,109 @@ # pkgs.callPackage -前面我们介绍并大量使用了 `import xxx.nix` 来导入 Nix 文件，这种语法只是单纯地返回该文件的执行结果，不会对该结果进行进一步处理。 -比如说 `xxx.nix` 的内容是形如 `{...}: {...}`，那么 `import xxx.nix` 的结果就是该文件中定义的这个函数。 +`pkgs.callPackage` 被用于参数化构建 Nix 包，为了理解它的用处，我们首先考虑下不使用 `pkgs.callPakcage` 的情况下，我们要如何定义一个 Nix 包（也就是 Derivation）。 -`pkgs.callPackage` 也是用来导入 Nix 文件的，它的语法是 `pkgs.callPackage xxx.nix { ... }`. 但跟 `import` 不同的是，它导入的 nix 文件内容必须是一个 Derivation 或者返回 Derivation 的函数，它的执行结果一定是一个 Derivation，也就是一个软件包。 +## 1. 不使用 `pkgs.callPackage` 的情况 + +我们可以使用如下代码来定义一个 Nix 包： + +```nix +pkgs.writeShellScriptBin "hello" ''echo "hello, ryan!"'' +``` + +使用 `nix repl` 来验证一下，能看到它的执行结果确实是一个 Derivation： + +```shell +› nix repl -f '' +Welcome to Nix 2.13.5. Type :? for help. + +Loading installable ''... +Added 19203 variables. + +nix-repl> pkgs.writeShellScriptBin "hello" '' echo "hello, xxx!" '' +«derivation /nix/store/zhgar12vfhbajbchj36vbbl3mg6762s8-hello.drv» +``` + +上面这个 Derivation 的定义很短，就一行，但 nixpkgs 中大部分的 Derivation 的定义都要比这复杂很多。 +前面我们介绍并大量使用了 `import xxx.nix` 来从其他 Nix 文件中导入 Nix 表达式，我们可以在这里也使用这种方法来提升代码的可维护性： + +1. 将上面这一行 Derivation 的定义存放到单独的文件 `hello.nix` 中。 + 1. 但 `hello.nix` 自身的上下文中不包含 `pkgs` 这个变量，所以需要修改下其内容，将 `pkgs` 作为参数传递给 `hello.nix`。 +1. 在需要使用这个 Derivation 的地方，使用 `import ./hello.nix pkgs` 来导入它并使用 `pkgs` 作为参数来执行其中定义的函数。 + +仍然使用 `nix repl` 来验证一下，能看到它的执行结果仍然是一个 Derivation： + +```shell +› cat hello.nix +pkgs: + pkgs.writeShellScriptBin "hello" '' echo "hello, xxx!" '' + +› nix repl -f '' +Welcome to Nix 2.13.5. Type :? for help. + +warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring +Loading installable ''... +Added 19203 variables. + +nix-repl> import ./hello.nix pkgs +«derivation /nix/store/zhgar12vfhbajbchj36vbbl3mg6762s8-hello.drv» +``` + +## 2. 使用 `pkgs.callPackage` 的情况 + +在前面不使用 `pkgs.callPackage` 的例子中，我们直接将 `pkgs` 作为参数传到了 `hello.nix` 中，这样做的缺点有： + +1. `hello` 这个 derivation 的所有其他依赖项都只能从 `pkgs` 中获取，耦合度太高。 + 1. 比如说我们如果需要其他自定义依赖项，就必须修改 `pkgs` 或者修改 `hello.nix` 的内容，而这两个都很麻烦。 +1. 在 `hello.nix` 变复杂的情况下，很难判断 `hello.nix` 到底依赖了 `pkgs` 中的哪些 Derivation，很难分析 Derivation 之间的依赖关系。 + +而 `pkgs.callPackage` 作为一个参数化构建 Derivation 的工具函数，可解决上述两个问题。 +首先看看源码中此函数的定义与注释 [nixpkgs/lib/customisation.nix#L101-L121](https://github.com/NixOS/nixpkgs/blob/fe138d3/lib/customisation.nix#L101-L121 +)： + +```nix + /* Call the package function in the file `fn` with the required + arguments automatically. The function is called with the + arguments `args`, but any missing arguments are obtained from + `autoArgs`. This function is intended to be partially + parameterised, e.g., + + callPackage = callPackageWith pkgs; + pkgs = { + libfoo = callPackage ./foo.nix { }; + libbar = callPackage ./bar.nix { }; + }; + + If the `libbar` function expects an argument named `libfoo`, it is + automatically passed as an argument. Overrides or missing + arguments can be supplied in `args`, e.g. + + libbar = callPackage ./bar.nix { + libfoo = null; + enableX11 = true; + }; + */ + callPackageWith = autoArgs: fn: args: + let + f = if lib.isFunction fn then fn else import fn; + fargs = lib.functionArgs f; + + # All arguments that will be passed to the function + # This includes automatic ones and ones passed explicitly + allArgs = builtins.intersectAttrs fargs autoArgs // args; + + # ...... 省略后面的内容 ...... +``` + +简单的说，它的使用格式是 `pkgs.callPackage fn args`，其中 `fn` 是一个 nix 文件或者函数，`args` 是一个 attribute set，它的工作流程是： + +1. `pkgs.callPackge fn args` 会先判断 `fn` 是一个函数还是一个文件，如果是文件就先通过 `import xxx.nix` 导入其中定义的函数。 + 1. 第一步执行完毕得到的是一个函数，其参数通常会有 `lib`, `stdenv`, `fetchurl` 等参数，可能还会带有一些自定义参数。 +2. 之后，`pkgs.callPackge fn args` 会将 `args` 与 `pkgs` 这个 attribute set 合并。如果存在冲突，`args` 中的参数会覆盖 `pkgs` 中的参数。 +3. 再之后，`pkgs.callPackge fn args` 会从上一步得到的 attribute set 中提取出 `fn` 函数的参数，并使用它们来执行 `fn` 函数。 +4. 函数执行结果是一个 Derivation，也就是一个 Nix 包。 那可以作为 `pkgs.callPackge` 参数的 nix 文件具体长啥样呢，可以去看看我们前面举例过的 `hello.nix` `fcitx5-rime.nix` `vscode/with-extensions.nix` `firefox/common.nix`，它们都可以被 `pkgs.callPackage` 导入。 -当 `pkgs.callPackge xxx.nix {...}` 中的 `xxx.nix`，其内容为一个函数时（绝大多数 nix 包都是如此），执行流程如下： - -1. `pkgs.callPackge xxx.nix {...}` 会先 `import xxx.nix`，得到其中定义的函数，该函数的参数通常会有 `lib`, `stdenv`, `fetchurl` 等参数，以及一些自定义参数，自定义参数通常都有默认值。 -2. 接着 `pkgs.callPackge` 会首先从当前环境中查找名称匹配的值，作为将要传递给前述函数的参数。像 `lib` `stdenv` `fetchurl` 这些都是 nixpkgs 中的函数，在这一步就会查找到它们。 -3. 接着 `pkgs.callPackge` 会将其第二个参数 `{...}` 与前一步得到的参数集（attribute set）进行合并，得到一个新的参数列表，然后将其传递给该函数并执行。 -4. 函数执行结果是一个 Derivation，也就是一个软件包。 - -这个函数比较常见的用途是用来导入一些自定义的 Nix 包。 - 比如说我们自定义了一个 NixOS 内核配置 `kernel.nix`，并且将开发版名称与内核源码作为了可变更参数： ```nix @@ -64,6 +152,16 @@ 就如上面所展示的，通过 `pkgs.callPackage` 我们可以给 `kernel.nix` 定义的函数传入不同的 `src` 与 `boardName`，来生成不同的内核包，这样就可以使用同一份 `kernel.nix` 来适配不同的内核源码与不同的开发板了。 +`pkgs.callPackage` 的优势在于： + +1. Derivation 的定义被参数化，定义中的所有函数参数就是 Derivation 的所有依赖项，这样就可以很方便的分析 Derivation 之间的依赖关系。 +2. Derivation 的所有依赖项与其他自定义参数都可以很方便地被替换（通过使用 `pkgs.callPackage` 的第二个参数），Derivation 定义的可复用性大大提升。 +3. 在实现了前两条功能的情况下，并未增加代码的复杂度，所有 `pkgs` 中的依赖项都可以被自动注入，不需要手动传递。 + +因此我们总是推荐使用 `pkgs.callPackage` 来定义 Derivation。 + ## 参考 - [Chapter 13. Callpackage Design Pattern - Nix Pills](https://nixos.org/guides/nix-pills/callpackage-design-pattern.html) +- [callPackage, a tool for the lazy - The Summer of Nix](https://summer.nixos.org/blog/callpackage-a-tool-for-the-lazy/) +- [Document what callPackage does and its preconditions - Nixpkgs Issues](https://github.com/NixOS/nixpkgs/issues/36354)