Mirrors/nix-pills
2
0
Fork 0
mirror of https://github.com/NixOS/nix-pills synced 2024-11-14 15:47:10 +00:00
Code Issues Projects Releases Packages Wiki Activity

Merge pull request #2 from disassembler/sam-nix-pills

Browse source 
port pills #12-16
This commit is contained in:
Graham Christensen 2017-08-18 07:14:26 -04:00 committed by GitHub
commit 6883dbcefc
38 changed files with 965 additions and 25 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

241
pills/12-inputs-design-pattern.xml
View file

@ -1,8 +1,239 @@
<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="inputs-design-pattern">
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="inputs-design-pattern">
<title>inputs design pattern</title>
<title>inputs design pattern</title>
<para>
Welcome to the 12th Nix pill. In the previous <link linkend="garbage-collector">11th pill</link> we stopped packaging and cleaned up the system with the garbage collector.
</para>
<para>
We restart our packaging, but we will improve a different aspect. We only packaged an hello world program so far, what if we want to create a repository of multiple packages?
</para>
<section>
<title>Repositories in Nix</title>
<para>
Nix is a tool for build and deployment, it does not enforce any particular repository format. A repository of packages is the main usage for Nix, but not the only possibility. See it more like a consequence due to the need of organizing packages.
</para>
<para>
Nix is a language, and it is powerful enough to let you choose the format of your own repository. In this sense, it is not declarative, but functional.
</para>
<para>
There is no preset directory structure or preset packaging policy. It's all about you and Nix.
</para>
<para>
The <literal>nixpkgs</literal> repository has a certain structure, which evolved and evolves with the time. Like other languages, Nix has its own history and therefore I'd like to say that it also has its own design patterns. Especially when packaging, you often do the same task again and again except for different software. It's inevitable to identify patterns during this process. Some of these patterns get reused if the community thinks it's a good way to package the software.
</para>
<para>
Some of the patterns I'm going to show do not apply only to Nix, but to other systems of course.
</para>
</section>
<section>
<title>The single repository pattern</title>
<para>
Before introducing the "<literal>inputs</literal>" pattern, we can start talking about another pattern first which I'd like to call "<literal>single repository</literal>" pattern.
</para>
<para>
Systems like Debian scatter packages in several small repositories. Personally, this makes it hard to track interdependent changes and to contribute to new packages.
</para>
<para>
Systems like Gentoo instead, put package descriptions all in a single repository.
</para>
<para>
The nix reference for packages is <link xlink:href="https://github.com/NixOS/nixpkgs">nixpkgs</link>, a single repository of all descriptions of all packages. I find this approach very natural and attractive for new contributions.
</para>
<para>
From now on, we will adopt this technique. The natural implementation in Nix is to create a top-level Nix expression, and one expression for each package. The top-level expression imports and combines all expressions in a giant attribute set with name -> package pairs.
</para>
<para>
But isn't that heavy? It isn't, because Nix is a lazy language, it evaluates only what's needed! And that's why <literal>nixpkgs</literal> is able to maintain such a big software repository in a giant attribute set.
</para>
</section>
<section>
<title>Packaging graphviz</title>
<para>
We have packaged <package>GNU hello worl</package>, I guess you would like to package something else for creating at least a repository of two projects :-) . I chose <package>graphviz</package>, which uses the standard autotools build system, requires no patching and dependencies are optional.
</para>
<para>
Download <package>graphviz</package> from here. The <filename>graphviz.nix</filename> expression is straightforward:
</para>
<screen><xi:include href="./12/graphviz-derivation.txt" parse="text" /></screen>
<para>
Build with <command>nix-build graphviz.nix</command> and you will get runnable binaries under <filename>result/bin</filename>. Notice how we did reuse the same <filename>autotools.nix</filename> of <filename>hello.nix.</filename> Let's create a simple png:
</para>
<screen><xi:include href="./12/simple-png.txt" parse="text" /></screen>
<para>
Oh of course... <package>graphviz</package> can't know about png. It built only the output formats it supports natively, without using any extra library.
</para>
<para>
I remind you, in <filename>autotools.nix</filename> there's a <literal>buildInputs</literal> variable which gets concatenated to <literal>baseInputs</literal>. That would be the perfect place to add a build dependency. We created that variable exactly for this reason to be overridable from package expressions.
</para>
<para>
This 2.38 version of <package>graphviz</package> has several plugins to output png. For simplicity, we will use <package>libgd</package>.
</para>
</section>
<section>
<title>Digression about gcc and ld wrappers</title>
<para>
The <package>gd</package>, <package>jpeg</package>, <package>fontconfig</package> and <package>bzip2</package> libraries (dependencies of <package>gd</package>) don't use <command>pkg-config</command> to specify which flags to pass to the compiler. Since there's no global location for libraries, we need to tell <command>gcc</command> and <command>ld</command> where to find includes and libs.
</para>
<para>
The <literal>nixpkgs</literal> provides <package>gcc</package> and <package>binutils</package>, and we are using them for our packaging. Not only, it also <link xlink:href="http://nixos.org/nixpkgs/manual/#ssec-setup-hooks">provides wrappers</link> for them which allow passing extra arguments to <command>gcc</command> and <command>ld</command>, bypassing the project build systems:
<itemizedlist>
<listitem><para><varname>NIX_CFLAGS_COMPILE</varname>: extra flags to <command>gcc</command> at compile time</para></listitem>
<listitem><para><varname>NIX_LDFLAGS</varname>: extra flags to <command>ld</command></para></listitem>
</itemizedlist>
</para>
<para>
What can we do about it? We can employ the same trick we did for <varname>PATH</varname>: automatically filling the variables from <literal>buildInputs</literal>. This is the relevant snippet of <filename>setup.sh</filename>:
</para>
<screen><xi:include href="./12/setup-sh.txt" parse="text" /></screen>
<para>
Now by adding derivations to <literal>buildInputs</literal>, will add the <filename>lib</filename>, <filename>include</filename> and <filename>bin</filename> paths automatically in <filename>setup.sh</filename>.
</para>
<para>
The <arg>-rpath</arg> flag in <command>ld</command> is needed because at runtime, the executable must use exactly that version of the library.
</para>
<para>
If unneeded paths are specified, the <literal>fixup</literal> phase will shrink the <literal>rpath</literal> for us!
</para>
</section>
<section>
<title>Completing graphviz with gd</title>
<para>
Finish the expression for <package>graphviz</package> with <package>gd</package> support (note the use of the <literal>with</literal> expression in <literal>buildInputs</literal> to avoid repeating <literal>pkgs</literal>):
</para>
<screen><xi:include href="./12/graphviz-gd-derivation.txt" parse="text" /></screen>
<para>
Now you can create the png! Ignore any error from <package>fontconfig</package>, especially if you are in a <literal>chroot</literal>.
</para>
</section>
<section>
<title>The repository expression</title>
<para>
Now that we have two packages, what's a good way to put them together in a single repository? We do something like <literal>nixpkgs</literal> does. With <literal>nixpkgs</literal>, we <literal>import</literal> it and then we peek derivations by accessing the giant attribute set.
</para>
<para>
For us nixers, this a good technique, because it abstracts from the file names. We don't refer to a package by <filename>REPO/some/sub/dir/package.nix</filename> but by <literal>importedRepo.package</literal> (or <literal>pkgs.package</literal> in our examples).
</para>
<para>
Create a default.nix in the current directory:
</para>
<screen><xi:include href="./12/repository.txt" parse="text" /></screen>
<para>
Ready to use! Try it with <command>nix-repl</command>:
</para>
<screen><xi:include href="./12/repository-test-nix-repl.txt" parse="text" /></screen>
<para>
With <command>nix-build</command>:
</para>
<screen><xi:include href="./12/repository-test-nix-build.txt" parse="text" /></screen>
<para>
The <arg>-A</arg> argument is used to access an attribute of the set from the given .nix expression.
</para>
<para>
<emphasis role="bold">Important:</emphasis> why did we choose the <filename>default.nix</filename>? Because when a directory (by default the current directory) has a <filename>default.nix</filename>, that <filename>default.nix</filename> will be used (see <literal>import</literal> <link xlink:href="http://nixos.org/nix/manual/#ssec-builtins">here</link>). In fact you can run <command>nix-build -A hello</command> without specifying <filename>default.nix</filename>.
</para>
<para>
For pythoners, it is similar to <filename>__init__.py</filename>.
</para>
<para>
With <command>nix-env</command>, to install the package in your user environment:
</para>
<screen><xi:include href="./12/nix-env-install-graphviz.txt" parse="text" /></screen>
<para>
The <arg>-f</arg> option is used to specify the expression to use, in this case the current directory, therefore <filename>./default.nix</filename>.
</para>
<para>
The <arg>-i</arg> stands for installation.
</para>
<para>
The <arg>-A</arg> is the same as above for <command>nix-build</command>.
</para>
<para>
We reproduced the very basic behavior of <literal>nixpkgs</literal>.
</para>
</section>
<section>
<title>The inputs pattern</title>
<para>
After a long preparation, we finally arrived. I know you have a big doubt in this moment. It's about the <filename>hello.nix</filename> and <filename>graphviz.nix</filename>. They are very, very dependent on <literal>nixpkgs</literal>:
<itemizedlist>
<listitem><para>First big problem: they import <literal>nixpkgs</literal> directly. In <filename>autotools.nix</filename> instead we pass <literal>nixpkgs</literal> as an argument. That's a much better approach.</para></listitem>
<listitem><para>Second problem: what if we want a variant of <package>graphviz</package> without <package>libgd</package> support?</para></listitem>
<listitem><para>Third problem: what if we want to test <package>graphviz</package> with a particular <package>libgd</package> version?</para></listitem>
</itemizedlist>
</para>
<para>
The current answer to the above questions is: change the expression to match your needs (or change the callee to match your needs).
</para>
<para>
With the <literal>inputs</literal> pattern, we choose to give another answer: let the user change the <literal>inputs</literal> of the expression (or change the caller to pass different inputs).
</para>
<para>
By inputs of an expression, we refer to the set of derivations needed to build that expression. In this case:
<itemizedlist>
<listitem><para><literal>mkDerivation</literal> from <package>autotools</package>. Recall that <literal>mkDerivation</literal> has an implicit dependency on the toolchain.</para></listitem>
<listitem><para><package>libgd</package> and its dependencies.</para></listitem>
</itemizedlist>
</para>
<para>
The src is also an input but it's pointless to change the source from the caller. For version bumps, in <literal>nixpkgs</literal> we prefer to write another expression (e.g. because patches are needed or different inputs are needed).
</para>
<para>
<emphasis role="underline">Goal:</emphasis> make package expressions independent of the repository.
</para>
<para>
How do we achieve that? The answer is simple: use functions to declare inputs for a derivation. Doing it for <filename>graphviz.nix</filename>, will make the derivation independent of the repository and customizable:
</para>
<screen><xi:include href="./12/graphviz-mkderivation.txt" parse="text" /></screen>
<para>
I recall that "<literal>{...}: ...</literal>" is the syntax for defining functions accepting an attribute set as argument.
</para>
<para>
We made <package>gd</package> and its dependencies optional. If <literal>gdSupport</literal> is true (by default), we will fill <literal>buildInputs</literal> and thus <package>graphviz</package> will be built with <package>gd</package> support, otherwise it won't.
</para>
<para>
Now back to default.nix:
</para>
<screen><xi:include href="./12/repository-mkderivation.txt" parse="text" /></screen>
<para>
So we factorized the import of <literal>nixpkgs</literal> and <literal>mkDerivation</literal>, and also added a variant of <package>graphviz</package> with <package>gd</package> support disabled. The result is that both <filename>hello.nix</filename> (exercise for the reader) and <filename>graphviz.nix</filename> are independent of the repository and customizable by passing specific inputs.
</para>
<para>
If you wanted to build <package>graphviz</package> with a specific version of <package>gd</package>, it would suffice to pass <literal>gd = ...;</literal>.
</para>
<para>
If you wanted to change the toolchain, you may pass a different <literal>mkDerivation</literal> function.
</para>
<para>
Clearing up the syntax:
<itemizedlist>
<listitem><para>In the end we return an attribute set from <filename>default.nix</filename>. With "<literal>let</literal>" we define some local variables.</para></listitem>
<listitem><para>We bring <literal>pkgs</literal> into the scope when defining the packages set, which is very convenient instead of typing everytime "<literal>pkgs</literal>".</para></listitem>
<listitem><para>We import <filename>hello.nix</filename> and <filename>graphviz.nix</filename>, which will return a function, and call it with a set of inputs to get back the derivation.</para></listitem>
<listitem><para>The "<literal>inherit x</literal>" syntax is equivalent to "<literal>x = x</literal>". So "<literal>inherit gd</literal>" here, combined to the above "<literal>with pkgs;</literal>" is equivalent to "<literal>x = pkgs.gd</literal>".</para></listitem>
</itemizedlist>
</para>
<para>
You can find the whole repository at the <link xlink:href="https://gist.github.com/lethalman/734b168a0258b8a38ca2">pill 12</link> gist.
</para>
</section>
<section>
<title>Conclusion</title>
<para>
The "<literal>inputs</literal>" pattern allows our expressions to be easily customizable through a set of arguments. These arguments could be flags, derivations, or whatelse. Our package expressions are functions, don't think there's any magic in there.
</para>
<para>
It also makes the expressions independent of the repository. Given that all the needed information is passed through arguments, it is possible to use that expression in any other context.
</para>
</section>
<section>
<title>Next pill</title>
<para>
...we will talk about the "<literal>callPackage</literal>" design pattern. It is tedious to specify the names of the inputs twice, once in the top-level <filename>default.nix</filename>, and once in the package expression. With <literal>callPackage</literal>, we will implicitly pass the necessary inputs from the top-level expression.
</para>
</section>
</chapter>

7
pills/12/graphviz-derivation.txt Normal file
View file

@ -0,0 +1,7 @@
let
pkgs = import <nixpkgs> {};
mkDerivation = import ./autotools.nix pkgs;
in mkDerivation {
name = "graphviz";
src = ./graphviz-2.38.0.tar.gz;
}

8
pills/12/graphviz-gd-derivation.txt Normal file
View file

@ -0,0 +1,8 @@
let
pkgs = import <nixpkgs> {};
mkDerivation = import ./autotools.nix pkgs;
in mkDerivation {
name = "graphviz";
src = ./graphviz-2.38.0.tar.gz;
buildInputs = with pkgs; [ gd fontconfig libjpeg bzip2 ];
}

7
pills/12/graphviz-mkderivation.txt Normal file
View file

@ -0,0 +1,7 @@
{ mkDerivation, gdSupport ? true, gd, fontconfig, libjpeg, bzip2 }:
mkDerivation {
name = "graphviz";
src = ./graphviz-2.38.0.tar.gz;
buildInputs = if gdSupport then [ gd fontconfig libjpeg bzip2 ] else [];
}

3
pills/12/nix-env-install-graphviz.txt Normal file
View file

@ -0,0 +1,3 @@
$ nix-env -f . -iA graphviz
[...]
$ dot -V

11
pills/12/repository-mkderivation.txt Normal file
View file

@ -0,0 +1,11 @@
let
pkgs = import <nixpkgs> {};
mkDerivation = import ./autotools.nix pkgs;
in with pkgs; {
hello = import ./hello.nix { inherit mkDerivation; };
graphviz = import ./graphviz.nix { inherit mkDerivation gd fontconfig libjpeg bzip2; };
graphvizCore = import ./graphviz.nix {
inherit mkDerivation gd fontconfig libjpeg bzip2;
gdSupport = false;
};
}

4
pills/12/repository-test-nix-build.txt Normal file
View file

@ -0,0 +1,4 @@
$ nix-build default.nix -A hello
[...]
$ result/bin/hello
Hello, world!

7
pills/12/repository-test-nix-repl.txt Normal file
View file

@ -0,0 +1,7 @@
$ nix-repl
nix-repl> :l default.nix
Added 2 variables.
nix-repl> hello
«derivation /nix/store/dkib02g54fpdqgpskswgp6m7bd7mgx89-hello.drv»
nix-repl> graphviz
«derivation /nix/store/zqv520v9mk13is0w980c91z7q1vkhhil-graphviz.drv»

4
pills/12/repository.txt Normal file
View file

@ -0,0 +1,4 @@
{
hello = import ./hello.nix;
graphviz = import ./graphviz.nix;
}

11
pills/12/setup-sh.txt Normal file
View file

@ -0,0 +1,11 @@
for p in $baseInputs $buildInputs; do
if [ -d $p/bin ]; then
export PATH="$p/bin${PATH:+:}$PATH"
fi
if [ -d $p/include ]; then
export NIX_CFLAGS_COMPILE="-I $p/include${NIX_CFLAGS_COMPILE:+ }$NIX_CFLAGS_COMPILE"
fi
if [ -d $p/lib ]; then
export NIX_LDFLAGS="-rpath $p/lib -L $p/lib${NIX_LDFLAGS:+ }$NIX_LDFLAGS"
fi
done

2
pills/12/simple-png.txt Normal file
View file

@ -0,0 +1,2 @@
$ echo 'graph test { a -- b }'|result/bin/dot -Tpng -o test.png
Format: "png" not recognized. Use one of: canon cmap [...]

141
pills/13-callpackage-design-pattern.xml
View file

@ -1,8 +1,139 @@
<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="callpackage-design-pattern">
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="callpackage-design-pattern">
<title>callpackage design pattern</title>
<title>callpackage design pattern</title>
<para>
Welcome to the 13th Nix pill. In the previous <link linkend="inputs-design-pattern">12th pill</link> we have introduced the first basic design pattern for organizing a repository of software. In addition we packaged <package>graphviz</package> to have at least another package for our little repository.
</para>
<para>
The next design pattern worth noting is what I'd like to call the <literal>callPackage</literal> pattern. This technique is extensively used in <link xlink:href="https://github.com/NixOS/nixpkgs">nixpkgs</link>, it's the current standard for importing packages in a repository.
</para>
<section>
<title>The callPackage convenience</title>
<para>
In the previous pill, we underlined the fact that the inputs pattern is great to decouple packages from the repository, in that we can pass manually the inputs to the derivation. The derivation declares its inputs, and the caller passes the arguments.
</para>
<para>
However as with usual programming languages, we declare parameter names, and then we have to pass arguments. We do the job twice. With package management, we often see common patterns. In the case of <literal>nixpkgs</literal> it's the following.
</para>
<para>
Some package derivation:
</para>
<screen><xi:include href="./13/package-derivation.txt" parse="text" /></screen>
<para>
Repository derivation:
</para>
<screen><xi:include href="./13/repository-derivation.txt" parse="text" /></screen>
<para>
Where inputs may even be packages in the repository itself (note the rec keyword). The pattern here is clear, often inputs have the same name of the attributes in the repository itself. Our desire is to pass those inputs from the repository automatically, and in case be able to specify a particular argument (that is, override the automatically passed default argument).
</para>
<para>
To achieve this, we will define a <literal>callPackage</literal> function with the following synopsis:
</para>
<screen><xi:include href="./13/callpackage-function-call.txt" parse="text" /></screen>
<para>
What should it do?
<itemizedlist>
<listitem><para>Import the given expression, which in turn returns a function.</para></listitem>
<listitem><para>Determine the name of its arguments.</para></listitem>
<listitem><para>Pass default arguments from the repository set, and let us override those arguments.</para></listitem>
</itemizedlist>
</para>
</section>
<section>
<title>Implementing callPackage</title>
<para>
First of all, we need a way to introspect (reflection or whatever) at runtime the argument names of a function. That's because we want to automatically pass such arguments.
</para>
<para>
Then <literal>callPackage</literal> requires access to the whole packages set, because it needs to find the packages to pass automatically.
</para>
<para>
We start off simple with <command></command>:
</para>
<screen><xi:include href="./13/get-args-function.txt" parse="text" /></screen>
<para>
Nix provides a builtin function to introspect the names of the arguments of a function. In addition, for each argument, it tells whether the argument has a default value or not. We don't really care about default values in our case. We are only interested in the argument names.
</para>
<para>
Now we need a set with all the <literal>values</literal>, let's call it <literal>values</literal>. And a way to intersect the attributes of values with the function arguments:
</para>
<screen><xi:include href="./13/intersect-attr-values.txt" parse="text" /></screen>
<para>
Perfect, note from the example above that the <literal>intersectAttrs</literal> returns a set whose names are the intersection, and the attribute values are taken from the second set.
</para>
<para>
We're done, we have a way to get argument names from a function, and match with an existing set of attributes. This is our simple implementation of <literal>callPackage</literal>:
</para>
<screen><xi:include href="./13/callpackage-function.txt" parse="text" /></screen>
<para>
Clearing up the syntax:
<itemizedlist>
<listitem><para>We define a <literal>callPackage</literal> variable which is a function.</para></listitem>
<listitem><para>The second parameter is the function to "autocall".</para></listitem>
<listitem><para>We take the argument names of the function and intersect with the set of all values.</para></listitem>
<listitem><para>Finally we call the passed function <literal>f</literal> with the resulting intersection.</para></listitem>
</itemizedlist>
</para>
<para>
In the code above, I've also shown that the <literal>callPackage</literal> call is equivalent to directly calling <literal>add a b</literal>.
</para>
<para>
We achieved what we wanted. Automatically call functions given a set of possible arguments. If an argument is not found in the set, that's nothing special. It's a function call with a missing parameter, and that's an error (unless the function has varargs <literal>...</literal> as explained in the <link linkend="functions-and-imports">5th pill</link>).
</para>
<para>
Or not. We missed something. Being able to override some of the parameters. We may not want to always call functions with values taken from the big set. Then we add a further parameter, which takes a set of overrides:
</para>
<screen><xi:include href="./13/callpackage-function-overrides.txt" parse="text" /></screen>
<para>
Apart from the increasing number of parenthesis, it should be clear that we simply do a set union between the default arguments, and the overriding set.
</para>
</section>
<section>
<title>Use callPackage to simplify the repository</title>
<para>
Given our brand new tool, we can simplify the repository expression (default.nix).
</para>
<para>
Let me write it down first:
</para>
<screen><xi:include href="./13/callpackage-usage.txt" parse="text" /></screen>
<para>
Wow, there's a lot to say here:
<itemizedlist>
<listitem><para>We renamed the old <literal>pkgs</literal> of the previous pill to <literal>nixpkgs</literal>. Our package set is now instead named <literal>pkgs</literal>. Sorry for the confusion.</para></listitem>
<listitem><para>We needed a way to pass pkgs to <literal>callPackage</literal> somehow. Instead of returning the set of packages directly from <filename>default.nix</filename>, we first assign it to a <literal>let</literal> variable and reuse it in <literal>callPackage</literal>.</para></listitem>
<listitem><para>For convenience, in <literal>callPackage</literal> we first import the file, instead of calling it directly. Otherwise for each package we would have to write the <literal>import</literal>.</para></listitem>
<listitem><para>Since our expressions use packages from <literal>nixpkgs</literal>, in <literal>callPackage</literal> we use <literal>allPkgs</literal>, which is the union of <literal>nixpkgs</literal> and our packages.</para></listitem>
<listitem><para>We moved <literal>mkDerivation</literal> in <literal>pkgs</literal> itself, so that it gets also passed automatically.</para></listitem>
</itemizedlist>
</para>
<para>
Note how easy is to override arguments in the case of <package>graphviz</package> without <package>gd</package>. But most importantly, how easy it was to merge two repositories: <literal>nixpkgs</literal> and our <literal>pkgs</literal>!
</para>
<para>
The reader should notice a magic thing happening. We're defining <literal>pkgs</literal> in terms of <literal>callPackage</literal>, and <literal>callPackage</literal> in terms of <literal>pkgs</literal>. That magic is possible thanks to lazy evaluation.
</para>
</section>
<section>
<title>Conclusion</title>
<para>
The "<literal>callPackage</literal>" pattern has simplified a lot our repository. We're able to import packages that require some named arguments and call them automatically, given the set of all packages.
</para>
<para>
We've also introduced some useful builtin functions that allows us to introspect Nix functions and manipulate attributes. These builtin functions are not usually used when packaging software, rather to provide tools for packaging. That's why they are not documented in the <link xlink:href="http://nixos.org/nix/manual/">nix manual</link>.
</para>
<para>
Writing a repository in nix is an evolution of writing convenient functions for combining the packages. This demonstrates even more how nix is a generic tool to build and deploy something, and how suitable it is to create software repositories with your own conventions.
</para>
</section>
<section>
<title>Next pill</title>
<para>
...we will talk about the "<literal>override</literal>" design pattern. The <literal>graphvizCore</literal> seems straightforward. It starts from <filename>graphviz.nix</filename> and builds it without <package>gd</package>. Now I want to give you another point of view: what if we instead wanted to start from <literal>pkgs.graphviz</literal> and disable <package>gd</package>?
</para>
</section>
</chapter>

4
pills/13/callpackage-function-call.txt Normal file
View file

@ -0,0 +1,4 @@
{
lib1 = callPackage package1.nix { };
program2 = callPackage package2.nix { someoverride = overriddenDerivation; };
}

5
pills/13/callpackage-function-overrides.txt Normal file
View file

@ -0,0 +1,5 @@
nix-repl> callPackage = set: f: overrides: f ((builtins.intersectAttrs (builtins.functionArgs f) set) // overrides)
nix-repl> callPackage values add { }
8
nix-repl> callPackage values add { b = 12; }
15

5
pills/13/callpackage-function.txt Normal file
View file

@ -0,0 +1,5 @@
nix-repl> callPackage = set: f: f (builtins.intersectAttrs (builtins.functionArgs f) set)
nix-repl> callPackage values add
8
nix-repl> with values; add { inherit a b; }
8

13
pills/13/callpackage-usage.txt Normal file
View file

@ -0,0 +1,13 @@
let
nixpkgs = import <nixpkgs> {};
allPkgs = nixpkgs // pkgs;
callPackage = path: overrides:
let f = import path;
in f ((builtins.intersectAttrs (builtins.functionArgs f) allPkgs) // overrides);
pkgs = with nixpkgs; {
mkDerivation = import ./autotools.nix nixpkgs;
hello = callPackage ./hello.nix { };
graphviz = callPackage ./graphviz.nix { };
graphvizCore = callPackage ./graphviz.nix { gdSupport = false; };
};
in pkgs

3
pills/13/get-args-function.txt Normal file
View file

@ -0,0 +1,3 @@
nix-repl> add = { a ? 3, b }: a+b
nix-repl> builtins.functionArgs add
{ a = true; b = false; }

5
pills/13/intersect-attr-values.txt Normal file
View file

@ -0,0 +1,5 @@
nix-repl> values = { a = 3; b = 5; c = 10; }
nix-repl> builtins.intersectAttrs values (builtins.functionArgs add)
{ a = true; b = false; }
nix-repl> builtins.intersectAttrs (builtins.functionArgs add) values
{ a = 3; b = 5; }

2
pills/13/package-derivation.txt Normal file
View file

@ -0,0 +1,2 @@
{ input1, input2, ... }:
...

4
pills/13/repository-derivation.txt Normal file
View file

@ -0,0 +1,4 @@
rec {
lib1 = import package1.nix { inherit input1 input2 ...; };
program2 = import package1.nix { inherit inputX inputY lib1 ...; };
}

146
pills/14-override-design-pattern.xml
View file

@ -1,8 +1,144 @@
<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="override-design-pattern">
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="override-design-pattern">
<title>override design pattern</title>
<title>override design pattern</title>
<para>
Welcome to the 14th Nix pill. In the previous <link linkend="callpackage-design-pattern">13th</link> pill we have introduced the <literal>callPackage</literal> pattern, used to simplify the composition of software in a repository.
</para>
<para>
The next design pattern is less necessary but useful in many cases and it's a good exercise to learn more about Nix.
</para>
<section>
<title>About composability</title>
<para>
Functional languages are known for being able to compose functions. In particular, you gain a lot from functions that are able to manipulate the original value into a new value having the same structure. So that in the end we're able to call multiple functions to have the desired modifications.
</para>
<para>
In Nix we mostly talk about <emphasis role="bold">functions</emphasis> that accept inputs in order to return <emphasis role="bold">derivations</emphasis>. In our world we want nice utility functions that are able to manipulate those structures. These utilities add some useful properties to the original value, and we must be able to apply more utilities on top of it.
</para>
<para>
For example let's say we have an initial derivation drv and we want it to be a drv with debugging information and also to apply some custom patches:
</para>
<screen>debugVersion (applyPatches [ ./patch1.patch ./patch2.patch ] drv)</screen>
<para>
The final result will be still the original derivation plus some changes. That's both interesting and very different from other packaging approaches, which is a consequence of using a functional language to describe packages.
</para>
<para>
Designing such utilities is not trivial in a functional language that is not statically typed, because understanding what can or cannot be composed is difficult. But we try to do the best.
</para>
</section>
<section>
<title>The override pattern</title>
<para>
In the <link linkend="inputs-design-pattern">pill 12</link> we introduced the inputs design pattern. We do not return a derivation picking dependencies directly from the repository, rather we declare the inputs and let the callers pass the necessary arguments.
</para>
<para>
In our repository we have a set of attributes that import the expressions of the packages and pass these arguments, getting back a derivation. Let's take for example the <package>graphviz</package> attribute:
</para>
<screen>graphviz = import ./graphviz.nix { inherit mkDerivation gd fontconfig libjpeg bzip2; };</screen>
<para>
If we wanted to produce a derivation of <package>graphviz</package> with a customized <package>gd</package> version, we would have to repeat most of the above plus specifying an alternative <package>gd</package>:
</para>
<screen><xi:include href="./14/mygraphviz.txt" parse="text" /></screen>
<para>
That's hard to maintain. Using callPackage it would be easier:
</para>
<screen>mygraphviz = callPackage ./graphviz.nix { gd = customgd; };</screen>
<para>
But we may still be diverging from the original <package>graphviz</package> in the repository.
</para>
<para>
We would like to avoid specifying the nix expression again, instead reuse the original <package>graphviz</package> attribute in the repository and add our overrides like this:
</para>
<screen>mygraphviz = graphviz.override { gd = customgd; };</screen>
<para>
The difference is obvious, as well as the advantages of this approach.
</para>
<para>
<emphasis role="underline">Note:</emphasis> that <literal>.override</literal> is not a "method" in the OO sense as you may think. Nix is a functional language. That <literal>.override</literal> is simply an attribute of a set.
</para>
</section>
<section>
<title>The override implementation</title>
<para>
I remind you, the <package>graphviz</package> attribute in the repository is the derivation returned by the function imported from <filename>graphviz.nix</filename>. We would like to add a further attribute named "<literal>override</literal>" to the returned set.
</para>
<para>
Let's start simple by first creating a function "<literal>makeOverridable</literal>" that takes a function and a set of original arguments to be passed to the function.
</para>
<para>
<emphasis role="underline">Contract:</emphasis> the wrapped function must return a set.
</para>
<para>
Let's write a lib.nix:
</para>
<screen><xi:include href="./14/make-overridable-lib.txt" parse="text" /></screen>
<para>
So <literal>makeOverridable</literal> takes a function and a set of original arguments. It returns the original returned set, plus a new <literal>override</literal> attribute.
</para>
<para>
This <literal>override</literal> attribute is a function taking a set of new arguments, and returns the result of the original function called with the original arguments unified with the new arguments. What a mess.
</para>
<para>
Let's try it with nix-repl:
</para>
<screen><xi:include href="./14/nix-repl-make-overridable-test.txt" parse="text" /></screen>
<para>
Note that the function <literal>f</literal> does not return the plain sum but a set, because of the contract. You didn't forget already, did you? :-)
</para>
<para>
The variable <literal>res</literal> is the result of the function call without any override. It's easy to see in the definition of <literal>makeOverridable</literal>. In addition you can see the new <literal>override</literal> attribute being a function.
</para>
<para>
Calling that <literal>.override</literal> with a set will invoke the original function with the overrides, as expected.
</para>
<para>
But: we can't override again! Because the returned set with result 15 does not have an <literal>override</literal> attribute!
</para>
<para>
That's bad, it breaks further compositions.
</para>
<para>
The solution is simple, the <literal>.override</literal> function should make the result overridable again:
</para>
<screen><xi:include href="./14/nix-repl-make-overridable-test.txt" parse="text" /></screen>
<para>
Please note the <literal>rec</literal> keyword. It's necessary so that we can refer to <literal>makeOverridable</literal> from <literal>makeOverridable</literal> itself.
</para>
<para>
Now let's try overriding twice:
</para>
<screen><xi:include href="./14/nix-repl-make-overridable-twice.txt" parse="text" /></screen>
<para>
Success! The result is 30, as expected because a is overridden to 10 in the first override, and b to 20.
</para>
<para>
Now it would be nice if <literal>callPackage</literal> made our derivations overridable. That was the goal of this pill after all. This is an exercise for the reader.
</para>
</section>
<section>
<title>Conclusion</title>
<para>
The "<literal>override</literal>" pattern simplifies the way we customize packages starting from an existing set of packages. This opens a world of possibilities about using a central repository like <literal>nixpkgs</literal>, and defining overrides on our local machine without even modifying the original package.
</para>
<para>
Dream of a custom isolated <command>nix-shell</command> environment for testing <package>graphviz</package> with a custom <package>gd</package>:
</para>
<screen>debugVersion (graphviz.override { gd = customgd; })</screen>
<para>
Once a new version of the overridden package comes out in the repository, the customized package will make use of it automatically.
</para>
<para>
The key in Nix is to find powerful yet simple abstractions in order to let the user customize his environment with highest consistency and lowest maintenance time, by using predefined composable components.
</para>
</section>
<section>
<title>Next pill</title>
<para>
...we will talk about Nix search paths. By search path I mean a place in the file system where Nix looks for expressions. You may have wondered, where does that holy <literal>&lt;nixpkgs&gt;</literal> come from?
</para>
</section>
</chapter>

7
pills/14/make-overridable-lib.txt Normal file
View file

@ -0,0 +1,7 @@
{
makeOverridable = f: origArgs:
let
origRes = f origArgs;
in
origRes // { override = newArgs: f (origArgs // newArgs); };
}

4
pills/14/mygraphviz.txt Normal file
View file

@ -0,0 +1,4 @@
mygraphviz = import ./graphviz.nix {
inherit mkDerivation fontconfig libjpeg bzip2;
gd = customgd;
};

11
pills/14/nix-repl-make-overridable-test.txt Normal file
View file

@ -0,0 +1,11 @@
$ nix-repl
nix-repl> :l lib.nix
Added 1 variables.
nix-repl> f = { a, b }: { result = a+b; }
nix-repl> f { a = 3; b = 5; }
{ result = 8; }
nix-repl> res = makeOverridable f { a = 3; b = 5; }
nix-repl> res
{ override = «lambda»; result = 8; }
nix-repl> res.override { a = 10; }
{ result = 15; }

9
pills/14/nix-repl-make-overridable-twice.txt Normal file
View file

@ -0,0 +1,9 @@
nix-repl> :l lib.nix
Added 1 variables.
nix-repl> f = { a, b }: { result = a+b; }
nix-repl> res = makeOverridable f { a = 3; b = 5; }
nix-repl> res2 = res.override { a = 10; }
nix-repl> res2
{ override = «lambda»; result = 15; }
nix-repl> res2.override { b = 20; }
{ override = «lambda»; result = 30; }

7
pills/14/rec-make-overridable.txt Normal file
View file

@ -0,0 +1,7 @@
rec {
makeOverridable = f: origArgs:
let
origRes = f origArgs;
in
origRes // { override = newArgs: makeOverridable f (origArgs // newArgs); };
}

148
pills/15-nix-search-paths.xml
View file

@ -1,8 +1,146 @@
<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="nix-search-paths">
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="nix-search-paths">
<title>nix search paths</title>
<title>nix search paths</title>
<para>
Welcome to the 15th Nix pill. In the previous <link linkend="override-design-pattern">14th</link> pill we have introduced the "override" pattern, useful for writing variants of derivations by passing different inputs.
</para>
<para>
Assuming you followed the previous posts, I hope you are now ready to understand <literal>nixpkgs</literal>. But we have to find <literal>nixpkgs</literal> in our system first! So this is the step: introducing some options and environment variables used by nix tools.
</para>
<section>
<title>The NIX_PATH</title>
<para>
The <link xlink:href="http://nixos.org/nix/manual/#sec-common-env">NIX_PATH environment variable</link> is very important. It's very similar to the <varname>PATH</varname> environment variable. The syntax is similar, several paths are separated by a colon <literal>:</literal>. Nix will then search for something in those paths from left to right.
</para>
<para>
Who uses <varname>NIX_PATH</varname>? The nix expressions! Yes, <varname>NIX_PATH</varname> is not of much use by the nix tools themselves, rather it's used when writing nix expressions.
</para>
<para>
In the shell for example, when you execute the command <command>ping</command>, it's being searched in the <varname>PATH</varname> directories. The first one found is the one being used.
</para>
<para>
In nix it's exactly the same, however the syntax is different. Instead of just typing <command>ping</command> you have to type <literal>&lt;ping&gt;</literal>. Yes, I know... you are already thinking of <literal>&lt;nixpkgs&gt;</literal>. However don't stop reading here, let's keep going.
</para>
<para>
What's <varname>NIX_PATH</varname> good for? Nix expressions may refer to an "abstract" path such as <literal>&lt;nixpkgs&gt;</literal>, and it's possible to override it from the command line.
</para>
<para>
For ease we will use <command>nix-instantiate --eval</command> to do our tests. I remind you, <link xlink:href="http://nixos.org/nix/manual/#sec-nix-instantiate">nix-instantiate</link> is used to evaluate nix expressions and generate the .drv files. Here we are not interested in building derivations, so evaluation is enough. It can be used for one-shot expressions.
</para>
</section>
<section>
<title>Fake it a little</title>
<para>
It's useless from a nix view point, but I think it's useful for your own understanding. Let's use <varname>PATH</varname> itself as <varname>NIX_PATH</varname>, and try to locate <command>ping</command> (or another binary if you don't have it).
</para>
<screen><xi:include href="./15/nix-instantiate-ping.txt" parse="text" /></screen>
<para>
Great. At first attempt nix obviously said could not be found anywhere in the search path. Note that the <arg>-I</arg> option accepts a single directory. Paths added with <arg>-I</arg> take precedence over <varname>NIX_PATH</varname>.
</para>
<para>
The <varname>NIX_PATH</varname> also accepts a different yet very handy syntax: "<literal>somename=somepath</literal>". That is, instead of searching inside a directory for a name, we specify exactly the value of that name.
</para>
<screen><xi:include href="./15/ping-custom-path.txt" parse="text" /></screen>
<para>
Note in the second case how Nix checks whether the path exists or not.
</para>
</section>
<section>
<title>The path to repository</title>
<para>
You are out of curiosity, right?
</para>
<screen><xi:include href="./15/nixpkgs-path.txt" parse="text" /></screen>
<para>
You may have a different path, depending on how you added channels etc.. Anyway that's the whole point. The <literal>&lt;nixpkgs&gt;</literal> stranger that we used in our nix expressions, is referring to a path in the filesystem specified by <varname>NIX_PATH</varname>.
</para>
<para>
You can list that directory and realize it's simply a checkout of the nixpkgs repository at a specific commit (hint: <literal>.version-suffix</literal>).
</para>
<para>
The <varname>NIX_PATH</varname> variable is exported by <filename>nix.sh</filename>, and that's the reason why I always asked you to <link xlink:href="http://nixos.org/nix/manual/#idm47361539530016">source nix.s</link> at the beginning of my posts.
</para>
<para>
You may wonder: then I can also specify a different <link xlink:href="https://github.com/NixOS/nixpkgs">nixpkgs</link> path to, e.g., a <command>git checkout</command> of <literal>nixpkgs</literal>? Yes, you can and I encourage doing that. We'll talk about this in the next pill.
</para>
<para>
Let's define a path for our repository, then! Let's say all the <filename>default.nix</filename>, <filename>graphviz.nix</filename> etc. are under <filename>/home/nix/mypkgs</filename>:
</para>
<screen><xi:include href="./15/mypkgs-path.txt" parse="text" /></screen>
<para>
Yes, <command>nix-build</command> also accepts paths with angular brackets. We first evaluate the whole repository (<filename>default.nix</filename>) and then peek the <package>graphviz</package> attribute.
</para>
</section>
<section>
<title>A big word about nix-env</title>
<para>
The <link xlink:href="http://nixos.org/nix/manual/#sec-nix-env">nix-env</link> command is a little different than <command>nix-instantiate</command> and <command>nix-build</command>. Whereas <command>nix-instantiate</command> and <command>nix-build</command> require a starting nix expression, <command>nix-env</command> does not.
</para>
<para>
You may be crippled by this concept at the beginning, you may think <command>nix-env</command> uses <varname>NIX_PATH</varname> to find the <literal>nixpkgs</literal> repository. But that's not it.
</para>
<para>
The <command>nix-env</command> command uses <filename>~/.nix-defexpr</filename>, which is also part of <varname>NIX_PATH</varname> by default, but that's only a coincidence. If you empty <varname>NIX_PATH</varname>, <command>nix-env</command> will still be able to find derivations because of <filename>~/.nix-defexpr</filename>.
</para>
<para>
So if you run <command>nix-env -i graphvi</command> inside your repository, it will install the nixpkgs one. Same if you set <varname>NIX_PATH</varname> to point to your repository.
</para>
<para>
In order to specify an alternative to <filename>~/.nix-defexpr</filename> it's possible to use the <arg>-f</arg> option:
</para>
<screen><xi:include href="./15/mypkgs-graphviz-multiple.txt" parse="text" /></screen>
<para>
Oh why did it say there's another derivation named <package>graphviz</package>? Because both <literal>graphviz</literal> and <literal>graphvizCore</literal> attributes in our repository have the name "graphviz" for the derivation:
</para>
<screen><xi:include href="./15/mypkgs-query-all.txt" parse="text" /></screen>
<para>
By default <command>nix-env</command> parses all derivations and use the derivation names to interpret the command line. So in this case "graphviz" matched two derivations. Alternatively, like for <command>nix-build</command>, one can use <arg>-A</arg> to specify an attribute name instead of a derivation name:
</para>
<screen><xi:include href="./15/mypkgs-install-attr-graphviz.txt" parse="text" /></screen>
<para>
This form, other than being more precise, it's also faster because <command>nix-env</command> does not have to parse all the derivations.
</para>
<para>
For completeness: you may install <literal>graphvizCore</literal> with <arg>-A,</arg> since without the <arg>-A</arg> switch it's ambiguous.
</para>
<para>
In summary, it may happen when playing with nix that <command>nix-env</command> peeks a different derivation than <command>nix-build</command>. In such case you probably specified <varname>NIX_PATH</varname>, but <command>nix-env</command> is instead looking into <filename>~/.nix-defexpr</filename>.
</para>
<para>
Why is <command>nix-env</command> having this different behavior? I don't know specifically by myself either, but the answers could be:
<itemizedlist>
<listitem><para><command>nix-env</command> tries to be generic, thus it does not look for <literal>nixpkgs</literal> in <varname>NIX_PATH</varname>, rather it looks in <filename>~/.nix-defexpr</filename>.</para></listitem>
<listitem><para><command>nix-env</command> is able to merge multiple trees in <filename>~/.nix-defexpr</filename> by looking at all the possible derivations</para></listitem>
</itemizedlist>
</para>
<para>
It may also happen to you <emphasis role="bold">that you cannot match a derivation name when installing</emphasis>, because of the derivation name vs <arg>-A</arg> switch described above. Maybe <command>nix-env</command> wanted to be more friendly in this case for default user setups.
</para>
<para>
It may or may not make sense for you, or it's like that for historical reasons, but that's how it works currently, unless somebody comes up with a better idea.
</para>
</section>
<section>
<title>Conclusion</title>
<para>
The <varname>NIX_PATH</varname> variable is the search path used by nix when using the angular brackets syntax. It's possible to refer to "abstract" paths inside nix expressions and define the "concrete" path by means of <varname>NIX_PATH</varname>, or the usual <arg>-I</arg> flag in nix tools.
</para>
<para>
We've also explained some of the uncommon <command>nix-env</command> behaviors for newcomers. The <command>nix-env</command> tool does not use <varname>NIX_PATH</varname> to search for packages, but rather for <filename>~/.nix-defexpr</filename>. Beware of that!
</para>
<para>
In general do not abuse <varname>NIX_PATH</varname>, when possible use relative paths when writing your own nix expressions. Of course, in the case of <literal>&lt;nixpkgs&gt;</literal> in our repository, that's a perfectly fine usage of <varname>NIX_PATH</varname>. Instead, inside our repository itself, refer to expressions with relative paths like <filename>./hello.nix</filename>.
</para>
</section>
<section>
<title>Next pill</title>
<para>
...we will finally dive into <literal>nixpkgs</literal>. Most of the techniques we have developed in this series are already in <literal>nixpkgs</literal>, like <literal>mkDerivation</literal>, <literal>callPackage</literal>, <literal>override</literal>, etc., but of course better. With time, those base utilities get enhanced by the community with more features in order to handle more and more use cases and in a more general way.
</para>
</section>
</chapter>

4
pills/15/mypkgs-graphviz-multiple.txt Normal file
View file

@ -0,0 +1,4 @@
$ nix-env -f '<mypkgs>' -i graphviz
warning: there are multiple derivations named `graphviz'; using the first one
replacing old `graphviz'
installing `graphviz'

3
pills/15/mypkgs-install-attr-graphviz.txt Normal file
View file

@ -0,0 +1,3 @@
$ nix-env -f '<mypkgs>' -i -A graphviz
replacing old `graphviz'
installing `graphviz'

4
pills/15/mypkgs-path.txt Normal file
View file

@ -0,0 +1,4 @@
$ export NIX_PATH=mypkgs=/home/nix/mypkgs:$NIX_PATH
$ nix-instantiate --eval '<mypkgs>'
{ graphviz = <code>; graphvizCore = <code>; hello = <code>; mkDerivation = <code>; }

4
pills/15/mypkgs-query-all.txt Normal file
View file

@ -0,0 +1,4 @@
$ nix-env -f '<mypkgs>' -qaP
graphviz graphviz
graphvizCore graphviz
hello hello

6
pills/15/nix-instantiate-ping.txt Normal file
View file

@ -0,0 +1,6 @@
$ nix-instantiate --eval -E '<ping>'
error: file `ping' was not found in the Nix search path (add it using $NIX_PATH or -I)
$ NIX_PATH=$PATH nix-instantiate --eval -E '<ping>'
/bin/ping
$ nix-instantiate -I /bin --eval -E '<ping>'
/bin/ping

4
pills/15/nixpkgs-path.txt Normal file
View file

@ -0,0 +1,4 @@
$ nix-instantiate --eval -E '<nixpkgs>'
/home/nix/.nix-defexpr/channels/nixpkgs
$ echo $NIX_PATH
nixpkgs=/home/nix/.nix-defexpr/channels/nixpkgs

4
pills/15/ping-custom-path.txt Normal file
View file

@ -0,0 +1,4 @@
$ NIX_PATH="ping=/bin/ping" nix-instantiate --eval -E '<ping>'
/bin/ping
$ NIX_PATH="ping=/bin/foo" nix-instantiate --eval -E '<ping>'
error: file `ping' was not found in the Nix search path (add it using $N

128
pills/16-nixpkgs-parameters.xml
View file

@ -1,8 +1,126 @@
<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="nixpkgs-parameters">
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
version="5.0"
xml:id="nixpkgs-parameters">
<title>nixpkgs parameters</title>
<title>nixpkgs parameters</title>
<para>
Welcome to the 16th Nix pill. In the previous <link linkend="nix-search-paths">15th</link> pill we've realized how nix finds expressions with the angular brackets syntax, so that we finally know where is <literal>&lt;nixpkgs&gt;</literal> located on our system.
</para>
<para>
We can start diving into the <link xlink:href="https://github.com/NixOS/nixpkgs">nixpkgs repository</link>, through all the various tools and design patterns. Please note that also <literal>nixpkgs</literal> has its own manual, underlying the difference between the general <literal>nix</literal> language and the <literal>nixpkgs</literal> repository.
</para>
<section>
<title>The default.nix expression</title>
<para>
We will not start inspecting packages at the beginning, rather the general structure of <literal>nixpkgs</literal>.
</para>
<para>
In our custom repository we created a <filename>default.nix</filename> which composed the expressions of the various packages.
</para>
<para>
Also <literal>nixpkgs</literal> has its own <link xlink:href="https://github.com/NixOS/nixpkgs/blob/master/default.nix">default.nix</link>, which is the one being loaded when referring to <literal>&lt;nixpkgs&gt;</literal>. It does a simple thing: check whether the <literal>nix</literal> version is at least 1.7 (at the time of writing this blog post). Then import <link xlink:href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix">pkgs/top-level/all-packages.nix</link>. From now on, we will refer to this set of packages as <emphasis role="bold">pkgs</emphasis>.
</para>
<para>
The <filename>all-packages.nix</filename> is then the file that composes all the packages. Note the <filename>pkgs/</filename> subdirectory, while nixos is in the <filename>nixos/</filename> subdirectory.
</para>
<para>
The <filename>all-packages.nix</filename> is a bit contrived. First of all, it's a function. It accepts a couple of interesting parameters:
<itemizedlist>
<listitem><para><literal>system</literal>: defaults to the current system</para></listitem>
<listitem><para><literal>config</literal>: defaults to null</para></listitem>
<listitem><para>others...</para></listitem>
</itemizedlist>
</para>
<para>
The <emphasis role="bold">system</emphasis> parameter, as per comment in the expression, it's the system for which the packages will be built. It allows for example to install i686 packages on amd64 machines.
</para>
<para>
The <emphasis role="bold">config</emphasis> parameter is a simple attribute set. Packages can read some of its values and change the behavior of some derivations.
</para>
</section>
<section>
<title>The system parameter</title>
<para>
You will find this parameter in many other .nix expressions (e.g. release expressions). The reason is that, given pkgs accepts a system parameter, then whenever you want to import pkgs you also want to pass through the value of system. E.g.:
</para>
<para>
<filename>myrelease.nix</filename>:
</para>
<screen><xi:include href="./16/myrelease-nix.txt" parse="text" /></screen>
<para>
Why is it useful? With this parameter it's very easy to select a set of packages for a particular system. For example:
</para>
<screen>nix-build -A psmisc --argstr system i686-linux</screen>
<para>
This will build the <package>psmisc</package> derivation for i686-linux instead of x86_64-linux. This concept is very similar to multi-arch of Debian.
</para>
<para>
The setup for cross compiling is also in <literal>nixpkgs</literal>, however it's a little contrived to talk about it and I don't know much of it either.
</para>
</section>
<section>
<title>The config parameter</title>
<para>
I'm sure on the wiki or other manuals you've read about <filename>~/.nixpkgs/config.nix</filename> and I'm sure you've wondered whether that's hardcoded in nix. It's not, it's in <link xlink:href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix#L45">nixpkgs</link>.
</para>
<para>
The <filename>all-packages.nix</filename> expression accepts the <literal>config</literal> parameter. If it's <literal>null</literal>, then it reads the <varname>NIXPKGS_CONFIG</varname> environment variable. If not specified, <literal>nixpkgs</literal> will peek <filename>$HOME/.nixpkgs/config.nix</filename>.
</para>
<para>
After determining <filename>config.nix</filename>, it will be imported as nix expression, and that will be the value of <literal>config</literal> (in case it hasn't been passed as parameter to import <literal>&lt;nixpkgs&gt;</literal>).
</para>
<para>
The <literal>config</literal> is available in the resulting repository:
</para>
<screen><xi:include href="./16/config-foo-bar.txt" parse="text" /></screen>
<para>
What attributes go in <literal>config</literal> is a matter of convenience and conventions.
</para>
<para>
For example, <literal>config.allowUnfree</literal> is an attribute that forbids building packages that have an unfree license by default. The <literal>config.pulseaudio</literal> setting tells whether to build packages with <package>pulseaudio</package> support or not where applicable and when the derivation obeys to the setting.
</para>
</section>
<section>
<title>About .nix functions</title>
<para>
A <literal>.nix</literal> file contains a nix expression. Thus it can also be a function. I remind you that <command>nix-build</command> expects the expression to return a derivation. Therefore it's natural to return straight a derivation from a <literal>.nix</literal> file. However, it's also very natural for the <literal>.nix</literal> file to accept some parameters, in order to tweak the derivation being returned.
</para>
<para>
In this case, nix does a trick:
<itemizedlist>
<listitem><para>If the expression is a derivation, well build it.</para></listitem>
<listitem><para>If the expression is a function, call it and build the resulting derivation.</para></listitem>
</itemizedlist>
</para>
<para>
For example you can nix-build the <literal>.nix</literal> file below:
</para>
<screen><xi:include href="./16/pkgs-psmisc.txt" parse="text" /></screen>
<para>
Nix is able to call the function because the pkgs parameter has a default value. This allows you to pass a different value for pkgs using the <literal>--arg</literal> option.
</para>
<para>
Does it work if you have a function returning a function that returns a derivation? No, Nix only calls the function it encounters once.
</para>
</section>
<section>
<title>Conclusion</title>
<para>
We've unleashed the <literal>&lt;nixpkgs&gt;</literal> repository. It's a function that accepts some parameters, and returns the set of all packages. Due to laziness, only the accessed derivations will be built.
</para>
<para>
You can use this repository to build your own packages as we've seen in the previous pill when creating our own repository.
</para>
<para>
Lately I'm a little busy with the NixOS 14.11 release and other stuff, and I'm also looking toward migrating from blogger to a more coder-oriented blogging platform. So sorry for the delayed and shorter pills :)
</para>
</section>
<section>
<title>Next pill</title>
<para>
...we will talk about overriding packages in the <literal>nixpkgs</literal> repository. What if you want to change some options of a library and let all other packages pick the new library? One possibility is to use, like described above, the <literal>config</literal> parameter when applicable. The other possibility is to override derivations.
</para>
</section>
</chapter>

7
pills/16/config-foo-bar.txt Normal file
View file

@ -0,0 +1,7 @@
$ nix-repl
nix-repl> pkgs = import <nixpkgs> {}
nix-repl> pkgs.config
{ }
nix-repl> pkgs = import <nixpkgs> { config = { foo = "bar"; }; }
nix-repl> pkgs.config
{ foo = "bar"; }

4
pills/16/myrelease-nix.txt Normal file
View file

@ -0,0 +1,4 @@
{ system ? builtins.currentSystem }:
let pkgs = import <nixpkgs> { inherit system; };
...

3
pills/16/pkgs-psmisc.txt Normal file
View file

@ -0,0 +1,3 @@
{ pkgs ? import <nixpkgs> {} }:
pkgs.psmisc