Mirrors/nix-pills
2
0
Fork 0
mirror of https://github.com/NixOS/nix-pills synced 2024-11-10 13:54:14 +00:00
Code Issues Projects Releases Packages Wiki Activity

Ported Pill 6.

Browse source
This commit is contained in:
Matan Bendix Shenhav 2017-08-13 19:30:56 +03:00
parent dfc15beb0d
commit e302529c5f
24 changed files with 491 additions and 6 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

8
pills/05-functions-and-imports.xml
View file

@ -3,13 +3,11 @@ xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude" version="5.0"
xml:id="functions-and-imports">
<title>functions and imports</title>
<title>functions and imports</title>
<section>
<title>Functions</title>
<para>
Welcome to the fifth Nix pill. In the previous <link
linkend="basics-of-the-language">fourth pill</link> we touched the
linkend="basics-of-language">fourth pill</link> we touched the
Nix language for a moment. We introduced basic types and values of
the Nix language, and basic expressions such as
<code>if</code>, <code>with</code> and
@ -31,8 +29,6 @@ xml:id="functions-and-imports">
~/.nix-profile/etc/profile.d/nix.sh</code>
</para>
</section>
<section>
<title>Nameless and single parameter</title>

390
pills/06-our-first-derivation.xml
View file

@ -5,4 +5,394 @@
xml:id="our-first-derivation">
<title>our first derivation</title>
<para>
Welcome to the sixth Nix pill. In the previous <link
linkend="functions-and-imports">fifth pill</link> we introduced functions and imports. Functions and imports are very simple concepts that allows for building complex abstractions and composition of modules to build a flexible Nix system.
</para>
<para>
In this post we finally arrived to writing a derivation. Derivations are the building blocks of a Nix system, from a file system view point. The Nix language is used to describe such derivations.
</para>
<para>
I remind you how to enter the Nix environment: <code>source ~/.nix-profile/etc/profile.d/nix.sh</code>
</para>
<section>
<title>The derivation function</title>
<para>
The <link xlink:href="https://nixos.org/nix/manual/#ssec-derivation">derivation built-in function</link> is used to create derivations. I invite you to read the link in the Nix manual about the derivation built-in. A derivation from a Nix language view point is simply a set, with some attributes. Therefore you can pass the derivation around with variables like anything else.
</para>
<para>
That's where the real power comes in.
</para>
<para>
The <code>derivation</code> function receives a set as first argument. This set requires at least the following three attributes:
</para>
<itemizedlist mark='bullet'>
<listitem>
<para>
name: the name of the derivation. In the nix store the format is hash-name, that's the name.
</para>
</listitem>
<listitem>
<para>
system: is the name of the system in which the derivation can be built. For example, x86_64-linux.
</para>
</listitem>
<listitem>
<para>
builder: it is the binary program that builds the derivation.
</para>
</listitem>
</itemizedlist>
<para>
First of all, what's the name of our system as seen by nix?
</para>
<screen><xi:include href="./06/current-system.txt" parse="text" /></screen>
<para>
Let's try to fake the name of the system:
</para>
<screen><xi:include href="./06/fake-system.txt" parse="text" /></screen>
<para>
Oh oh, what's that? Did it build the derivation? No it didn't, but it <emphasis role="strong">did create the .drv file</emphasis>. nix-repl does not build derivations unless you tell to do so.
</para>
</section>
<section>
<title>Digression about .drv files</title>
<para>
What's that <filename>.drv</filename> file? It is the specification of how to build the derivation, without all the Nix language fuzz.
</para>
<para>
Before continuing, some analogies with the C language:
</para>
<itemizedlist mark='bullet'>
<listitem>
<para>
<filename>.nix</filename> files are like <filename>.c</filename> files
</para>
</listitem>
<listitem>
<para>
<filename>.drv</filename> files are intermediate files like <filename>.o</filename> files. The <filename>.drv</filename> describes how to build a derivation, it's the bare minimum information.
</para>
</listitem>
<listitem>
<para>
out paths are then the product of the build
</para>
</listitem>
</itemizedlist>
<para>
Both drv paths and out paths are stored in the nix store as you can see.
</para>
<para>
What's in that <filename>.drv</filename> file? You can read it, but it's better to pretty print it.
</para>
<screen><xi:include href="./06/pretty-print.txt" parse="text" /></screen>
<para>
If you feel the above command being too slow (ignore the meaning of the command below, just do it):
</para>
<screen><xi:include href="./06/pretty-print-fast.txt" parse="text" /></screen>
<para>
The installed <code>pp-aterm</code> program can be used to pretty print <filename>.drv</filename> files:
</para>
<screen><xi:include href="./06/pp-aterm.txt" parse="text" /></screen>
<para>
Ok we can see there's an out path, but it does not exist yet. We never told Nix to build it, but we know beforehand where the build output will be. Why?
</para>
<para>
Think, if Nix ever built the derivation just because we accessed it in Nix, we would have to wait a long time if it was, say, Firefox. That's why Nix let us know the path beforehand and keep evaluating the Nix expressions, but it's still empty because no build was ever made.
</para>
<para>
<emphasis role="underline">Important</emphasis>: the hash of the out path is based solely on the input derivations in the current version of Nix, not on the contents of the build product. It's possible however to have <link xlink:href="https://en.wikipedia.org/wiki/Content-addressable_storage">content-addressable</link> derivations for e.g. tarballs as we'll see later on.
</para>
<para>
Many things are empty in that <filename>.drv</filename>, however I write a summary of the <link xlink:href="http://nixos.org/~eelco/pubs/phd-thesis.pdf">.drv format</link> for you:
</para>
<orderedlist>
<listitem>
<para>
The output paths (they can be multiple ones). By default nix creates one out path called "out".
</para>
</listitem>
<listitem>
<para>
The list of input derivations. It's empty because we are not referring to any other derivation. Otherwise, there would a list of other .drv files.
</para>
</listitem>
<listitem>
<para>
The system and the builder executable (yes, it's a fake one).
</para>
</listitem>
<listitem>
<para>
Then a list of environment variables passed to the builder.
</para>
</listitem>
</orderedlist>
<para>
That's it, the minimum necessary information to build our derivation.
</para>
<para>
<emphasis role="underline">Important note</emphasis>: the environment variables passed to the builder are just those you see in the .drv plus some other Nix related configuration (number of cores, temp dir, ...). The builder will not inherit any variable from your running shell, otherwise builds would suffer from <link xlink:href="https://wiki.debian.org/ReproducibleBuilds">non-determinism</link>.
</para>
<para>
Back to our fake derivation.
</para>
<para>
Let's build our really fake derivation:
</para>
<screen><xi:include href="./06/build-derivation.txt" parse="text" /></screen>
<para>
The <code>:b</code> is a nix-repl specific command to build a derivation. You can see more commands with <code>:?</code> . So in the output you can see that it takes the <filename>.drv</filename> as information on how to build the derivation. Then it says it's trying to produce our out path. Finally the error we were waiting for: that derivation can't be built on our system.
</para>
<para>
We're doing the build inside nix-repl, but what if we don't want to use nix-repl? You can <emphasis role="strong">realise</emphasis> a <filename>.drv</filename> with:
</para>
<screen><xi:include href="./06/realise-derivation.txt" parse="text" /></screen>
<para>
You will get the same output as before.
</para>
<para>
Let's fix the system attribute:
</para>
<screen><xi:include href="./06/fix-attribute.txt" parse="text" /></screen>
<para>
A step forward: of course, that <code>mybuilder</code> executable does not really exist. Stop for a moment.
</para>
</section>
<section>
<title>What's in a derivation set</title>
<para>
I find useful to inspect the returned value from the derivation function for you. First of all, the returned value is a plain set:
</para>
<screen><xi:include href="./06/inspect-values.txt" parse="text" /></screen>
<para>
You can guess what <code>builtins.isAttrs</code> does, it returns true if the argument is a set. While <code>builtins.attrNames</code> returns a list of keys of the given set. Some kind of reflection, you might say.
</para>
<para>
Start from drvAttrs:
</para>
<screen><xi:include href="./06/drvattrs.txt" parse="text" /></screen>
<para>
That's basically the input we gave to the derivation function. Also <code>d.name</code>, <code>d.system</code> and <code>d.builder</code> attributes are straight the ones we gave as input.
</para>
<screen><xi:include href="./06/check-drvattrs.txt" parse="text" /></screen>
<para>
So out is just the derivation itself, it seems weird but the reason is that we only have one output from the derivation. That's also the reason why <code>d.all</code> is a singleton. We'll see multiple outputs later.
</para>
<para>
The <code>d.drvPath</code> is the path of the <filename>.drv</filename> file: <filename>/nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-<emphasis role="strong">myname.drv</emphasis></filename>.
</para>
<para>
Something interesting is the <code>type</code> attribute. It's <code>"derivation"</code>. Nix does add a little of magic to sets with type derivation, but not that much. To let you understand, you can create yourself a set with that type, it's a simple set:
</para>
<screen><xi:include href="./06/type-derivation.txt" parse="text" /></screen>
<para>
Of course it has no other information, so Nix doesn't know what to say :-) But you get it, the <code>type = "derivation"</code> is just a convention for Nix and for us to understand the set is a derivation.
</para>
<para>
When writing packages, we are interested in the outputs. The other metadata is needed for Nix to know how to create the drv path and the out path.
</para>
<para>
The outPath attribute is the build path in the nix store: <filename>/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-<emphasis role="strong">myname</emphasis></filename>.
</para>
</section>
<section>
<title>Referring to other derivations</title>
<para>
Just like dependencies in other package managers, how do we refer to other packages? How do we refer to other derivations in terms of files on the disk? We use the outPath. The outPath tells where the files are of that derivation. To make it more convenient, Nix is able to do a conversion from a derivation set to a string.
</para>
<screen><xi:include href="./06/outpath.txt" parse="text" /></screen>
<para>
Nix does the "set to string conversion" as long as there is the <code>outPath</code> attribute (much like a toString method in other languages):
</para>
<screen><xi:include href="./06/tostring.txt" parse="text" /></screen>
<para>
Say we want to use binaries from coreutils (ignore the nixpkgs etc.):
</para>
<screen><xi:include href="./06/coreutils.txt" parse="text" /></screen>
<para>
Apart the nixpkgs stuff, just think we added to the scope a series of variables. One of them is coreutils. It is the derivation of the coreutils package you all know of from other Linux distributions. It contains basic binaries for GNU/Linux systems (you may have multiple derivations of coreutils in the nix store, no worries):
</para>
<screen><xi:include href="./06/list-coreutils.txt" parse="text" /></screen>
<para>
I remind you, inside strings it's possible to interpolate Nix expressions with <code>${...}</code>:
</para>
<screen><xi:include href="./06/interpolate.txt" parse="text" /></screen>
<para>
That's very convenient, because then we could refer to e.g. the bin/true binary like this:
</para>
<screen><xi:include href="./06/reference.txt" parse="text" /></screen>
</section>
<section>
<title>An almost working derivation</title>
<para>
In the previous attempt we used a fake builder, <code>mybuilder</code> which obviously does not exist. But we can use for example bin/true, which always exits with 0 (success).
</para>
<screen><xi:include href="./06/test-build.txt" parse="text" /></screen>
<para>
Another step forward, it executed the builder (bin/true), but the builder did not create the out path of course, it just exited with 0.
</para>
<para>
<emphasis role="underline">Obvious note</emphasis>: everytime we change the derivation, a new hash is created.
</para>
<para>
Let's examine the new <filename>.drv</filename> now that we referred to another derivation:
</para>
<screen><xi:include href="./06/examine-build.txt" parse="text" /></screen>
<para>
Aha! Nix added a dependency to our myname.drv, it's the coreutils.drv. Before doing our build, Nix should build the coreutils.drv. But since coreutils is already in our nix store, no build is needed, it's already there with out path <filename>/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-<emphasis role="strong">coreutils-8.21</emphasis></filename>.
</para>
</section>
<section>
<title>When is the derivation built</title>
<para>
Nix does not build derivations <emphasis role="strong">during evaluation</emphasis> of Nix expressions. In fact, that's why we have to do ":b drv" in nix-repl, or use nix-store -r in the first place.
</para>
<para>
An important separation is made in Nix:
</para>
<itemizedlist mark='bullet'>
<listitem>
<para>
<emphasis role="strong">Instantiate/Evaluation time</emphasis>: the Nix expression is parsed, interpreted and finally returns a derivation set. During evaluation, you can refer to other derivations because Nix will create .drv files and we will know out paths beforehand. This is achieved with <link xlink:href="https://nixos.org/nix/manual/#sec-nix-instantiate">nix-instantiate</link>.
</para>
</listitem>
<listitem>
<para>
<emphasis role="strong">Realise/Build time</emphasis>: the .drv from the derivation set is built, first building .drv inputs (build dependencies). This is achieved with <link xlink:href="https://nixos.org/nix/manual/#rsec-nix-store-realise">nix-store -r</link>.
</para>
</listitem>
</itemizedlist>
<para>
Think of it as of compile time and link time like with C/C++ projects. You first compile all source files to object files. Then link object files in a single executable.
</para>
<para>
In Nix, first the Nix expression (usually in a .nix file) is compiled to .drv, then each .drv is built and the product is installed in the relative out paths.
</para>
</section>
<section>
<title>Conclusion</title>
<para>
Is that complicated to create a package for Nix? No it's not.
</para>
<para>
We're walking through the fundamentals of Nix derivations, to understand how they work, how they are represented.Packaging in Nix is certainly easier than that, but we're not there yet in this post. More Nix pills are needed.
</para>
<para>
With the derivation function we provide a set of information on how to build a package, and we get back the information about where the package was built. Nix converts a set to a string when there's an outPath, that's very convenient. With that, it's easy to refer to other derivations.
</para>
<para>
When Nix builds a derivation, it first creates a .drv file from a derivation expression, and uses it to build the output. It does so recursively for all the dependencies (inputs). It "executes" the .drv files like a machine. Not much magic after all.
</para>
</section>
<section>
<title>Next pill</title>
<para>
...we will finally write our first <emphasis role="strong">working</emphasis> derivation. Yes, this post is about "our first derivation", but I never said it was a working one <inlinemediaobject><imageobject><imagedata fileref="06/troll.svg"/></imageobject></inlinemediaobject>.
</para>
</section>
</chapter>

7
pills/06/build-derivation.txt Normal file
View file

@ -0,0 +1,7 @@
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; }
nix-repl> :b d
[...]
these derivations will be built:
/nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv
building path(s) `/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname'
error: a `mysystem' is required to build `/nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv', but I am a `x86_64-linux'

2
pills/06/check-drvattrs.txt Normal file
View file

@ -0,0 +1,2 @@
nix-repl> (d == d.out)
true

6
pills/06/coreutils.txt Normal file
View file

@ -0,0 +1,6 @@
nix-repl> :l <nixpkgs>
Added 3950 variables.
nix-repl> coreutils
«derivation /nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-coreutils-8.21.drv»
nix-repl> builtins.toString coreutils
"/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21"

2
pills/06/current-system.txt Normal file
View file

@ -0,0 +1,2 @@
nix-repl> builtins.currentSystem
"x86_64-linux"

2
pills/06/drvattrs.txt Normal file
View file

@ -0,0 +1,2 @@
nix-repl> d.drvAttrs
{ builder = "mybuilder"; name = "myname"; system = "mysystem"; }

13
pills/06/examine-build.txt Normal file
View file

@ -0,0 +1,13 @@
$ pp-aterm -i /nix/store/d4xczdij7xazjfm5kn4nmphx63mpv676-myname.drv
Derive(
[("out", "/nix/store/fy5lyr5iysn4ayyxvpnsya8r5y5bwjnl-myname", "", "")]
, [("/nix/store/1zcs1y4n27lqs0gw4v038i303pb89rw6-coreutils-8.21.drv", ["out"])]
, []
, "x86_64-linux"
, "/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21/bin/true"
, []
, [ ("builder", "/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21/bin/true")
, ("name", "myname")
, ("out", "/nix/store/fy5lyr5iysn4ayyxvpnsya8r5y5bwjnl-myname")
, ("system", "x86_64-linux")
]

3
pills/06/fake-system.txt Normal file
View file

@ -0,0 +1,3 @@
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; }
nix-repl> d
«derivation /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv»

4
pills/06/fix-attribute.txt Normal file
View file

@ -0,0 +1,4 @@
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = builtins.currentSystem; }
nix-repl> :b d
[...]
build error: invalid file name `mybuilder'

5
pills/06/inspect-values.txt Normal file
View file

@ -0,0 +1,5 @@
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; }
nix-repl> builtins.isAttrs d
true
nix-repl> builtins.attrNames d
[ "all" "builder" "drvAttrs" "drvPath" "name" "out" "outPath" "outputName" "system" "type" ]

4
pills/06/interpolate.txt Normal file
View file

@ -0,0 +1,4 @@
nix-repl> "${d}"
"/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname"
nix-repl> "${coreutils}"
"/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21"

2
pills/06/list-coreutils.txt Normal file
View file

@ -0,0 +1,2 @@
$ ls /nix/store/*coreutils*/bin
[...]

4
pills/06/outpath.txt Normal file
View file

@ -0,0 +1,4 @@
nix-repl> d.outPath
"/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-<emphasis role="strong">myname</emphasis>"
nix-repl> builtins.toString d
"/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-<emphasis role="strong">myname</emphasis>"

13
pills/06/pp-aterm.txt Normal file
View file

@ -0,0 +1,13 @@
$ pp-aterm -i /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv
Derive(
[("out", "/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname", "", "")]
, []
, []
, "mysystem"
, "mybuilder"
, []
, [ ("builder", "mybuilder")
, ("name", "myname")
, ("out", "/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname")
, ("system", "mysystem")
]

1
pills/06/pretty-print-fast.txt Normal file
View file

@ -0,0 +1 @@
$ nix-env -iA strategoPackages018.strategoxt -f '<nixpkgs>'

1
pills/06/pretty-print.txt Normal file
View file

@ -0,0 +1 @@
$ nix-env -i strategoxt

1
pills/06/realise-derivation.txt Normal file
View file

@ -0,0 +1 @@
$ nix-store -r /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv

2
pills/06/reference.txt Normal file
View file

@ -0,0 +1,2 @@
nix-repl> "${coreutils}/bin/true"
"/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21/bin/true"

5
pills/06/test-build.txt Normal file
View file

@ -0,0 +1,5 @@
nix-repl> :l <nixpkgs>
nix-repl> d = derivation { name = "myname"; builder = "${coreutils}/bin/true"; system = builtins.currentSystem; }
nix-repl> :b d
[...]
builder for `/nix/store/d4xczdij7xazjfm5kn4nmphx63mpv676-myname.drv' failed to produce output path `/nix/store/fy5lyr5iysn4ayyxvpnsya8r5y5bwjnl-myname'

4
pills/06/tostring.txt Normal file
View file

@ -0,0 +1,4 @@
nix-repl> builtins.toString { outPath = "foo"; }
"foo"
nix-repl> builtins.toString { a = "b"; }
error: cannot coerce a set to a string, at (string):1:1

BIN
pills/06/troll.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 69 KiB

16
pills/06/troll.svg Normal file
View file

File diff suppressed because one or more lines are too long
Side by side

After

Width:  |  Height:  |  Size: 26 KiB

2
pills/06/type-derivation.txt Normal file
View file

@ -0,0 +1,2 @@
nix-repl> { type = "derivation"; }
«derivation ???»