Mirrors/nix-pills
2
0
Fork 0
mirror of https://github.com/NixOS/nix-pills synced 2024-11-10 05:44:14 +00:00
Code Issues Projects Releases Packages Wiki Activity

Replace \' with '.

Browse source
This commit is contained in:
Noam Yorav-Raphael 2024-04-04 00:17:01 +03:00 committed by Jan Tojnar
parent 38c9f8c47c
commit cd05ea088e
21 changed files with 291 additions and 291 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
pills/00-preface.md
View file

@ -1,6 +1,6 @@
# Preface
This is a ported version of the **Nix Pills**, a series of blog posts written by **Luca Bruno** (aka Lethalman) and originally published in 2014 and 2015. It provides a tutorial introduction into the Nix package manager and Nixpkgs package collection, in the form of short chapters called \'pills\'.
This is a ported version of the **Nix Pills**, a series of blog posts written by **Luca Bruno** (aka Lethalman) and originally published in 2014 and 2015. It provides a tutorial introduction into the Nix package manager and Nixpkgs package collection, in the form of short chapters called 'pills'.
Since the Nix Pills are considered a classic introduction to Nix, an effort to port them to the current format was led by Graham Christensen (aka grahamc / gchristensen) and other contributors in 2017.

38
pills/01-why-you-should-give-it-try.md
View file

@ -4,7 +4,7 @@
Welcome to the first post of the \"[Nix](https://nixos.org/nix) in pills\" series. Nix is a purely functional package manager and deployment system for POSIX.
There\'s a lot of documentation that describes what Nix, [NixOS](https://nixos.org/nixos) and related projects are. But the purpose of this post is to convince you to give Nix a try. Installing NixOS is not required, but sometimes I may refer to NixOS as a real world example of Nix usage for building a whole operating system.
There's a lot of documentation that describes what Nix, [NixOS](https://nixos.org/nixos) and related projects are. But the purpose of this post is to convince you to give Nix a try. Installing NixOS is not required, but sometimes I may refer to NixOS as a real world example of Nix usage for building a whole operating system.
## Rationale for this series
@ -12,7 +12,7 @@ The [Nix](https://nixos.org/manual/nix), [Nixpkgs](https://nixos.org/manual/nixp
This series aims to complement the existing explanations from the more formal documents.
The following is a description of Nix. Just as with pills, I\'ll try to be as short as possible.
The following is a description of Nix. Just as with pills, I'll try to be as short as possible.
## Not being purely functional
@ -20,17 +20,17 @@ Most, if not all, widely used package managers ([dpkg](https://wiki.debian.org/d
There are some attempts to mitigate this problem. Debian, for example, partially solves the problem with the [alternatives](https://wiki.debian.org/DebianAlternatives) system.
So while in theory it\'s possible with some current systems to install multiple versions of the same package, in practice it\'s very painful.
So while in theory it's possible with some current systems to install multiple versions of the same package, in practice it's very painful.
Let\'s say you need an nginx service and also an nginx-openresty service. You have to create a new package that changes all the paths to have, for example, an `-openresty` suffix.
Let's say you need an nginx service and also an nginx-openresty service. You have to create a new package that changes all the paths to have, for example, an `-openresty` suffix.
Or suppose that you want to run two different instances of mysql: 5.2 and 5.5. The same thing applies, plus you have to also make sure the two mysqlclient libraries do not collide.
This is not impossible but it *is* very inconvenient. If you want to install two whole stacks of software like GNOME 3.10 and GNOME 3.12, you can imagine the amount of work.
From an administrator\'s point of view: you can use containers. The typical solution nowadays is to create a container per service, especially when different versions are needed. That somewhat solves the problem, but at a different level and with other drawbacks. For example, needing orchestration tools, setting up a shared cache of packages, and new machines to monitor rather than simple services.
From an administrator's point of view: you can use containers. The typical solution nowadays is to create a container per service, especially when different versions are needed. That somewhat solves the problem, but at a different level and with other drawbacks. For example, needing orchestration tools, setting up a shared cache of packages, and new machines to monitor rather than simple services.
From a developer\'s point of view: you can use virtualenv for python, or jhbuild for gnome, or whatever else. But then how do you mix the two stacks? How do you avoid recompiling the same thing when it could instead be shared? Also you need to set up your development tools to point to the different directories where libraries are installed. Not only that, there\'s the risk that some of the software incorrectly uses system libraries.
From a developer's point of view: you can use virtualenv for python, or jhbuild for gnome, or whatever else. But then how do you mix the two stacks? How do you avoid recompiling the same thing when it could instead be shared? Also you need to set up your development tools to point to the different directories where libraries are installed. Not only that, there's the risk that some of the software incorrectly uses system libraries.
And so on. Nix solves all this at the packaging level and solves it well. A single tool to rule them all.
@ -38,30 +38,30 @@ And so on. Nix solves all this at the packaging level and solves it well. A sing
Nix makes no assumptions about the global state of the system. This has many advantages, but also some drawbacks of course. The core of a Nix system is the Nix store, usually installed under `/nix/store`, and some tools to manipulate the store. In Nix there is the notion of a *derivation* rather than a package. The difference can be subtle at the beginning, so I will often use the words interchangeably.
Derivations/packages are stored in the Nix store as follows: `/nix/store/hash-name`, where the hash uniquely identifies the derivation (this isn\'t quite true, it\'s a little more complex), and the name is the name of the derivation.
Derivations/packages are stored in the Nix store as follows: `/nix/store/hash-name`, where the hash uniquely identifies the derivation (this isn't quite true, it's a little more complex), and the name is the name of the derivation.
Let\'s take a bash derivation as an example: `/nix/store/s4zia7hhqkin1di0f187b79sa2srhv6k-bash-4.2-p45/`. This is a directory in the Nix store which contains `bin/bash`.
Let's take a bash derivation as an example: `/nix/store/s4zia7hhqkin1di0f187b79sa2srhv6k-bash-4.2-p45/`. This is a directory in the Nix store which contains `bin/bash`.
What that means is that there\'s no `/bin/bash`, there\'s only that self-contained build output in the store. The same goes for coreutils and everything else. To make them convenient to use from the shell, Nix will arrange for binaries to appear in your `PATH` as appropriate.
What that means is that there's no `/bin/bash`, there's only that self-contained build output in the store. The same goes for coreutils and everything else. To make them convenient to use from the shell, Nix will arrange for binaries to appear in your `PATH` as appropriate.
What we have is basically a store of all packages (with different versions occupying different locations), and everything in the Nix store is immutable.
In fact, there\'s no ldconfig cache either. So where does bash find libc?
In fact, there's no ldconfig cache either. So where does bash find libc?
$ ldd `which bash`
libc.so.6 => /nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib/libc.so.6 (0x00007f0248cce000)
It turns out that when bash was built, it was built against that specific version of glibc in the Nix store, and at runtime it will require exactly that glibc version.
Don\'t be confused by the version in the derivation name: it\'s only a name for us humans. You may end up having two derivations with the same name but different hashes: it\'s the hash that really matters.
Don't be confused by the version in the derivation name: it's only a name for us humans. You may end up having two derivations with the same name but different hashes: it's the hash that really matters.
What does all this mean? It means that you could run mysql 5.2 with glibc-2.18, and mysql 5.5 with glibc-2.19. You could use your python module with python 2.7 compiled with gcc 4.6 and the same python module with python 3 compiled with gcc 4.8, all in the same system.
In other words: no dependency hell, not even a dependency resolution algorithm. Straight dependencies from derivations to other derivations.
From an administrator\'s point of view: if you want an old PHP version for one application, but want to upgrade the rest of the system, that\'s not painful any more.
From an administrator's point of view: if you want an old PHP version for one application, but want to upgrade the rest of the system, that's not painful any more.
From a developer\'s point of view: if you want to develop webkit with llvm 3.4 and 3.3, that\'s not painful any more.
From a developer's point of view: if you want to develop webkit with llvm 3.4 and 3.3, that's not painful any more.
## Mutable vs. immutable
@ -69,15 +69,15 @@ When upgrading a library, most package managers replace it in-place. All new app
Since Nix derivations are immutable, upgrading a library like glibc means recompiling all applications, because the glibc path to the Nix store has been hardcoded.
So how do we deal with security updates? In Nix we have some tricks (still pure) to solve this problem, but that\'s another story.
So how do we deal with security updates? In Nix we have some tricks (still pure) to solve this problem, but that's another story.
Another problem is that unless software has in mind a pure functional model, or can be adapted to it, it can be hard to compose applications at runtime.
Let\'s take Firefox for example. On most systems, you install flash, and it starts working in Firefox because Firefox looks in a global path for plugins.
Let's take Firefox for example. On most systems, you install flash, and it starts working in Firefox because Firefox looks in a global path for plugins.
In Nix, there\'s no such global path for plugins. Firefox therefore must know explicitly about the path to flash. The way we handle this problem is to wrap the Firefox binary so that we can setup the necessary environment to make it find flash in the nix store. That will produce a new Firefox derivation: be aware that it takes a few seconds, and it makes composition harder at runtime.
In Nix, there's no such global path for plugins. Firefox therefore must know explicitly about the path to flash. The way we handle this problem is to wrap the Firefox binary so that we can setup the necessary environment to make it find flash in the nix store. That will produce a new Firefox derivation: be aware that it takes a few seconds, and it makes composition harder at runtime.
There are no upgrade/downgrade scripts for your data. It doesn\'t make sense with this approach, because there\'s no real derivation to be upgraded. With Nix you switch to using other software with its own stack of dependencies, but there\'s no formal notion of upgrade or downgrade when doing so.
There are no upgrade/downgrade scripts for your data. It doesn't make sense with this approach, because there's no real derivation to be upgraded. With Nix you switch to using other software with its own stack of dependencies, but there's no formal notion of upgrade or downgrade when doing so.
If there is a data format change, then migrating to the new data format remains your own responsibility.
@ -87,9 +87,9 @@ Nix lets you compose software at build time with maximum flexibility, and with b
It however *currently* falls short when working with dynamic composition at runtime or replacing low level libraries, due to the need to rebuild dependencies.
That may sound scary, however after running NixOS on both a server and a laptop desktop, I\'m very satisfied so far. Some of the architectural problems just need some man-power, other design problems still need to be solved as a community.
That may sound scary, however after running NixOS on both a server and a laptop desktop, I'm very satisfied so far. Some of the architectural problems just need some man-power, other design problems still need to be solved as a community.
Considering [Nixpkgs](https://nixos.org/nixpkgs/) ([github link](https://github.com/NixOS/nixpkgs)) is a completely new repository of all the existing software, with a completely fresh concept, and with few core developers but overall year-over-year increasing contributions, the current state is more than acceptable and beyond the experimental stage. In other words, it\'s worth your investment.
Considering [Nixpkgs](https://nixos.org/nixpkgs/) ([github link](https://github.com/NixOS/nixpkgs)) is a completely new repository of all the existing software, with a completely fresh concept, and with few core developers but overall year-over-year increasing contributions, the current state is more than acceptable and beyond the experimental stage. In other words, it's worth your investment.
## Next pill\...

46
pills/02-install-on-your-running.md
View file

@ -2,15 +2,15 @@
Welcome to the second Nix pill. In the [first](01-why-you-should-give-it-try.md) pill we briefly described Nix.
Now we\'ll install Nix on our running system and understand what changed in our system after the installation. **If you\'re using NixOS, Nix is already installed; you can skip to the [next](03-enter-environment.md) pill.**
Now we'll install Nix on our running system and understand what changed in our system after the installation. **If you're using NixOS, Nix is already installed; you can skip to the [next](03-enter-environment.md) pill.**
For installation instructions, please refer to the Nix Reference Manual on [ Installing Nix](https://nixos.org/manual/nix/stable/installation/installing-binary.html).
## Installation
These articles are not a tutorial on *using* Nix. Instead, we\'re going to walk through the Nix system to understand the fundamentals.
These articles are not a tutorial on *using* Nix. Instead, we're going to walk through the Nix system to understand the fundamentals.
The first thing to note: derivations in the Nix store refer to other derivations which are themselves in the Nix store. They don\'t use `libc` from our system or anywhere else. It\'s a self-contained store of all the software we need to bootstrap up to any particular package.
The first thing to note: derivations in the Nix store refer to other derivations which are themselves in the Nix store. They don't use `libc` from our system or anywhere else. It's a self-contained store of all the software we need to bootstrap up to any particular package.
<div class="info">
@ -24,7 +24,7 @@ Start looking at the output of the install command:
copying Nix to /nix/store..........................
That\'s the `/nix/store` we were talking about in the first article. We\'re copying in the necessary software to bootstrap a Nix system. You can see bash, coreutils, the C compiler toolchain, perl libraries, sqlite and Nix itself with its own tools and libnix.
That's the `/nix/store` we were talking about in the first article. We're copying in the necessary software to bootstrap a Nix system. You can see bash, coreutils, the C compiler toolchain, perl libraries, sqlite and Nix itself with its own tools and libnix.
You may have noticed that `/nix/store` can contain not only directories, but also files, still always in the form \<hash-name\>.
@ -34,17 +34,17 @@ Right after copying the store, the installation process initializes a database:
initialising Nix database...
Yes, Nix also has a database. It\'s stored under `/nix/var/nix/db`. It is a sqlite database that keeps track of the dependencies between derivations.
Yes, Nix also has a database. It's stored under `/nix/var/nix/db`. It is a sqlite database that keeps track of the dependencies between derivations.
The schema is very simple: there\'s a table of valid paths, mapping from an auto increment integer to a store path.
The schema is very simple: there's a table of valid paths, mapping from an auto increment integer to a store path.
Then there\'s a dependency relation from path to paths upon which they depend.
Then there's a dependency relation from path to paths upon which they depend.
You can inspect the database by installing sqlite (`nix-env -iA sqlite -f '<nixpkgs>'`) and then running `sqlite3 /nix/var/nix/db/db.sqlite`.
<div class="info">
Note: If this is the first time you\'re using Nix after the initial installation, remember you must close and open your terminals first, so that your shell environment will be updated.
Note: If this is the first time you're using Nix after the initial installation, remember you must close and open your terminals first, so that your shell environment will be updated.
</div>
@ -68,7 +68,7 @@ A profile in Nix is a general and convenient concept for realizing rollbacks. Pr
Generations can be switched and rolled back atomically, which makes them convenient for managing changes to your system.
Let\'s take a closer look at our profile:
Let's take a closer look at our profile:
<pre><code class="hljs">$ ls -l ~/.nix-profile/
bin -> /nix/store/ig31y9gfpp8pf3szdd7d4sf29zr7igbr-<b>nix-2.1.3</b>/bin
@ -82,13 +82,13 @@ That nix-2.1.3 derivation in the Nix store is Nix itself, with binaries and libr
The contents of this profile are special, because only one program has been installed in our profile, therefore e.g. the `bin` directory points to the only program which has been installed (Nix itself).
But that\'s only the contents of the latest generation of our profile. In fact, `~/.nix-profile` itself is a symbolic link to `/nix/var/nix/profiles/default`.
But that's only the contents of the latest generation of our profile. In fact, `~/.nix-profile` itself is a symbolic link to `/nix/var/nix/profiles/default`.
In turn, that\'s a symlink to `default-1-link` in the same directory. Yes, that means it\'s the first generation of the `default` profile.
In turn, that's a symlink to `default-1-link` in the same directory. Yes, that means it's the first generation of the `default` profile.
Finally, `default-1-link` is a symlink to the nix store \"user-environment\" derivation that you saw printed during the installation process.
We\'ll talk about `manifest.nix` more in the next article.
We'll talk about `manifest.nix` more in the next article.
## Nixpkgs expressions
@ -105,19 +105,19 @@ The installer downloaded the package descriptions from commit `a1a2851`.
The second profile we discover is the channels profile. `~/.nix-defexpr/channels` points to `/nix/var/nix/profiles/per-user/nix/channels` which points to `channels-1-link` which points to a Nix store directory containing the downloaded Nix expressions.
Channels are a set of packages and expressions available for download. Similar to Debian stable and unstable, there\'s a stable and unstable channel. In this installation, we\'re tracking `nixpkgs-unstable`.
Channels are a set of packages and expressions available for download. Similar to Debian stable and unstable, there's a stable and unstable channel. In this installation, we're tracking `nixpkgs-unstable`.
Don\'t worry about Nix expressions yet, we\'ll get to them later.
Don't worry about Nix expressions yet, we'll get to them later.
Finally, for your convenience, the installer modified `~/.profile` to automatically enter the Nix environment. What `~/.nix-profile/etc/profile.d/nix.sh` really does is simply to add `~/.nix-profile/bin` to `PATH` and `~/.nix-defexpr/channels/nixpkgs` to `NIX_PATH`. We\'ll discuss `NIX_PATH` later.
Finally, for your convenience, the installer modified `~/.profile` to automatically enter the Nix environment. What `~/.nix-profile/etc/profile.d/nix.sh` really does is simply to add `~/.nix-profile/bin` to `PATH` and `~/.nix-defexpr/channels/nixpkgs` to `NIX_PATH`. We'll discuss `NIX_PATH` later.
Read `nix.sh`, it\'s short.
Read `nix.sh`, it's short.
## FAQ: Can I change /nix to something else?
You can, but there\'s a good reason to keep using `/nix` instead of a different directory. All the derivations depend on other derivations by using absolute paths. We saw in the first article that bash referenced a glibc under a specific absolute path in `/nix/store`.
You can, but there's a good reason to keep using `/nix` instead of a different directory. All the derivations depend on other derivations by using absolute paths. We saw in the first article that bash referenced a glibc under a specific absolute path in `/nix/store`.
You can see for yourself, don\'t worry if you see multiple bash derivations:
You can see for yourself, don't worry if you see multiple bash derivations:
$ ldd /nix/store/*bash*/bin/bash
[...]
@ -128,19 +128,19 @@ Keeping the store in `/nix` means we can grab the binary cache from nixos.org (j
- Thus bash would need to point to glibc under `/foo/store`, instead of under `/nix/store`
- So the binary cache can\'t help, because we need a *different* bash, and so we\'d have to recompile everything ourselves.
- So the binary cache can't help, because we need a *different* bash, and so we'd have to recompile everything ourselves.
After all `/nix` is a sensible place for the store.
## Conclusion
We\'ve installed Nix on our system, fully isolated and owned by the `nix` user as we\'re still coming to terms with this new system.
We've installed Nix on our system, fully isolated and owned by the `nix` user as we're still coming to terms with this new system.
We learned some new concepts like profiles and channels. In particular, with profiles we\'re able to manage multiple generations of a composition of packages, while with channels we\'re able to download binaries from `nixos.org`.
We learned some new concepts like profiles and channels. In particular, with profiles we're able to manage multiple generations of a composition of packages, while with channels we're able to download binaries from `nixos.org`.
The installation put everything under `/nix`, and some symlinks in the Nix user home. That\'s because every user is able to install and use software in her own environment.
The installation put everything under `/nix`, and some symlinks in the Nix user home. That's because every user is able to install and use software in her own environment.
I hope I left nothing uncovered so that you think there\'s some kind of magic going on behind the scenes. It\'s all about putting components in the store and symlinking these components together.
I hope I left nothing uncovered so that you think there's some kind of magic going on behind the scenes. It's all about putting components in the store and symlinking these components together.
## Next pill\...

40
pills/03-enter-environment.md
View file

@ -4,11 +4,11 @@ Welcome to the third Nix pill. In the [second pill](02-install-on-your-running.m
## Enter the environment
**If you\'re using NixOS, you can skip to the [next](#install-something) step.**
**If you're using NixOS, you can skip to the [next](#install-something) step.**
In the previous article we created a Nix user, so let\'s start by switching to it with `su - nix`. If your `~/.profile` got evaluated, then you should now be able to run commands like `nix-env` and `nix-store`.
In the previous article we created a Nix user, so let's start by switching to it with `su - nix`. If your `~/.profile` got evaluated, then you should now be able to run commands like `nix-env` and `nix-store`.
If that\'s not the case:
If that's not the case:
$ source ~/.nix-profile/etc/profile.d/nix.sh
@ -16,7 +16,7 @@ To remind you, `~/.nix-profile/etc` points to the `nix-2.1.3` derivation. At thi
## Install something
Finally something practical! Installation into the Nix environment is an interesting process. Let\'s install `hello`, a simple CLI tool which prints `Hello world` and is mainly used to test compilers and package installations.
Finally something practical! Installation into the Nix environment is an interesting process. Let's install `hello`, a simple CLI tool which prints `Hello world` and is mainly used to test compilers and package installations.
Back to the installation:
@ -30,7 +30,7 @@ Now you can run `hello`. Things to notice:
- We installed software as a user, and only for the Nix user.
- It created a new user environment. That\'s a new generation of our Nix user profile.
- It created a new user environment. That's a new generation of our Nix user profile.
- The [nix-env](https://nixos.org/manual/nix/stable/command-ref/nix-env.html) tool manages environments, profiles and their generations.
@ -48,20 +48,20 @@ Listing installed derivations:
nix-2.1.3
hello-2.10
So, where did `hello` really get installed? `which hello` is `~/.nix-profile/bin/hello` which points to the store. We can also list the derivation paths with `nix-env -q --out-path`. So that\'s what those derivation paths are called: the **output** of a build.
So, where did `hello` really get installed? `which hello` is `~/.nix-profile/bin/hello` which points to the store. We can also list the derivation paths with `nix-env -q --out-path`. So that's what those derivation paths are called: the **output** of a build.
## Path merging
At this point you probably want to run `man` to get some documentation. Even if you already have man system-wide outside of the Nix environment, you can install and use it within Nix with `nix-env -i man-db`. As usual, a new generation will be created, and `~/.nix-profile` will point to it.
Let\'s inspect the [profile](https://nixos.org/manual/nix/stable/package-management/profiles.html) a bit:
Let's inspect the [profile](https://nixos.org/manual/nix/stable/package-management/profiles.html) a bit:
$ ls -l ~/.nix-profile/
dr-xr-xr-x 2 nix nix 4096 Jan 1 1970 bin
lrwxrwxrwx 1 nix nix 55 Jan 1 1970 etc -> /nix/store/ig31y9gfpp8pf3szdd7d4sf29zr7igbr-nix-2.1.3/etc
[...]
Now that\'s interesting. When only `nix-2.1.3` was installed, `bin` was a symlink to `nix-2.1.3`. Now that we\'ve actually installed some things (`man`, `hello`), it\'s a real directory, not a symlink.
Now that's interesting. When only `nix-2.1.3` was installed, `bin` was a symlink to `nix-2.1.3`. Now that we've actually installed some things (`man`, `hello`), it's a real directory, not a symlink.
$ ls -l ~/.nix-profile/bin/
[...]
@ -72,18 +72,18 @@ Now that\'s interesting. When only `nix-2.1.3` was installed, `bin` was a symlin
hello -> /nix/store/58r35bqb4f3lxbnbabq718svq9i2pda3-hello-2.10/bin/hello
[...]
Okay, that\'s clearer now. `nix-env` merged the paths from the installed derivations. `which man` points to the Nix profile, rather than the system `man`, because `~/.nix-profile/bin` is at the head of `$PATH`.
Okay, that's clearer now. `nix-env` merged the paths from the installed derivations. `which man` points to the Nix profile, rather than the system `man`, because `~/.nix-profile/bin` is at the head of `$PATH`.
## Rolling back and switching generation
The last command installed `man`. We should be at generation 3, unless you changed something in the middle. Let\'s say we want to rollback to the old generation:
The last command installed `man`. We should be at generation 3, unless you changed something in the middle. Let's say we want to rollback to the old generation:
$ nix-env --rollback
switching from generation 3 to 2
Now `nix-env -q` does not list `man` anymore. `` ls -l `which man` `` should now be your system copy.
Enough with the rollback, let\'s go back to the most recent generation:
Enough with the rollback, let's go back to the most recent generation:
$ nix-env -G 3
switching from generation 2 to 3
@ -96,7 +96,7 @@ You can of course also [ uninstall](https://nixos.org/manual/nix/stable/command-
So far we learned how to query and manipulate the environment. But all of the environment components point to the store.
To query and manipulate the store, there\'s the `nix-store` command. We can do some interesting things, but we\'ll only see some queries for now.
To query and manipulate the store, there's the `nix-store` command. We can do some interesting things, but we'll only see some queries for now.
To show the direct runtime dependencies of `hello`:
@ -106,7 +106,7 @@ To show the direct runtime dependencies of `hello`:
The argument to `nix-store` can be anything as long as it points to the Nix store. It will follow symlinks.
It may not make sense to you right now, but let\'s print reverse dependencies of `hello`:
It may not make sense to you right now, but let's print reverse dependencies of `hello`:
$ nix-store -q --referrers `which hello`
/nix/store/58r35bqb4f3lxbnbabq718svq9i2pda3-hello-2.10
@ -128,7 +128,7 @@ The closures of a derivation is a list of all its dependencies, recursively, inc
$ nix-store -qR `which man`
[...]
Copying all those derivations to the Nix store of another machine makes you able to run `man` out of the box on that other machine. That\'s the base of deployment using Nix, and you can already foresee the potential when deploying software in the cloud (hint: `nix-copy-closures` and `nix-store --export`).
Copying all those derivations to the Nix store of another machine makes you able to run `man` out of the box on that other machine. That's the base of deployment using Nix, and you can already foresee the potential when deploying software in the cloud (hint: `nix-copy-closures` and `nix-store --export`).
A nicer view of the closure:
@ -141,7 +141,7 @@ The same applies to environments. As an exercise, run `nix-store -q --tree ~/.ni
## Dependency resolution
There isn\'t anything like `apt` which solves a SAT problem in order to satisfy dependencies with lower and upper bounds on versions. There\'s no need for this because all the dependencies are static: if a derivation X depends on a derivation Y, then it always depends on it. A version of X which depended on Z would be a different derivation.
There isn't anything like `apt` which solves a SAT problem in order to satisfy dependencies with lower and upper bounds on versions. There's no need for this because all the dependencies are static: if a derivation X depends on a derivation Y, then it always depends on it. A version of X which depended on Z would be a different derivation.
## Recovering the hard way
@ -150,7 +150,7 @@ There isn\'t anything like `apt` which solves a SAT problem in order to satisfy
uninstalling 'nix-2.1.3'
[...]
Oops, that uninstalled all derivations from the environment, including Nix. That means we can\'t even run `nix-env`, what now?
Oops, that uninstalled all derivations from the environment, including Nix. That means we can't even run `nix-env`, what now?
Previously we got `nix-env` from the environment. Environments are a convenience for the user, but Nix is still there in the store!
@ -166,14 +166,14 @@ The second option is to install Nix, thus creating a new generation:
## Channels
So where are we getting packages from? We said something about this already in the [second article](02-install-on-your-running.md). There\'s a list of channels from which we get packages, although usually we use a single channel. The tool to manage channels is [nix-channel](https://nixos.org/manual/nix/stable/command-ref/nix-channel.html).
So where are we getting packages from? We said something about this already in the [second article](02-install-on-your-running.md). There's a list of channels from which we get packages, although usually we use a single channel. The tool to manage channels is [nix-channel](https://nixos.org/manual/nix/stable/command-ref/nix-channel.html).
$ nix-channel --list
nixpkgs http://nixos.org/channels/nixpkgs-unstable
If you\'re using NixOS, you may not see any output from the above command (if you\'re using the default), or you may see a channel whose name begins with \"nixos-\" instead of \"nixpkgs\".
If you're using NixOS, you may not see any output from the above command (if you're using the default), or you may see a channel whose name begins with \"nixos-\" instead of \"nixpkgs\".
That\'s essentially the contents of `~/.nix-channels`.
That's essentially the contents of `~/.nix-channels`.
<div class="info">
@ -199,4 +199,4 @@ A quick analogy with programming languages: you have the heap with all the objec
## Next pill
\...we will learn the basics of the Nix language. The Nix language is used to describe how to build derivations, and it\'s the basis for everything else, including NixOS. Therefore it\'s very important to understand both the syntax and the semantics of the language.
\...we will learn the basics of the Nix language. The Nix language is used to describe how to build derivations, and it's the basis for everything else, including NixOS. Therefore it's very important to understand both the syntax and the semantics of the language.

38
pills/04-basics-of-language.md
View file

@ -2,9 +2,9 @@
Welcome to the fourth Nix pill. In the [previous article](03-enter-environment.md) we learned about Nix environments. We installed software as a user, managed their profile, switched between generations, and queried the Nix store. Those are the very basics of system administration using Nix.
The [Nix language](https://nixos.org/manual/nix/stable/expressions/expression-language.html) is used to write expressions that produce derivations. The [nix-build](https://nixos.org/manual/nix/stable/command-ref/nix-build.html) tool is used to build derivations from an expression. Even as a system administrator that wants to customize the installation, it\'s necessary to master Nix. Using Nix for your jobs means you get the features we saw in the previous articles for free.
The [Nix language](https://nixos.org/manual/nix/stable/expressions/expression-language.html) is used to write expressions that produce derivations. The [nix-build](https://nixos.org/manual/nix/stable/command-ref/nix-build.html) tool is used to build derivations from an expression. Even as a system administrator that wants to customize the installation, it's necessary to master Nix. Using Nix for your jobs means you get the features we saw in the previous articles for free.
The syntax of Nix is quite unfamiliar, so looking at existing examples may lead you to think that there\'s a lot of magic happening. In reality, it\'s mostly about writing utility functions to make things convenient.
The syntax of Nix is quite unfamiliar, so looking at existing examples may lead you to think that there's a lot of magic happening. In reality, it's mostly about writing utility functions to make things convenient.
On the other hand, the same syntax is great for describing packages, so learning the language itself will pay off when writing package expressions.
@ -22,7 +22,7 @@ Important: Values in Nix are immutable.
## Value types
Nix 2.0 contains a command named `nix repl` which is a simple command line tool for playing with the Nix language. In fact, Nix is a [pure, lazy, functional language](https://nixos.org/manual/nix/stable/expressions/expression-language.html), not only a set of tools to manage derivations. The `nix repl` syntax is slightly different to Nix syntax when it comes to assigning variables, but it shouldn\'t be confusing so long as you bear it in mind. I prefer to start with `nix repl` before cluttering your mind with more complex expressions.
Nix 2.0 contains a command named `nix repl` which is a simple command line tool for playing with the Nix language. In fact, Nix is a [pure, lazy, functional language](https://nixos.org/manual/nix/stable/expressions/expression-language.html), not only a set of tools to manage derivations. The `nix repl` syntax is slightly different to Nix syntax when it comes to assigning variables, but it shouldn't be confusing so long as you bear it in mind. I prefer to start with `nix repl` before cluttering your mind with more complex expressions.
Launch `nix repl`. First of all, Nix supports basic arithmetic operations: `+`, `-`, `*` and `/`. (To exit `nix repl`, use the command `:q`. Help is available through the `:?` command.)
@ -40,7 +40,7 @@ Attempting to perform division in Nix can lead to some surprises.
nix-repl> 6/3
/home/nix/6/3
What happened? Recall that Nix is not a general purpose language, it\'s a domain-specific language for writing packages. Integer division isn\'t actually that useful when writing package expressions. Nix parsed `6/3` as a relative path to the current directory. To get Nix to perform division instead, leave a space after the `/`. Alternatively, you can use `builtins.div`.
What happened? Recall that Nix is not a general purpose language, it's a domain-specific language for writing packages. Integer division isn't actually that useful when writing package expressions. Nix parsed `6/3` as a relative path to the current directory. To get Nix to perform division instead, leave a space after the `/`. Alternatively, you can use `builtins.div`.
nix-repl> 6/ 3
2
@ -52,15 +52,15 @@ Other operators are `||`, `&&` and `!` for booleans, and relational operators su
Nix has integer, floating point, string, path, boolean and null [simple](https://nixos.org/manual/nix/stable/expressions/language-values.html) types. Then there are also lists, sets and functions. These types are enough to build an operating system.
Nix is strongly typed, but it\'s not statically typed. That is, you cannot mix strings and integers, you must first do the conversion.
Nix is strongly typed, but it's not statically typed. That is, you cannot mix strings and integers, you must first do the conversion.
As demonstrated above, expressions will be parsed as paths as long as there\'s a slash not followed by a space. Therefore to specify the current directory, use `./.` In addition, Nix also parses urls specially.
As demonstrated above, expressions will be parsed as paths as long as there's a slash not followed by a space. Therefore to specify the current directory, use `./.` In addition, Nix also parses urls specially.
Not all urls or paths can be parsed this way. If a syntax error occurs, it\'s still possible to fallback to plain strings. Literal urls and paths are convenient for additional safety.
Not all urls or paths can be parsed this way. If a syntax error occurs, it's still possible to fallback to plain strings. Literal urls and paths are convenient for additional safety.
## Identifier
There\'s not much to say here, except that dash (`-`) is allowed in identifiers. That\'s convenient since many packages use dash in their names. In fact:
There's not much to say here, except that dash (`-`) is allowed in identifiers. That's convenient since many packages use dash in their names. In fact:
nix-repl> a-b
error: undefined variable `a-b' at (string):1:1
@ -71,7 +71,7 @@ As you can see, `a-b` is parsed as identifier, not as a subtraction.
## Strings
It\'s important to understand the syntax for strings. When learning to read Nix expressions, you may find dollars (`$`) ambiguous, but they are very important . Strings are enclosed by double quotes (`"`), or two single quotes (`''`).
It's important to understand the syntax for strings. When learning to read Nix expressions, you may find dollars (`$`) ambiguous, but they are very important . Strings are enclosed by double quotes (`"`), or two single quotes (`''`).
nix-repl> "foo"
"foo"
@ -80,7 +80,7 @@ It\'s important to understand the syntax for strings. When learning to read Nix
In other languages like Python you can also use single quotes for strings (e.g. `'foo'`), but not in Nix.
It\'s possible to [interpolate](https://nixos.org/manual/nix/stable/expressions/language-values.html) whole Nix expressions inside strings with the `${...}` syntax and only that syntax, not `$foo` or `{$foo}` or anything else.
It's possible to [interpolate](https://nixos.org/manual/nix/stable/expressions/language-values.html) whole Nix expressions inside strings with the `${...}` syntax and only that syntax, not `$foo` or `{$foo}` or anything else.
nix-repl> foo = "strval"
nix-repl> "$foo"
@ -92,7 +92,7 @@ It\'s possible to [interpolate](https://nixos.org/manual/nix/stable/expressions/
Note: ignore the `foo = "strval"` assignment, special syntax in `nix repl`.
As said previously, you cannot mix integers and strings. You need to explicitly include conversions. We\'ll see this later: function calls are another story.
As said previously, you cannot mix integers and strings. You need to explicitly include conversions. We'll see this later: function calls are another story.
Using the syntax with two single quotes is useful for writing double quotes inside strings without needing to escape them:
@ -101,7 +101,7 @@ Using the syntax with two single quotes is useful for writing double quotes insi
nix-repl> ''${foo}''
"strval"
Escaping `${...}` within double quoted strings is done with the backslash. Within two single quotes, it\'s done with `''`:
Escaping `${...}` within double quoted strings is done with the backslash. Within two single quotes, it's done with `''`:
nix-repl> "\${foo}"
"${foo}"
@ -134,7 +134,7 @@ To access elements in the attribute set:
nix-repl> s."123"
"num"
Yes, you can use strings to address keys which aren\'t valid identifiers.
Yes, you can use strings to address keys which aren't valid identifiers.
Inside an attribute set you cannot normally refer to elements of the same attribute set:
@ -157,7 +157,7 @@ These are expressions, not statements.
nix-repl> if a > b then "yes" else "no"
"no"
You can\'t have only the `then` branch, you must specify also the `else` branch, because an expression must have a value in all cases.
You can't have only the `then` branch, you must specify also the `else` branch, because an expression must have a value in all cases.
## Let expressions
@ -171,7 +171,7 @@ The syntax is: first assign variables, then `in`, then an expression which can u
nix-repl> let a = 3; b = 4; in a + b
7
Let\'s write two `let` expressions, one inside the other:
Let's write two `let` expressions, one inside the other:
nix-repl> let a = 3; in let b = 4; in a + b
7
@ -193,7 +193,7 @@ You can refer to variables in the `let` expression when assigning variables, lik
nix-repl> let a = 4; b = a + 5; in b
9
So beware when you want to refer to a variable from the outer scope, but it\'s also defined in the current let expression. The same applies to recursive attribute sets.
So beware when you want to refer to a variable from the outer scope, but it's also defined in the current let expression. The same applies to recursive attribute sets.
## With expression
@ -205,7 +205,7 @@ This kind of expression is something you rarely see in other languages. You can
nix-repl> with longName; a + b
7
That\'s it, it takes an attribute set and includes symbols from it in the scope of the inner expression. Of course, only valid identifiers from the keys of the set will be included. If a symbol exists in the outer scope and would also be introduced by the `with`, it will *not* be shadowed. You can however still refer to the attribute set:
That's it, it takes an attribute set and includes symbols from it in the scope of the inner expression. Of course, only valid identifiers from the keys of the set will be included. If a symbol exists in the outer scope and would also be introduced by the `with`, it will *not* be shadowed. You can however still refer to the attribute set:
nix-repl> let a = 10; in with longName; a + b
14
@ -219,8 +219,8 @@ Nix evaluates expressions only when needed. This is a great feature when working
nix-repl> let a = builtins.div 4 0; b = 6; in b
6
Since `a` is not needed, there\'s no error about division by zero, because the expression is not in need to be evaluated. That\'s why we can have all the packages defined on demand, yet have access to specific packages very quickly.
Since `a` is not needed, there's no error about division by zero, because the expression is not in need to be evaluated. That's why we can have all the packages defined on demand, yet have access to specific packages very quickly.
## Next pill
\...we will talk about functions and imports. In this pill I\'ve tried to avoid function calls as much as possible, otherwise the post would have been too long.
\...we will talk about functions and imports. In this pill I've tried to avoid function calls as much as possible, otherwise the post would have been too long.

26
pills/05-functions-and-imports.md
View file

@ -2,7 +2,7 @@
Welcome to the fifth Nix pill. In the previous [fourth pill](04-basics-of-language.md) we touched the Nix language for a moment. We introduced basic types and values of the Nix language, and basic expressions such as `if`, `with` and `let`. I invite you to re-read about these expressions and play with them in the repl.
Functions help to build reusable components in a big repository like [nixpkgs](https://github.com/NixOS/nixpkgs/). The Nix manual has a [great explanation of functions](https://nixos.org/manual/nix/stable/expressions/language-constructs.html#functions). Let\'s go: pill on one hand, Nix manual on the other hand.
Functions help to build reusable components in a big repository like [nixpkgs](https://github.com/NixOS/nixpkgs/). The Nix manual has a [great explanation of functions](https://nixos.org/manual/nix/stable/expressions/language-constructs.html#functions). Let's go: pill on one hand, Nix manual on the other hand.
I remind you how to enter the Nix environment: `source ~/.nix-profile/etc/profile.d/nix.sh`
@ -13,7 +13,7 @@ Functions are anonymous (lambdas), and only have a single parameter. The syntax
nix-repl> x: x*2
«lambda»
So here we defined a function that takes a parameter `x`, and returns `x*2`. The problem is that we cannot use it in any way, because it\'s unnamed\... joke!
So here we defined a function that takes a parameter `x`, and returns `x*2`. The problem is that we cannot use it in any way, because it's unnamed\... joke!
We can store functions in variables.
@ -25,13 +25,13 @@ We can store functions in variables.
As usual, please ignore the special syntax for assignments inside `nix repl`. So, we defined a function `x: x*2` that takes one parameter `x`, and returns `x*2`. This function is then assigned to the variable `double`. Finally we did our first function call: `double 3`.
Big note: it\'s not like many other programming languages where you write `double(3)`. It really is `double 3`.
Big note: it's not like many other programming languages where you write `double(3)`. It really is `double 3`.
In summary: to call a function, name the variable, then space, then the argument. Nothing else to say, it\'s as easy as that.
In summary: to call a function, name the variable, then space, then the argument. Nothing else to say, it's as easy as that.
## More than one parameter
How do we create a function that accepts more than one parameter? For people not used to functional programming, this may take a while to grasp. Let\'s do it step by step.
How do we create a function that accepts more than one parameter? For people not used to functional programming, this may take a while to grasp. Let's do it step by step.
nix-repl> mul = a: (b: a*b)
nix-repl> mul
@ -43,7 +43,7 @@ How do we create a function that accepts more than one parameter? For people not
We defined a function that takes the parameter `a`, the body returns another function. This other function takes a parameter `b` and returns `a*b`. Therefore, calling `mul 3` returns this kind of function: `b: 3*b`. In turn, we call the returned function with `4`, and get the expected result.
You don\'t have to use parentheses at all, Nix has sane priorities when parsing the code:
You don't have to use parentheses at all, Nix has sane priorities when parsing the code:
nix-repl> mul = a: b: a*b
nix-repl> mul
@ -55,7 +55,7 @@ You don\'t have to use parentheses at all, Nix has sane priorities when parsing
nix-repl> mul (6+7) (8+9)
221
Much more readable, you don\'t even notice that functions only receive one argument. Since the argument is separated by a space, to pass more complex expressions you need parentheses. In other common languages you would write `mul(6+7, 8+9)`.
Much more readable, you don't even notice that functions only receive one argument. Since the argument is separated by a space, to pass more complex expressions you need parentheses. In other common languages you would write `mul(6+7, 8+9)`.
Given that functions have only one parameter, it is straightforward to use **partial application**:
@ -80,7 +80,7 @@ Now this is a very cool feature of Nix. It is possible to pattern match over a s
In the first case we defined a function that accepts a single parameter. We then access attributes `a` and `b` from the given set. Note how the parentheses-less syntax for function calls is very elegant in this case, instead of doing `mul({ a=3; b=4; })` in other languages.
In the second case we defined an argument set. It\'s like defining a set, except without values. We require that the passed set contains the keys `a` and `b`. Then we can use those `a` and `b` in the function body directly.
In the second case we defined an argument set. It's like defining a set, except without values. We require that the passed set contains the keys `a` and `b`. Then we can use those `a` and `b` in the function body directly.
nix-repl> mul = { a, b }: a*b
nix-repl> mul { a = 3; b = 4; c = 6; }
@ -111,11 +111,11 @@ However, in the function body you cannot access the \"c\" attribute. The solutio
nix-repl> mul { a = 3; b = 4; c = 2; }
24
That\'s it, you give a name to the whole parameter with name@ before the set pattern.
That's it, you give a name to the whole parameter with name@ before the set pattern.
Advantages of using argument sets:
- Named unordered arguments: you don\'t have to remember the order of the arguments.
- Named unordered arguments: you don't have to remember the order of the arguments.
- You can pass sets, that adds a whole new layer of flexibility and convenience.
@ -129,7 +129,7 @@ You may find similarities with [Python \*\*kwargs](https://docs.python.org/3/faq
The `import` function is built-in and provides a way to parse a `.nix` file. The natural approach is to define each component in a `.nix` file, then compose by importing these files.
Let\'s start with the bare metal.
Let's start with the bare metal.
`a.nix`:
@ -149,7 +149,7 @@ Let\'s start with the bare metal.
nix-repl> mul a b
12
Yes it\'s really that simple. You import a file, and it gets parsed as an expression. Note that the scope of the imported file does not inherit the scope of the importer.
Yes it's really that simple. You import a file, and it gets parsed as an expression. Note that the scope of the imported file does not inherit the scope of the importer.
`test.nix`:
@ -175,7 +175,7 @@ Explaining:
- In `test.nix` we return a function. It accepts a set, with default attributes `b`, `trueMsg` and `falseMsg`.
- `builtins.trace` is a [built-in function](https://nixos.org/manual/nix/stable/expressions/builtins.html) that takes two arguments. The first is the message to display, the second is the value to return. It\'s usually used for debugging purposes.
- `builtins.trace` is a [built-in function](https://nixos.org/manual/nix/stable/expressions/builtins.html) that takes two arguments. The first is the message to display, the second is the value to return. It's usually used for debugging purposes.
- Then we import `test.nix`, and call the function with that set.

64
pills/06-our-first-derivation.md
View file

@ -10,44 +10,44 @@ I remind you how to enter the Nix environment: `source ~/.nix-profile/etc/profil
The [derivation built-in function](https://nixos.org/manual/nix/stable/expressions/derivations.html) is used to create derivations. I invite you to read the link in the Nix manual about the derivation built-in. A derivation from a Nix language view point is simply a set, with some attributes. Therefore you can pass the derivation around with variables like anything else.
That\'s where the real power comes in.
That's where the real power comes in.
The `derivation` function receives a set as its first argument. This set requires at least the following three attributes:
- name: the name of the derivation. In the nix store the format is hash-name, that\'s the name.
- name: the name of the derivation. In the nix store the format is hash-name, that's the name.
- system: is the name of the system in which the derivation can be built. For example, x86_64-linux.
- builder: is the binary program that builds the derivation.
First of all, what\'s the name of our system as seen by nix?
First of all, what's the name of our system as seen by nix?
nix-repl> builtins.currentSystem
"x86_64-linux"
Let\'s try to fake the name of the system:
Let's try to fake the name of the system:
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; }
nix-repl> d
«derivation /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv»
Oh oh, what\'s that? Did it build the derivation? No it didn\'t, but it **did create the .drv file**. `nix repl` does not build derivations unless you tell it to do so.
Oh oh, what's that? Did it build the derivation? No it didn't, but it **did create the .drv file**. `nix repl` does not build derivations unless you tell it to do so.
## Digression about .drv files
What\'s that `.drv` file? It is the specification of how to build the derivation, without all the Nix language fuzz.
What's that `.drv` file? It is the specification of how to build the derivation, without all the Nix language fuzz.
Before continuing, some analogies with the C language:
- `.nix` files are like `.c` files.
- `.drv` files are intermediate files like `.o` files. The `.drv` describes how to build a derivation; it\'s the bare minimum information.
- `.drv` files are intermediate files like `.o` files. The `.drv` describes how to build a derivation; it's the bare minimum information.
- out paths are then the product of the build.
Both drv paths and out paths are stored in the nix store as you can see.
What\'s in that `.drv` file? You can read it, but it\'s better to pretty print it:
What's in that `.drv` file? You can read it, but it's better to pretty print it:
<div class="info">
@ -79,29 +79,29 @@ $ nix derivation show /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv
}
```
Ok, we can see there\'s an out path, but it does not exist yet. We never told Nix to build it, but we know beforehand where the build output will be. Why?
Ok, we can see there's an out path, but it does not exist yet. We never told Nix to build it, but we know beforehand where the build output will be. Why?
Think, if Nix ever built the derivation just because we accessed it in Nix, we would have to wait a long time if it was, say, Firefox. That\'s why Nix let us know the path beforehand and kept evaluating the Nix expressions, but it\'s still empty because no build was ever made.
Think, if Nix ever built the derivation just because we accessed it in Nix, we would have to wait a long time if it was, say, Firefox. That's why Nix let us know the path beforehand and kept evaluating the Nix expressions, but it's still empty because no build was ever made.
Important: the hash of the out path is based solely on the input derivations in the current version of Nix, not on the contents of the build product. It\'s possible however to have [content-addressable](https://en.wikipedia.org/wiki/Content-addressable_storage) derivations for e.g. tarballs as we\'ll see later on.
Important: the hash of the out path is based solely on the input derivations in the current version of Nix, not on the contents of the build product. It's possible however to have [content-addressable](https://en.wikipedia.org/wiki/Content-addressable_storage) derivations for e.g. tarballs as we'll see later on.
Many things are empty in that `.drv`, however I\'ll write a summary of the [.drv format](http://nixos.org/~eelco/pubs/phd-thesis.pdf) for you:
Many things are empty in that `.drv`, however I'll write a summary of the [.drv format](http://nixos.org/~eelco/pubs/phd-thesis.pdf) for you:
1. The output paths (there can be multiple ones). By default nix creates one out path called \"out\".
2. The list of input derivations. It\'s empty because we are not referring to any other derivation. Otherwise, there would be a list of other .drv files.
2. The list of input derivations. It's empty because we are not referring to any other derivation. Otherwise, there would be a list of other .drv files.
3. The system and the builder executable (yes, it\'s a fake one).
3. The system and the builder executable (yes, it's a fake one).
4. Then a list of environment variables passed to the builder.
That\'s it, the minimum necessary information to build our derivation.
That's it, the minimum necessary information to build our derivation.
Important note: the environment variables passed to the builder are just those you see in the .drv plus some other Nix related configuration (number of cores, temp dir, \...). The builder will not inherit any variable from your running shell, otherwise builds would suffer from [non-determinism](https://wiki.debian.org/ReproducibleBuilds).
Back to our fake derivation.
Let\'s build our really fake derivation:
Let's build our really fake derivation:
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = "mysystem"; }
nix-repl> :b d
@ -111,15 +111,15 @@ Let\'s build our really fake derivation:
building path(s) `/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname'
error: a `mysystem' is required to build `/nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv', but I am a `x86_64-linux'
The `:b` is a `nix repl` specific command to build a derivation. You can see more commands with `:?` . So in the output you can see that it takes the `.drv` as information on how to build the derivation. Then it says it\'s trying to produce our out path. Finally the error we were waiting for: that derivation can\'t be built on our system.
The `:b` is a `nix repl` specific command to build a derivation. You can see more commands with `:?` . So in the output you can see that it takes the `.drv` as information on how to build the derivation. Then it says it's trying to produce our out path. Finally the error we were waiting for: that derivation can't be built on our system.
We\'re doing the build inside `nix repl`, but what if we don\'t want to use `nix repl`? You can **realise** a `.drv` with:
We're doing the build inside `nix repl`, but what if we don't want to use `nix repl`? You can **realise** a `.drv` with:
$ nix-store -r /nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv
You will get the same output as before.
Let\'s fix the system attribute:
Let's fix the system attribute:
nix-repl> d = derivation { name = "myname"; builder = "mybuilder"; system = builtins.currentSystem; }
nix-repl> :b d
@ -128,7 +128,7 @@ Let\'s fix the system attribute:
A step forward: of course, that `mybuilder` executable does not really exist. Stop for a moment.
## What\'s in a derivation set
## What's in a derivation set
It is useful to start by inspecting the return value from the derivation function. In this case, the returned value is a plain set:
@ -145,21 +145,21 @@ Start from drvAttrs:
nix-repl> d.drvAttrs
{ builder = "mybuilder"; name = "myname"; system = "mysystem"; }
That\'s basically the input we gave to the derivation function. Also the `d.name`, `d.system` and `d.builder` attributes are exactly the ones we gave as input.
That's basically the input we gave to the derivation function. Also the `d.name`, `d.system` and `d.builder` attributes are exactly the ones we gave as input.
nix-repl> (d == d.out)
true
So out is just the derivation itself, it seems weird but the reason is that we only have one output from the derivation. That\'s also the reason why `d.all` is a singleton. We\'ll see multiple outputs later.
So out is just the derivation itself, it seems weird but the reason is that we only have one output from the derivation. That's also the reason why `d.all` is a singleton. We'll see multiple outputs later.
The `d.drvPath` is the path of the `.drv` file: `/nix/store/z3hhlxbckx4g3n9sw91nnvlkjvyw754p-myname.drv`.
Something interesting is the `type` attribute. It\'s `"derivation"`. Nix does add a little of magic to sets with type derivation, but not that much. To help you understand, you can create yourself a set with that type, it\'s a simple set:
Something interesting is the `type` attribute. It's `"derivation"`. Nix does add a little of magic to sets with type derivation, but not that much. To help you understand, you can create yourself a set with that type, it's a simple set:
nix-repl> { type = "derivation"; }
«derivation ???»
Of course it has no other information, so Nix doesn\'t know what to say :-) But you get it, the `type = "derivation"` is just a convention for Nix and for us to understand the set is a derivation.
Of course it has no other information, so Nix doesn't know what to say :-) But you get it, the `type = "derivation"` is just a convention for Nix and for us to understand the set is a derivation.
When writing packages, we are interested in the outputs. The other metadata is needed for Nix to know how to create the drv path and the out path.
@ -195,14 +195,14 @@ Apart from the nixpkgs stuff, just think we added to the scope a series of varia
$ ls /nix/store/*coreutils*/bin
[...]
I remind you, inside strings it\'s possible to interpolate Nix expressions with `${...}`:
I remind you, inside strings it's possible to interpolate Nix expressions with `${...}`:
nix-repl> "${d}"
"/nix/store/40s0qmrfb45vlh6610rk29ym318dswdr-myname"
nix-repl> "${coreutils}"
"/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21"
That\'s very convenient, because then we could refer to e.g. the bin/true binary like this:
That's very convenient, because then we could refer to e.g. the bin/true binary like this:
nix-repl> "${coreutils}/bin/true"
"/nix/store/8w4cbiy7wqvaqsnsnb3zvabq1cp2zhyz-coreutils-8.21/bin/true"
@ -221,7 +221,7 @@ Another step forward, it executed the builder (bin/true), but the builder did no
Obvious note: every time we change the derivation, a new hash is created.
Let\'s examine the new `.drv` now that we referred to another derivation:
Let's examine the new `.drv` now that we referred to another derivation:
```
$ nix derivation show /nix/store/qyfrcd53wmc0v22ymhhd5r6sz5xmdc8a-myname.drv
@ -251,11 +251,11 @@ $ nix derivation show /nix/store/qyfrcd53wmc0v22ymhhd5r6sz5xmdc8a-myname.drv
}
```
Aha! Nix added a dependency to our myname.drv, it\'s the coreutils.drv. Before doing our build, Nix should build the coreutils.drv. But since coreutils is already in our nix store, no build is needed, it\'s already there with out path `/nix/store/qrxs7sabhqcr3j9ai0j0cp58zfnny0jz-coreutils-8.29`.
Aha! Nix added a dependency to our myname.drv, it's the coreutils.drv. Before doing our build, Nix should build the coreutils.drv. But since coreutils is already in our nix store, no build is needed, it's already there with out path `/nix/store/qrxs7sabhqcr3j9ai0j0cp58zfnny0jz-coreutils-8.29`.
## When is the derivation built
Nix does not build derivations **during evaluation** of Nix expressions. In fact, that\'s why we have to do \":b drv\" in `nix repl`, or use nix-store -r in the first place.
Nix does not build derivations **during evaluation** of Nix expressions. In fact, that's why we have to do \":b drv\" in `nix repl`, or use nix-store -r in the first place.
An important separation is made in Nix:
@ -269,11 +269,11 @@ In Nix, first the Nix expression (usually in a .nix file) is compiled to .drv, t
## Conclusion
Is it that complicated to create a package for Nix? No, it\'s not.
Is it that complicated to create a package for Nix? No, it's not.
We\'re walking through the fundamentals of Nix derivations, to understand how they work, how they are represented. Packaging in Nix is certainly easier than that, but we\'re not there yet in this post. More Nix pills are needed.
We're walking through the fundamentals of Nix derivations, to understand how they work, how they are represented. Packaging in Nix is certainly easier than that, but we're not there yet in this post. More Nix pills are needed.
With the derivation function we provide a set of information on how to build a package, and we get back the information about where the package was built. Nix converts a set to a string when there\'s an `outPath`; that\'s very convenient. With that, it\'s easy to refer to other derivations.
With the derivation function we provide a set of information on how to build a package, and we get back the information about where the package was built. Nix converts a set to a string when there's an `outPath`; that's very convenient. With that, it's easy to refer to other derivations.
When Nix builds a derivation, it first creates a .drv file from a derivation expression, and uses it to build the output. It does so recursively for all the dependencies (inputs). It \"executes\" the .drv files like a machine. Not much magic after all.

38
pills/07-working-derivation.md
View file

@ -10,15 +10,15 @@ I remind you how to enter the Nix environment: `source ~/.nix-profile/etc/profil
## Using a script as a builder
What\'s the easiest way to run a sequence of commands for building something? A bash script. We write a custom bash script, and we want it to be our builder. Given a `builder.sh`, we want the derivation to run `bash builder.sh`.
What's the easiest way to run a sequence of commands for building something? A bash script. We write a custom bash script, and we want it to be our builder. Given a `builder.sh`, we want the derivation to run `bash builder.sh`.
We don\'t use hash bangs in `builder.sh`, because at the time we are writing it we do not know the path to bash in the nix store. Yes, even bash is in the nix store, everything is there.
We don't use hash bangs in `builder.sh`, because at the time we are writing it we do not know the path to bash in the nix store. Yes, even bash is in the nix store, everything is there.
We don\'t even use /usr/bin/env, because then we lose the cool stateless property of Nix. Not to mention that `PATH` gets cleared when building, so it wouldn\'t find bash anyway.
We don't even use /usr/bin/env, because then we lose the cool stateless property of Nix. Not to mention that `PATH` gets cleared when building, so it wouldn't find bash anyway.
In summary, we want the builder to be bash, and pass it an argument, `builder.sh`. Turns out the `derivation` function accepts an optional `args` attribute which is used to pass arguments to the builder executable.
First of all, let\'s write our `builder.sh` in the current directory:
First of all, let's write our `builder.sh` in the current directory:
declare -xp
echo foo > $out
@ -27,7 +27,7 @@ The command `declare -xp` lists exported variables (`declare` is a builtin bash
What we have to do is create something in the path `$out`, be it a file or a directory. In this case we are creating a file.
In addition, we print out the environment variables during the build process. We cannot use env for this, because env is part of coreutils and we don\'t have a dependency to it yet. We only have bash for now.
In addition, we print out the environment variables during the build process. We cannot use env for this, because env is part of coreutils and we don't have a dependency to it yet. We only have bash for now.
Like for coreutils in the previous pill, we get a blessed bash for free from our magic nixpkgs stuff:
@ -49,7 +49,7 @@ this derivation produced the following outputs:
out -> /nix/store/gczb4qrag22harvv693wwnflqy7lx5pb-foo
```
We did it! The contents of `/nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-foo` is really foo. We\'ve built our first derivation.
We did it! The contents of `/nix/store/w024zci0x1hh1wj6gjq0jagkc1sgrf5r-foo` is really foo. We've built our first derivation.
Note that we used `./builder.sh` and not `"./builder.sh"`. This way, it is parsed as a path, and Nix performs some magic which we will cover later. Try using the string version and you will find that it cannot find `builder.sh`. This is because it tries to find it relative to the temporary build directory.
@ -78,9 +78,9 @@ declare -x out="/nix/store/gczb4qrag22harvv693wwnflqy7lx5pb-foo"
declare -x system="x86_64-linux"
```
Let\'s inspect those environment variables printed during the build process.
Let's inspect those environment variables printed during the build process.
- `$HOME` is not your home directory, and `/homeless-shelter` doesn\'t exist at all. We force packages not to depend on `$HOME` during the build process.
- `$HOME` is not your home directory, and `/homeless-shelter` doesn't exist at all. We force packages not to depend on `$HOME` during the build process.
- `$PATH` plays the same game as `$HOME`
@ -88,15 +88,15 @@ Let\'s inspect those environment variables printed during the build process.
- `$PWD` and `$TMP` clearly show that nix created a temporary build directory
- Then `$builder`, `$name`, `$out`, and `$system` are variables set due to the .drv file\'s contents.
- Then `$builder`, `$name`, `$out`, and `$system` are variables set due to the .drv file's contents.
And that\'s how we were able to use `$out` in our derivation and put stuff in it. It\'s like Nix reserved a slot in the nix store for us, and we must fill it.
And that's how we were able to use `$out` in our derivation and put stuff in it. It's like Nix reserved a slot in the nix store for us, and we must fill it.
In terms of autotools, `$out` will be the `--prefix` path. Yes, not the make `DESTDIR`, but the `--prefix`. That\'s the essence of stateless packaging. You don\'t install the package in a global common path under `/`, you install it in a local isolated path under your nix store slot.
In terms of autotools, `$out` will be the `--prefix` path. Yes, not the make `DESTDIR`, but the `--prefix`. That's the essence of stateless packaging. You don't install the package in a global common path under `/`, you install it in a local isolated path under your nix store slot.
## The .drv contents
We added something else to the derivation this time: the args attribute. Let\'s see how this changed the .drv compared to the previous pill:
We added something else to the derivation this time: the args attribute. Let's see how this changed the .drv compared to the previous pill:
```
$ nix derivation show /nix/store/i76pr1cz0za3i9r6xq518bqqvd2raspw-foo.drv
@ -130,7 +130,7 @@ $ nix derivation show /nix/store/i76pr1cz0za3i9r6xq518bqqvd2raspw-foo.drv
}
```
Much like the usual .drv, except that there\'s a list of arguments in there passed to the builder (bash) with `builder.sh`... In the nix store..? Nix automatically copies files or directories needed for the build into the store to ensure that they are not changed during the build process and that the deployment is stateless and independent of the building machine. `builder.sh` is not only in the arguments passed to the builder, it\'s also in the input derivations.
Much like the usual .drv, except that there's a list of arguments in there passed to the builder (bash) with `builder.sh`... In the nix store..? Nix automatically copies files or directories needed for the build into the store to ensure that they are not changed during the build process and that the deployment is stateless and independent of the building machine. `builder.sh` is not only in the arguments passed to the builder, it's also in the input derivations.
Given that `builder.sh` is a plain file, it has no .drv associated with it. The store path is computed based on the filename and on the hash of its contents. Store paths are covered in detail in [a later pill](18-nix-store-paths.md).
@ -148,7 +148,7 @@ And its `simple_builder.sh`:
mkdir $out
gcc -o $out/simple $src
Don\'t worry too much about where those variables come from yet; let\'s write the derivation and build it:
Don't worry too much about where those variables come from yet; let's write the derivation and build it:
```
nix-repl> :l <nixpkgs>
@ -165,11 +165,11 @@ Now you can run `/nix/store/ni66p4jfqksbmsl616llx3fbs1d232d4-simple/simple` in y
We added two new attributes to the derivation call, `gcc` and `coreutils`. In `gcc = gcc;`, the name on the left is the name in the derivation set, and the name on the right refers to the gcc derivation from nixpkgs. The same applies for coreutils.
We also added the `src` attribute, nothing magical --- it\'s just a name, to which the path `./simple.c` is assigned. Like `simple-builder.sh`, `simple.c` will be added to the store.
We also added the `src` attribute, nothing magical --- it's just a name, to which the path `./simple.c` is assigned. Like `simple-builder.sh`, `simple.c` will be added to the store.
The trick: every attribute in the set passed to `derivation` will be converted to a string and passed to the builder as an environment variable. This is how the builder gains access to coreutils and gcc: when converted to strings, the derivations evaluate to their output paths, and appending `/bin` to these leads us to their binaries.
The same goes for the `src` variable. `$src` is the path to `simple.c` in the nix store. As an exercise, pretty print the .drv file. You\'ll see `simple_builder.sh` and `simple.c` listed in the input derivations, along with bash, gcc and coreutils .drv files. The newly added environment variables described above will also appear.
The same goes for the `src` variable. `$src` is the path to `simple.c` in the nix store. As an exercise, pretty print the .drv file. You'll see `simple_builder.sh` and `simple.c` listed in the input derivations, along with bash, gcc and coreutils .drv files. The newly added environment variables described above will also appear.
In `simple_builder.sh` we set the `PATH` for gcc and coreutils binaries, so that our build script can find the necessary utilities like mkdir and gcc.
@ -206,7 +206,7 @@ In the second line of `simple.nix`, we have an `import` function call. Recall th
Afterwards, we call the function with the empty set. We saw this already in [the fifth pill](05-functions-and-imports.md). To reiterate: `import <nixpkgs> {}` is calling two functions, not one. Reading it as `(import <nixpkgs>) {}` makes this clearer.
The value returned by the nixpkgs function is a set; more specifically, it\'s a set of derivations. Calling `import <nixpkgs> {}` into a `let`-expression creates the local variable `pkgs` and brings it into scope. This has an effect similar to the `:l <nixpkgs>` we used in nix repl, in that it allows us to easily access derivations such as `bash`, `gcc`, and `coreutils`, but those derivations will have to be explicitly referred to as members of the `pkgs` set (e.g., `pkgs.bash` instead of just `bash`).
The value returned by the nixpkgs function is a set; more specifically, it's a set of derivations. Calling `import <nixpkgs> {}` into a `let`-expression creates the local variable `pkgs` and brings it into scope. This has an effect similar to the `:l <nixpkgs>` we used in nix repl, in that it allows us to easily access derivations such as `bash`, `gcc`, and `coreutils`, but those derivations will have to be explicitly referred to as members of the `pkgs` set (e.g., `pkgs.bash` instead of just `bash`).
Below is a revised version of the `simple.nix` file, using the `inherit` keyword:
@ -224,10 +224,10 @@ Below is a revised version of the `simple.nix` file, using the `inherit` keyword
Here we also take the opportunity to introduce the [`inherit` keyword](https://nixos.org/manual/nix/stable/expressions/language-constructs.html#inheriting-attributes). `inherit foo;` is equivalent to `foo = foo;`. Similarly, `inherit gcc coreutils;` is equivalent to `gcc = gcc; coreutils = coreutils;`. Lastly, `inherit (pkgs) gcc coreutils;` is equivalent to `gcc = pkgs.gcc; coreutils = pkgs.coreutils;`.
This syntax only makes sense inside sets. There\'s no magic involved, it\'s simply a convenience to avoid repeating the same name for both the attribute name and the value in scope.
This syntax only makes sense inside sets. There's no magic involved, it's simply a convenience to avoid repeating the same name for both the attribute name and the value in scope.
## Next pill
We will generalize the builder. You may have noticed that we wrote two separate `builder.sh` scripts in this post. We would like to have a generic builder script instead, especially since each build script goes in the nix store: a bit of a waste.
*Is it really that hard to package stuff in Nix? No*, here we\'re studying the fundamentals of Nix.
*Is it really that hard to package stuff in Nix? No*, here we're studying the fundamentals of Nix.

28
pills/08-generic-builders.md
View file

@ -6,11 +6,11 @@ In this post, we will generalize the builder script, write a Nix expression for
## Packaging GNU hello world
In the previous pill we packaged a simple .c file, which was being compiled with a raw gcc call. That\'s not a good example of a project. Many use autotools, and since we\'re going to generalize our builder, it would be better to do it with the most used build system.
In the previous pill we packaged a simple .c file, which was being compiled with a raw gcc call. That's not a good example of a project. Many use autotools, and since we're going to generalize our builder, it would be better to do it with the most used build system.
[GNU hello world](https://www.gnu.org/software/hello/), despite its name, is a simple yet complete project which uses autotools. Fetch the latest tarball here: <https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz>.
Let\'s create a builder script for GNU hello world, hello_builder.sh:
Let's create a builder script for GNU hello world, hello_builder.sh:
export PATH="$gnutar/bin:$gcc/bin:$gnumake/bin:$coreutils/bin:$gawk/bin:$gzip/bin:$gnugrep/bin:$gnused/bin:$bintools/bin"
tar -xzf $src
@ -79,7 +79,7 @@ Please note the `--prefix=$out` we were talking about in the [previous pill](07-
## A generic builder
Let\'s create a generic `builder.sh` for autotools projects:
Let's create a generic `builder.sh` for autotools projects:
set -e
unset PATH
@ -104,19 +104,19 @@ What do we do here?
1. Exit the build on any error with `set -e`.
2. First `unset PATH`, because it\'s initially set to a non-existent path.
2. First `unset PATH`, because it's initially set to a non-existent path.
3. We\'ll see this below in detail, however for each path in `$buildInputs`, we append `bin` to `PATH`.
3. We'll see this below in detail, however for each path in `$buildInputs`, we append `bin` to `PATH`.
4. Unpack the source.
5. Find a directory where the source has been unpacked and `cd` into it.
6. Once we\'re set up, compile and install.
6. Once we're set up, compile and install.
As you can see, there\'s no reference to \"hello\" in the builder anymore. It still makes several assumptions, but it\'s certainly more generic.
As you can see, there's no reference to \"hello\" in the builder anymore. It still makes several assumptions, but it's certainly more generic.
Now let\'s rewrite `hello.nix`:
Now let's rewrite `hello.nix`:
let
pkgs = import <nixpkgs> { };
@ -140,7 +140,7 @@ Now let\'s rewrite `hello.nix`:
system = builtins.currentSystem;
}
All clear, except that buildInputs. However it\'s easier than any black magic you are thinking of at this moment.
All clear, except that buildInputs. However it's easier than any black magic you are thinking of at this moment.
Nix is able to convert a list to a string. It first converts the elements to strings, and then concatenates them separated by a space:
@ -162,7 +162,7 @@ Simple! The buildInputs variable is a string with out paths separated by space,
## A more convenient derivation function
We managed to write a builder that can be used for multiple autotools projects. But in the hello.nix expression we are specifying tools that are common to more projects; we don\'t want to pass them every time.
We managed to write a builder that can be used for multiple autotools projects. But in the hello.nix expression we are specifying tools that are common to more projects; we don't want to pass them every time.
A natural approach would be to create a function that accepts an attribute set, similar to the one used by the derivation function, and merge it with another attribute set containing values common to many projects.
@ -224,15 +224,15 @@ Then we rewrite `hello.nix` as follows:
src = ./hello-2.12.1.tar.gz;
}
Finally! We got a very simple description of a package! Below are a couple of remarks that you may find useful as you\'re continuing to understand the nix language:
Finally! We got a very simple description of a package! Below are a couple of remarks that you may find useful as you're continuing to understand the nix language:
- We assigned to pkgs the import that we did in the previous expressions in the \"with\". Don\'t be afraid, it\'s that straightforward.
- We assigned to pkgs the import that we did in the previous expressions in the \"with\". Don't be afraid, it's that straightforward.
- The mkDerivation variable is a nice example of partial application, look at it as (`import ./autotools.nix`) `pkgs`. First we import the expression, then we apply the `pkgs` parameter. That will give us a function that accepts the attribute set `attrs`.
- We create the derivation specifying only name and src. If the project eventually needed other dependencies to be in PATH, then we would simply add those to buildInputs (not specified in hello.nix because empty).
Note we didn\'t use any other library. Special C flags may be needed to find include files of other libraries at compile time, and ld flags at link time.
Note we didn't use any other library. Special C flags may be needed to find include files of other libraries at compile time, and ld flags at link time.
## Conclusion
@ -240,7 +240,7 @@ Nix gives us the bare metal tools for creating derivations, setting up a build e
Out of this pill we managed to create a generic builder for autotools projects, and a function `mkDerivation` that composes by default the common components used in autotools projects instead of repeating them in all the packages we would write.
We are familiarizing ourselves with the way a Nix system grows up: it\'s about creating and composing derivations with the Nix language.
We are familiarizing ourselves with the way a Nix system grows up: it's about creating and composing derivations with the Nix language.
Analogy: in C you create objects in the heap, and then you compose them inside new objects. Pointers are used to refer to other objects.

14
pills/09-automatic-runtime.md
View file

@ -6,7 +6,7 @@ Today we stop by the GNU `hello` program to analyze build and runtime dependenci
## Build dependencies
Let\'s start analyzing build dependencies for our GNU `hello` package:
Let's start analyzing build dependencies for our GNU `hello` package:
$ nix-instantiate hello.nix
/nix/store/z77vn965a59irqnrrjvbspiyl2rph0jp-hello.drv
@ -34,7 +34,7 @@ The `NAR` format is the \"Nix ARchive\". This format was designed due to existin
Thus the `NAR` format was developed as a simple, deterministic archive format. `NAR`s are used extensively within Nix, as we will see below.
For more rationale and implementation details behind `NAR` see [Dolstra\'s PhD Thesis](http://nixos.org/~eelco/pubs/phd-thesis.pdf).
For more rationale and implementation details behind `NAR` see [Dolstra's PhD Thesis](http://nixos.org/~eelco/pubs/phd-thesis.pdf).
To create NAR archives from store paths, we can use `nix-store --dump` and `nix-store --restore`.
@ -48,7 +48,7 @@ Nix handles runtime dependencies for us automatically. The technique it uses to
2. For each build dependency `.drv` and its relative out path, search the contents of the NAR for this out path.
3. If the path is found, then it\'s a runtime dependency.
3. If the path is found, then it's a runtime dependency.
The snippet below shows the dependencies for `hello`.
@ -61,16 +61,16 @@ The snippet below shows the dependencies for `hello`.
/nix/store/8jm0wksask7cpf85miyakihyfch1y21q-gcc-4.8.3
/nix/store/a42k52zwv6idmf50r9lps1nzwq9khvpf-hello
We see that `glibc` and `gcc` are runtime dependencies. Intuitively, `gcc` shouldn\'t be in this list! Displaying the printable strings in the `hello` binary shows that the out path of `gcc` does indeed appear:
We see that `glibc` and `gcc` are runtime dependencies. Intuitively, `gcc` shouldn't be in this list! Displaying the printable strings in the `hello` binary shows that the out path of `gcc` does indeed appear:
$ strings result/bin/hello|grep gcc
/nix/store/94n64qy99ja0vgbkf675nyk39g9b978n-glibc-2.19/lib:/nix/store/8jm0wksask7cpf85miyakihyfch1y21q-gcc-4.8.3/lib64
This is why Nix added `gcc`. But why is that path present in the first place? The answer is that it is the [ld rpath](http://en.wikipedia.org/wiki/Rpath): the list of directories where libraries can be found at runtime. In other distributions, this is usually not abused. But in Nix, we have to refer to particular versions of libraries, and thus the rpath has an important role.
The build process adds the `gcc` lib path thinking it may be useful at runtime, but this isn\'t necessary. To address issues like these, Nix provides a tool called [patchelf](https://nixos.org/patchelf.html), which reduces the rpath to the paths that are actually used by the binary.
The build process adds the `gcc` lib path thinking it may be useful at runtime, but this isn't necessary. To address issues like these, Nix provides a tool called [patchelf](https://nixos.org/patchelf.html), which reduces the rpath to the paths that are actually used by the binary.
Even after reducing the rpath, the `hello` binary would still depend upon `gcc` because of some debugging information. This unnecessarily increases the size of our runtime dependencies. We\'ll explore how `strip` can help us with that in the next section.
Even after reducing the rpath, the `hello` binary would still depend upon `gcc` because of some debugging information. This unnecessarily increases the size of our runtime dependencies. We'll explore how `strip` can help us with that in the next section.
## Another phase in the builder
@ -123,4 +123,4 @@ Approaching builds in this way makes packages self-contained, ensuring (apart fr
## Next pill
The next pill will introduce `nix-shell`. With `nix-build`, we\'ve always built derivations from scratch: the source gets unpacked, configured, built, and installed. But this can take a long time for large packages. What if we want to apply some small changes and compile incrementally instead, yet still want to keep a self-contained environment similar to `nix-build`? `nix-shell` enables this.
The next pill will introduce `nix-shell`. With `nix-build`, we've always built derivations from scratch: the source gets unpacked, configured, built, and installed. But this can take a long time for large packages. What if we want to apply some small changes and compile incrementally instead, yet still want to keep a self-contained environment similar to `nix-build`? `nix-shell` enables this.

12
pills/10-developing-with-nix-shell.md
View file

@ -10,9 +10,9 @@ Finally, we will modify our builder to work more ergonomically with a `nix-shell
The [nix-shell](https://nixos.org/manual/nix/stable/command-ref/nix-shell.html) tool drops us in a shell after setting up the environment variables necessary to hack on a derivation. It does not build the derivation; it only serves as a preparation so that we can run the build steps manually.
Recall that in a nix environment, we don\'t have access to libraries or programs unless they have been installed with `nix-env`. However, installing libraries with `nix-env` is not good practice. We prefer to have isolated environments for development, which `nix-shell` provides for us.
Recall that in a nix environment, we don't have access to libraries or programs unless they have been installed with `nix-env`. However, installing libraries with `nix-env` is not good practice. We prefer to have isolated environments for development, which `nix-shell` provides for us.
We can call `nix-shell` on any Nix expression which returns a derivation, but the resulting `bash` shell\'s `PATH` does not have the utilities we want:
We can call `nix-shell` on any Nix expression which returns a derivation, but the resulting `bash` shell's `PATH` does not have the utilities we want:
$ nix-shell hello.nix
[nix-shell]$ make
@ -29,7 +29,7 @@ This means that we can `source` our `builder.sh`, and it will build the derivati
[nix-shell]$ source builder.sh
...
The derivation didn\'t install, but it did build. Note the following:
The derivation didn't install, but it did build. Note the following:
- We sourced `builder.sh` and it ran all of the build steps, including setting up the `PATH` for us.
@ -47,9 +47,9 @@ There are a few things that we would like to change.
First, when we `source`d the `builder.sh` file, we obtained the file in the current directory. What we really wanted was the `builder.sh` that is stored in the nix store, as this is the file that would be used by `nix-build`. To achieve this, the correct technique is to pass an environment variable through the derivation. (Note that `$builder` is already defined, but it points to the bash executable rather than our `builder.sh`. Our `builder.sh` is passed as an argument to bash.)
Second, we don\'t want to run the whole builder: we only want to setup the necessary environment for manually building the project. Thus, we can break `builder.sh` into two files: a `setup.sh` for setting up the environment, and the real `builder.sh` that `nix-build` expects.
Second, we don't want to run the whole builder: we only want to setup the necessary environment for manually building the project. Thus, we can break `builder.sh` into two files: a `setup.sh` for setting up the environment, and the real `builder.sh` that `nix-build` expects.
During our refactoring, we will wrap the build phases in functions to give more structure to our design. Additionally, we\'ll move the `set -e` to the builder file instead of the setup file. The `set -e` is annoying in `nix-shell`, as it will terminate the shell if an error is encountered (such as a mistyped command.)
During our refactoring, we will wrap the build phases in functions to give more structure to our design. Additionally, we'll move the `set -e` to the builder file instead of the setup file. The `set -e` is annoying in `nix-shell`, as it will terminate the shell if an error is encountered (such as a mistyped command.)
Here is our modified `autotools.nix`. Noteworthy is the `setup = ./setup.sh;` attribute in the derivation, which adds `setup.sh` to the nix store and correspondingly adds a `$setup` environment variable in the builder.
@ -155,4 +155,4 @@ With `nix-shell` we are able to drop into an isolated environment suitable for d
## Next pill
In the next pill, we will clean up the nix store. We have written and built derivations which add to the nix store, but until now we haven\'t worried about cleaning up the used space in the store.
In the next pill, we will clean up the nix store. We have written and built derivations which add to the nix store, but until now we haven't worried about cleaning up the used space in the store.

8
pills/11-garbage-collector.md
View file

@ -6,9 +6,9 @@ Today, we will stop focusing on packaging and instead look at a critical compone
## How does garbage collection work?
Programming languages with garbage collectors use the concept of a set of \"garbage collector (or \'GC\') roots\" to keep track of \"live\" objects. A GC root is an object that is always considered \"live\" (unless explicitly removed as GC root). The garbage collection process starts from the GC roots and proceeds by recursively marking object references as \"live\". All other objects can be collected and deleted.
Programming languages with garbage collectors use the concept of a set of \"garbage collector (or 'GC') roots\" to keep track of \"live\" objects. A GC root is an object that is always considered \"live\" (unless explicitly removed as GC root). The garbage collection process starts from the GC roots and proceeds by recursively marking object references as \"live\". All other objects can be collected and deleted.
Instead of objects, Nix\'s garbage collection operates on store paths, [with the GC roots themselves being store paths](https://nixos.org/manual/nix/stable/package-management/garbage-collector-roots.html). . This approach is much more principled than traditional package managers such as `dpkg` or `rpm`, which may leave around unused packages or dangling files.
Instead of objects, Nix's garbage collection operates on store paths, [with the GC roots themselves being store paths](https://nixos.org/manual/nix/stable/package-management/garbage-collector-roots.html). . This approach is much more principled than traditional package managers such as `dpkg` or `rpm`, which may leave around unused packages or dangling files.
The implementation is very simple and transparent to the user. The primary GC roots are stored under `/nix/var/nix/gcroots`. If there is a symlink to a store path, then the linked store path is a GC root.
@ -27,7 +27,7 @@ Before we begin we first run the [nix garbage collector](https://nixos.org/manua
note: currently hard linking saves -0.00 MiB
1169 store paths deleted, 228.43 MiB freed
If we run the garbage collector again it won\'t find anything new to delete, as we expect. After running the garbage collector, the nix store only contains paths with references from the GC roots.
If we run the garbage collector again it won't find anything new to delete, as we expect. After running the garbage collector, the nix store only contains paths with references from the GC roots.
We now install a new program, `bsd-games`, inspect its store path, and examine its GC root. The `nix-store -q --roots` command is used to query the GC roots that refer to a given derivation. In this case, our current user environment refers to `bsd-games`:
@ -120,4 +120,4 @@ Garbage collection in Nix is a powerful mechanism to clean up your system. The `
## Next pill
In the next pill, we will package another project and introduce the \"inputs\" design pattern. We\'ve only played with a single derivation until now; however we\'d like to start organizing a small repository of software. The \"inputs\" pattern is widely used in nixpkgs; it allows us to decouple derivations from the repository itself and increase customization opportunities.
In the next pill, we will package another project and introduce the \"inputs\" design pattern. We've only played with a single derivation until now; however we'd like to start organizing a small repository of software. The \"inputs\" pattern is widely used in nixpkgs; it allows us to decouple derivations from the repository itself and increase customization opportunities.

12
pills/12-inputs-design-pattern.md
View file

@ -89,7 +89,7 @@ After building, `graphviz` can now create `png`s.
## The repository expression
Now that we have two packages, we want to combine them into a single repository. To do so, we\'ll mimic what `nixpkgs` does: we will create a single attribute set containing derivations. This attribute set can then be imported, and derivations can be selected by accessing the top-level attribute set.
Now that we have two packages, we want to combine them into a single repository. To do so, we'll mimic what `nixpkgs` does: we will create a single attribute set containing derivations. This attribute set can then be imported, and derivations can be selected by accessing the top-level attribute set.
Using this technique we are able to abstract from the file names. Instead of referring to a package by `REPO/some/sub/dir/package.nix`, this technique allows us to select a derivation as `importedRepo.package` (or `pkgs.package` in our examples).
@ -137,13 +137,13 @@ We reproduced the very basic behavior of `nixpkgs`: combining multiple derivatio
## The inputs pattern
The approach we\'ve taken so far has a few problems:
The approach we've taken so far has a few problems:
- First, `hello.nix` and `graphviz.nix` are dependent on `nixpkgs`, which they import directly. A better approach would be to pass in `nixpkgs` as an argument, as we did in `autotools.nix`.
- Second, we don\'t have a straightforward way to compile different variants of the same software, such as `graphviz` with or without `libgd` support.
- Second, we don't have a straightforward way to compile different variants of the same software, such as `graphviz` with or without `libgd` support.
- Third, we don\'t have a way to test `graphviz` with a particular `libgd` version.
- Third, we don't have a way to test `graphviz` with a particular `libgd` version.
Until now, our approach to addressing the above problems has been inadequate and required changing the nix expression to match our needs. With the `inputs` pattern, we provide another answer: let the user change the `inputs` of the expression.
@ -153,7 +153,7 @@ When we talk about \"the inputs of an expression\", we are referring to the set
- `libgd` and its dependencies.
The `./src` directory is also an input, but we wouldn\'t change the source from the caller. In `nixpkgs` we prefer to write another expression for version bumps (e.g. because patches or different inputs are needed).
The `./src` directory is also an input, but we wouldn't change the source from the caller. In `nixpkgs` we prefer to write another expression for version bumps (e.g. because patches or different inputs are needed).
Our goal is to make package expressions independent of the repository. To achieve this, we use functions to declare inputs for a derivation. For example, with `graphviz.nix`, we make the following changes to make the derivation independent of the repository and customizable:
@ -210,7 +210,7 @@ If we wanted to build `graphviz` with a specific version of `gd`, it would suffi
If we wanted to change the toolchain, we would simply pass a different `mkDerivation` function.
Let\'s talk a closer look at the snippet and dissect the syntax:
Let's talk a closer look at the snippet and dissect the syntax:
- The entire expression in `default.nix` returns an attribute set with the keys `hello`, `graphviz`, and `graphvizCore`.

20
pills/13-callpackage-design-pattern.md
View file

@ -2,7 +2,7 @@
Welcome to the 13th Nix pill. In the previous [12th pill](12-inputs-design-pattern.md), we introduced the first basic design pattern for organizing a repository of software. In addition, we packaged graphviz so that we had two packages to bundle into an example repository.
The next design pattern we will examine is called the `callPackage` pattern. This technique is extensively used in [nixpkgs](https://github.com/NixOS/nixpkgs), and it\'s the current de facto standard for importing packages in a repository. Its purpose is to reduce the duplication of identifiers between package derivation inputs and repository derivations.
The next design pattern we will examine is called the `callPackage` pattern. This technique is extensively used in [nixpkgs](https://github.com/NixOS/nixpkgs), and it's the current de facto standard for importing packages in a repository. Its purpose is to reduce the duplication of identifiers between package derivation inputs and repository derivations.
## The callPackage convenience
@ -51,15 +51,15 @@ Nix provides a builtin function to do this:
In addition to returning the argument names, the attribute set returned by `functionArgs` indicates whether or not the argument has a default value. For our purposes, we are only interested in the argument names; we do not care about the default values right now.
The next step is to make `callPackage` automatically pass inputs to our package derivations based on the argument names we\'ve just obtained with `functionArgs`.
The next step is to make `callPackage` automatically pass inputs to our package derivations based on the argument names we've just obtained with `functionArgs`.
To do this, we need two things:
- A package repository set containing package derivations that match the arguments names we\'ve obtained
- A package repository set containing package derivations that match the arguments names we've obtained
- A way to obtain an auto-populated attribute set combining the package repository and the return value of `functionArgs`.
The former is easy: we just have to set our package derivation\'s inputs to be package names in a repository, such as `nixpkgs`. For the latter, Nix provides another builtin function:
The former is easy: we just have to set our package derivation's inputs to be package names in a repository, such as `nixpkgs`. For the latter, Nix provides another builtin function:
nix-repl> values = { a = 3; b = 5; c = 10; }
nix-repl> builtins.intersectAttrs values (builtins.functionArgs add)
@ -67,7 +67,7 @@ The former is easy: we just have to set our package derivation\'s inputs to be p
nix-repl> builtins.intersectAttrs (builtins.functionArgs add) values
{ a = 3; b = 5; }
The `intersectAttrs` returns an attribute set whose names are the intersection of both arguments\' attribute names, with the attribute values taken from the second argument.
The `intersectAttrs` returns an attribute set whose names are the intersection of both arguments' attribute names, with the attribute values taken from the second argument.
This is all we need to do: we have obtained the argument names from a function, and populated these with an existing set of attributes. This is our simple implementation of `callPackage`:
@ -77,7 +77,7 @@ This is all we need to do: we have obtained the argument names from a function,
nix-repl> with values; add { inherit a b; }
8
Let\'s dissect the above snippet:
Let's dissect the above snippet:
- We define a `callPackage` variable which is a function.
@ -89,7 +89,7 @@ Let\'s dissect the above snippet:
- Finally, we call the passed function `f` with the resulting intersection.
In the snippet above, we\'ve also demonstrated that the `callPackage` call is equivalent to directly calling `add a b`.
In the snippet above, we've also demonstrated that the `callPackage` call is equivalent to directly calling `add a b`.
We achieved most of what we wanted: to automatically call functions given a set of possible arguments. If an argument is not found within the set we used to call the function, then we receive an error (unless the function has variadic arguments denoted with `...`, as explained in the [5th pill](05-functions-and-imports.md)).
@ -125,7 +125,7 @@ Given our `callPackages`, we can simplify the repository expression in `default.
in
pkgs
Let\'s examine this in detail:
Let's examine this in detail:
- The expression above defines our own package repository, which we call `pkgs`, that contains `hello` along with our two variants of `graphviz`.
@ -141,13 +141,13 @@ Let\'s examine this in detail:
Note how easily we overrode arguments in the case of `graphviz` without `gd`. In addition, note how easy it was to merge two repositories: `nixpkgs` and our `pkgs`!
The reader should notice a magic thing happening. We\'re defining `pkgs` in terms of `callPackage`, and `callPackage` in terms of `pkgs`. That magic is possible thanks to lazy evaluation: `builtins.intersectAttrs` doesn\'t need to know the values in `allPkgs` in order to perform intersection, only the keys that do not require `callPackage` evaluation.
The reader should notice a magic thing happening. We're defining `pkgs` in terms of `callPackage`, and `callPackage` in terms of `pkgs`. That magic is possible thanks to lazy evaluation: `builtins.intersectAttrs` doesn't need to know the values in `allPkgs` in order to perform intersection, only the keys that do not require `callPackage` evaluation.
## Conclusion
The \"`callPackage`\" pattern has simplified our repository considerably. We were able to import packages that require named arguments and call them automatically, given the set of all packages sourced from `nixpkgs`.
We\'ve also introduced some useful builtin functions that allows us to introspect Nix functions and manipulate attributes. These builtin functions are not usually used when packaging software, but rather act as tools for packaging. They are documented in the [Nix manual](https://nixos.org/manual/nix/stable/expressions/builtins.html).
We've also introduced some useful builtin functions that allows us to introspect Nix functions and manipulate attributes. These builtin functions are not usually used when packaging software, but rather act as tools for packaging. They are documented in the [Nix manual](https://nixos.org/manual/nix/stable/expressions/builtins.html).
Writing a repository in Nix is an evolution of writing convenient functions for combining the packages. This pill demonstrates how Nix can be a generic tool to build and deploy software, and how suitable it is to create software repositories with our own conventions.

20
pills/14-override-design-pattern.md
View file

@ -8,9 +8,9 @@ The next design pattern is less necessary, but is useful in many cases and is a
Functional languages are known for being able to compose functions. In particular, these languages gain expressivity from functions that manipulate an original value into a new value having the same structure. This allows us to compose multiple functions to perform the desired modifications.
In Nix, we mostly talk about **functions** that accept inputs in order to return **derivations**. In our world, we want utility functions that are able to manipulate those structures. These utilities add some useful properties to the original value, and we\'d like to be able to apply more utilities on top of the result.
In Nix, we mostly talk about **functions** that accept inputs in order to return **derivations**. In our world, we want utility functions that are able to manipulate those structures. These utilities add some useful properties to the original value, and we'd like to be able to apply more utilities on top of the result.
For example, let\'s say we have an initial derivation `drv` and we want to transform it into a `drv` with debugging information and custom patches:
For example, let's say we have an initial derivation `drv` and we want to transform it into a `drv` with debugging information and custom patches:
debugVersion (applyPatches [ ./patch1.patch ./patch2.patch ] drv)
@ -22,7 +22,7 @@ Designing such utilities is not trivial in a functional language without static
In [pill 12](12-inputs-design-pattern.md) we introduced the inputs design pattern. We do not return a derivation picking dependencies directly from the repository; rather we declare the inputs and let the callers pass the necessary arguments.
In our repository we have a set of attributes that import the expressions of the packages and pass these arguments, getting back a derivation. Let\'s take for example the graphviz attribute:
In our repository we have a set of attributes that import the expressions of the packages and pass these arguments, getting back a derivation. Let's take for example the graphviz attribute:
graphviz = import ./graphviz.nix { inherit mkDerivation gd fontconfig libjpeg bzip2; };
@ -40,7 +40,7 @@ If we wanted to produce a derivation of graphviz with a customized gd version, w
};
}
That\'s hard to maintain. Using `callPackage` would be easier:
That's hard to maintain. Using `callPackage` would be easier:
mygraphviz = callPackage ./graphviz.nix { gd = customgd; };
@ -58,7 +58,7 @@ Note: that `.override` is not a \"method\" in the OO sense as you may think. Nix
Recall that the graphviz attribute in the repository is the derivation returned by the function imported from `graphviz.nix`. We would like to add a further attribute named \"`override`\" to the returned set.
Let\'s start by first creating a function \"`makeOverridable`\". This function will take two arguments: a function (that must return a set) and the set of original arguments to be passed to the function.
Let's start by first creating a function \"`makeOverridable`\". This function will take two arguments: a function (that must return a set) and the set of original arguments to be passed to the function.
We will put this function in a `lib.nix`:
@ -75,7 +75,7 @@ We will put this function in a `lib.nix`:
This `override` attribute is a function taking a set of new arguments, and returns the result of the original function called with the original arguments unified with the new arguments. This is admittedly somewhat confusing, but the examples below should make it clear.
Let\'s try it with `nix repl`:
Let's try it with `nix repl`:
$ nix repl
nix-repl> :l lib.nix
@ -91,11 +91,11 @@ Let\'s try it with `nix repl`:
Note that, as we specified above, the function `f` does not return the plain sum. Instead, it returns a set with the sum bound to the name `result`.
The variable `res` contains the result of the function call without any override. It\'s easy to see in the definition of `makeOverridable`. In addition, you can see that the new `override` attribute is a function.
The variable `res` contains the result of the function call without any override. It's easy to see in the definition of `makeOverridable`. In addition, you can see that the new `override` attribute is a function.
Calling `res.override` with a set will invoke the original function with the overrides, as expected.
This is a good start, but we can\'t override again! This is because the returned set (with `result = 15`) does not have an `override` attribute of its own. This is bad; it breaks further composition.
This is a good start, but we can't override again! This is because the returned set (with `result = 15`) does not have an `override` attribute of its own. This is bad; it breaks further composition.
The solution is simple: the `.override` function should make the result overridable again:
@ -108,9 +108,9 @@ The solution is simple: the `.override` function should make the result overrida
origRes // { override = newArgs: makeOverridable f (origArgs // newArgs); };
}
Please note the `rec` keyword. It\'s necessary so that we can refer to `makeOverridable` from `makeOverridable` itself.
Please note the `rec` keyword. It's necessary so that we can refer to `makeOverridable` from `makeOverridable` itself.
Now let\'s try overriding twice:
Now let's try overriding twice:
nix-repl> :l lib.nix
Added 1 variables.

44
pills/15-nix-search-paths.md
View file

@ -6,21 +6,21 @@ Assuming you followed the previous posts, I hope you are now ready to understand
## The NIX_PATH
The [NIX_PATH environment variable](https://nixos.org/manual/nix/stable/command-ref/env-common.html) is very important. It\'s very similar to the `PATH` environment variable. The syntax is similar, several paths are separated by a colon `:`. Nix will then search for something in those paths from left to right.
The [NIX_PATH environment variable](https://nixos.org/manual/nix/stable/command-ref/env-common.html) is very important. It's very similar to the `PATH` environment variable. The syntax is similar, several paths are separated by a colon `:`. Nix will then search for something in those paths from left to right.
Who uses `NIX_PATH`? The nix expressions! Yes, `NIX_PATH` is not of much use by the nix tools themselves, rather it\'s used when writing nix expressions.
Who uses `NIX_PATH`? The nix expressions! Yes, `NIX_PATH` is not of much use by the nix tools themselves, rather it's used when writing nix expressions.
In the shell for example, when you execute the command `ping`, it\'s being searched in the `PATH` directories. The first one found is the one being used.
In the shell for example, when you execute the command `ping`, it's being searched in the `PATH` directories. The first one found is the one being used.
In nix it\'s exactly the same, however the syntax is different. Instead of just typing `ping` you have to type `<ping>`. Yes, I know\... you are already thinking of `<nixpkgs>`. However, don\'t stop reading here, let\'s keep going.
In nix it's exactly the same, however the syntax is different. Instead of just typing `ping` you have to type `<ping>`. Yes, I know\... you are already thinking of `<nixpkgs>`. However, don't stop reading here, let's keep going.
What\'s `NIX_PATH` good for? Nix expressions may refer to an \"abstract\" path such as `<nixpkgs>`, and it\'s possible to override it from the command line.
What's `NIX_PATH` good for? Nix expressions may refer to an \"abstract\" path such as `<nixpkgs>`, and it's possible to override it from the command line.
For ease we will use `nix-instantiate --eval` to do our tests. I remind you, [nix-instantiate](https://nixos.org/manual/nix/stable/command-ref/nix-instantiate.html) is used to evaluate nix expressions and generate the .drv files. Here we are not interested in building derivations, so evaluation is enough. It can be used for one-shot expressions.
## Fake it a little
It\'s useless from a nix view point, but I think it\'s useful for your own understanding. Let\'s use `PATH` itself as `NIX_PATH`, and try to locate `ping` (or another binary if you don\'t have it).
It's useless from a nix view point, but I think it's useful for your own understanding. Let's use `PATH` itself as `NIX_PATH`, and try to locate `ping` (or another binary if you don't have it).
$ nix-instantiate --eval -E '<ping>'
error: file `ping' was not found in the Nix search path (add it using $NIX_PATH or -I)
@ -49,15 +49,15 @@ You are out of curiosity, right?
$ echo $NIX_PATH
nixpkgs=/home/nix/.nix-defexpr/channels/nixpkgs
You may have a different path, depending on how you added channels etc.. Anyway that\'s the whole point. The `<nixpkgs>` stranger that we used in our nix expressions, is referring to a path in the filesystem specified by `NIX_PATH`.
You may have a different path, depending on how you added channels etc.. Anyway that's the whole point. The `<nixpkgs>` stranger that we used in our nix expressions, is referring to a path in the filesystem specified by `NIX_PATH`.
You can list that directory and realize it\'s simply a checkout of the nixpkgs repository at a specific commit (hint: `.version-suffix`).
You can list that directory and realize it's simply a checkout of the nixpkgs repository at a specific commit (hint: `.version-suffix`).
The `NIX_PATH` variable is exported by `nix.sh`, and that\'s the reason why I always asked you to [source nix.sh](https://nixos.org/manual/nix/stable/installation/env-variables.html) at the beginning of my posts.
The `NIX_PATH` variable is exported by `nix.sh`, and that's the reason why I always asked you to [source nix.sh](https://nixos.org/manual/nix/stable/installation/env-variables.html) at the beginning of my posts.
You may wonder: then I can also specify a different [nixpkgs](https://github.com/NixOS/nixpkgs) path to, e.g., a `git checkout` of `nixpkgs`? Yes, you can and I encourage doing that. We\'ll talk about this in the next pill.
You may wonder: then I can also specify a different [nixpkgs](https://github.com/NixOS/nixpkgs) path to, e.g., a `git checkout` of `nixpkgs`? Yes, you can and I encourage doing that. We'll talk about this in the next pill.
Let\'s define a path for our repository, then! Let\'s say all the `default.nix`, `graphviz.nix` etc. are under `/home/nix/mypkgs`:
Let's define a path for our repository, then! Let's say all the `default.nix`, `graphviz.nix` etc. are under `/home/nix/mypkgs`:
$ export NIX_PATH=mypkgs=/home/nix/mypkgs:$NIX_PATH
$ nix-instantiate --eval '<mypkgs>'
@ -69,20 +69,20 @@ Yes, `nix-build` also accepts paths with angular brackets. We first evaluate the
The [nix-env](https://nixos.org/manual/nix/stable/command-ref/nix-env.html) command is a little different than `nix-instantiate` and `nix-build`. Whereas `nix-instantiate` and `nix-build` require a starting nix expression, `nix-env` does not.
You may be crippled by this concept at the beginning, you may think `nix-env` uses `NIX_PATH` to find the `nixpkgs` repository. But that\'s not it.
You may be crippled by this concept at the beginning, you may think `nix-env` uses `NIX_PATH` to find the `nixpkgs` repository. But that's not it.
The `nix-env` command uses `~/.nix-defexpr`, which is also part of `NIX_PATH` by default, but that\'s only a coincidence. If you empty `NIX_PATH`, `nix-env` will still be able to find derivations because of `~/.nix-defexpr`.
The `nix-env` command uses `~/.nix-defexpr`, which is also part of `NIX_PATH` by default, but that's only a coincidence. If you empty `NIX_PATH`, `nix-env` will still be able to find derivations because of `~/.nix-defexpr`.
So if you run `nix-env -i graphviz` inside your repository, it will install the nixpkgs one. Same if you set `NIX_PATH` to point to your repository.
In order to specify an alternative to `~/.nix-defexpr` it\'s possible to use the -f option:
In order to specify an alternative to `~/.nix-defexpr` it's possible to use the -f option:
$ nix-env -f '<mypkgs>' -i graphviz
warning: there are multiple derivations named `graphviz'; using the first one
replacing old `graphviz'
installing `graphviz'
Oh why did it say there\'s another derivation named graphviz? Because both `graphviz` and `graphvizCore` attributes in our repository have the name \"graphviz\" for the derivation:
Oh why did it say there's another derivation named graphviz? Because both `graphviz` and `graphvizCore` attributes in our repository have the name \"graphviz\" for the derivation:
$ nix-env -f '<mypkgs>' -qaP
graphviz graphviz
@ -95,13 +95,13 @@ By default `nix-env` parses all derivations and uses the derivation names to int
replacing old `graphviz'
installing `graphviz'
This form, other than being more precise, it\'s also faster because `nix-env` does not have to parse all the derivations.
This form, other than being more precise, it's also faster because `nix-env` does not have to parse all the derivations.
For completeness: you must install `graphvizCore` with -A, since without the -A switch it\'s ambiguous.
For completeness: you must install `graphvizCore` with -A, since without the -A switch it's ambiguous.
In summary, it may happen when playing with nix that `nix-env` picks a different derivation than `nix-build`. In that case you probably specified `NIX_PATH`, but `nix-env` is instead looking into `~/.nix-defexpr`.
Why is `nix-env` having this different behavior? I don\'t know specifically by myself either, but the answers could be:
Why is `nix-env` having this different behavior? I don't know specifically by myself either, but the answers could be:
- `nix-env` tries to be generic, thus it does not look for `nixpkgs` in `NIX_PATH`, rather it looks in `~/.nix-defexpr`.
@ -109,15 +109,15 @@ Why is `nix-env` having this different behavior? I don\'t know specifically by m
It may also happen to you **that you cannot match a derivation name when installing**, because of the derivation name vs -A switch described above. Maybe `nix-env` wanted to be more friendly in this case for default user setups.
It may or may not make sense for you, or it\'s like that for historical reasons, but that\'s how it works currently, unless somebody comes up with a better idea.
It may or may not make sense for you, or it's like that for historical reasons, but that's how it works currently, unless somebody comes up with a better idea.
## Conclusion
The `NIX_PATH` variable is the search path used by nix when using the angular brackets syntax. It\'s possible to refer to \"abstract\" paths inside nix expressions and define the \"concrete\" path by means of `NIX_PATH`, or the usual -I flag in nix tools.
The `NIX_PATH` variable is the search path used by nix when using the angular brackets syntax. It's possible to refer to \"abstract\" paths inside nix expressions and define the \"concrete\" path by means of `NIX_PATH`, or the usual -I flag in nix tools.
We\'ve also explained some of the uncommon `nix-env` behaviors for newcomers. The `nix-env` tool does not use `NIX_PATH` to search for packages, but rather for `~/.nix-defexpr`. Beware of that!
We've also explained some of the uncommon `nix-env` behaviors for newcomers. The `nix-env` tool does not use `NIX_PATH` to search for packages, but rather for `~/.nix-defexpr`. Beware of that!
In general do not abuse `NIX_PATH`, when possible use relative paths when writing your own nix expressions. Of course, in the case of `<nixpkgs>` in our repository, that\'s a perfectly fine usage of `NIX_PATH`. Instead, inside our repository itself, refer to expressions with relative paths like `./hello.nix`.
In general do not abuse `NIX_PATH`, when possible use relative paths when writing your own nix expressions. Of course, in the case of `<nixpkgs>` in our repository, that's a perfectly fine usage of `NIX_PATH`. Instead, inside our repository itself, refer to expressions with relative paths like `./hello.nix`.
## Next pill

24
pills/16-nixpkgs-parameters.md
View file

@ -1,6 +1,6 @@
# Nixpkgs Parameters
Welcome to the 16th Nix pill. In the previous [15th](15-nix-search-paths.md) pill we\'ve realized how nix finds expressions with the angular brackets syntax, so that we finally know where `<nixpkgs>` is located on our system.
Welcome to the 16th Nix pill. In the previous [15th](15-nix-search-paths.md) pill we've realized how nix finds expressions with the angular brackets syntax, so that we finally know where `<nixpkgs>` is located on our system.
We can start diving into the [nixpkgs repository](https://github.com/NixOS/nixpkgs), through all the various tools and design patterns. Please note that also `nixpkgs` has its own manual, underlying the difference between the general `nix` language and the `nixpkgs` repository.
@ -14,7 +14,7 @@ Also `nixpkgs` has its own [default.nix](https://github.com/NixOS/nixpkgs/blob/m
The `all-packages.nix` is then the file that composes all the packages. Note the `pkgs/` subdirectory, while nixos is in the `nixos/` subdirectory.
The `all-packages.nix` is a bit contrived. First of all, it\'s a function. It accepts a couple of interesting parameters:
The `all-packages.nix` is a bit contrived. First of all, it's a function. It accepts a couple of interesting parameters:
- `system`: defaults to the current system
@ -22,7 +22,7 @@ The `all-packages.nix` is a bit contrived. First of all, it\'s a function. It ac
- others\...
The **system** parameter, as per comment in the expression, it\'s the system for which the packages will be built. It allows for example to install i686 packages on amd64 machines.
The **system** parameter, as per comment in the expression, it's the system for which the packages will be built. It allows for example to install i686 packages on amd64 machines.
The **config** parameter is a simple attribute set. Packages can read some of its values and change the behavior of some derivations.
@ -37,21 +37,21 @@ You will find this parameter in many other .nix expressions (e.g. release expres
let pkgs = import <nixpkgs> { inherit system; };
...
Why is it useful? With this parameter it\'s very easy to select a set of packages for a particular system. For example:
Why is it useful? With this parameter it's very easy to select a set of packages for a particular system. For example:
nix-build -A psmisc --argstr system i686-linux
This will build the psmisc derivation for i686-linux instead of x86_64-linux. This concept is very similar to multi-arch of Debian.
The setup for cross compiling is also in `nixpkgs`, however it\'s a little contrived to talk about it and I don\'t know much of it either.
The setup for cross compiling is also in `nixpkgs`, however it's a little contrived to talk about it and I don't know much of it either.
## The config parameter
I\'m sure on the wiki or other manuals you\'ve read about `~/.config/nixpkgs/config.nix` (previously `~/.nixpkgs/config.nix`) and I\'m sure you\'ve wondered whether that\'s hardcoded in nix. It\'s not, it\'s in [nixpkgs](https://github.com/NixOS/nixpkgs/blob/32c523914fdb8bf9cc7912b1eba023a8daaae2e8/pkgs/top-level/impure.nix#L28).
I'm sure on the wiki or other manuals you've read about `~/.config/nixpkgs/config.nix` (previously `~/.nixpkgs/config.nix`) and I'm sure you've wondered whether that's hardcoded in nix. It's not, it's in [nixpkgs](https://github.com/NixOS/nixpkgs/blob/32c523914fdb8bf9cc7912b1eba023a8daaae2e8/pkgs/top-level/impure.nix#L28).
The `all-packages.nix` expression accepts the `config` parameter. If it\'s `null`, then it reads the `NIXPKGS_CONFIG` environment variable. If not specified, `nixpkgs` will pick `$HOME/.config/nixpkgs/config.nix`.
The `all-packages.nix` expression accepts the `config` parameter. If it's `null`, then it reads the `NIXPKGS_CONFIG` environment variable. If not specified, `nixpkgs` will pick `$HOME/.config/nixpkgs/config.nix`.
After determining `config.nix`, it will be imported as a nix expression, and that will be the value of `config` (in case it hasn\'t been passed as parameter to import `<nixpkgs>`).
After determining `config.nix`, it will be imported as a nix expression, and that will be the value of `config` (in case it hasn't been passed as parameter to import `<nixpkgs>`).
The `config` is available in the resulting repository:
@ -69,7 +69,7 @@ For example, `config.allowUnfree` is an attribute that forbids building packages
## About .nix functions
A `.nix` file contains a nix expression. Thus it can also be a function. I remind you that `nix-build` expects the expression to return a derivation. Therefore it\'s natural to return straight a derivation from a `.nix` file. However, it\'s also very natural for the `.nix` file to accept some parameters, in order to tweak the derivation being returned.
A `.nix` file contains a nix expression. Thus it can also be a function. I remind you that `nix-build` expects the expression to return a derivation. Therefore it's natural to return straight a derivation from a `.nix` file. However, it's also very natural for the `.nix` file to accept some parameters, in order to tweak the derivation being returned.
In this case, nix does a trick:
@ -89,11 +89,11 @@ Does it work if you have a function returning a function that returns a derivati
## Conclusion
We\'ve unleashed the `<nixpkgs>` repository. It\'s a function that accepts some parameters, and returns the set of all packages. Due to laziness, only the accessed derivations will be built.
We've unleashed the `<nixpkgs>` repository. It's a function that accepts some parameters, and returns the set of all packages. Due to laziness, only the accessed derivations will be built.
You can use this repository to build your own packages as we\'ve seen in the previous pill when creating our own repository.
You can use this repository to build your own packages as we've seen in the previous pill when creating our own repository.
Lately I\'m a little busy with the NixOS 14.11 release and other stuff, and I\'m also looking toward migrating from blogger to a more coder-oriented blogging platform. So sorry for the delayed and shorter pills :)
Lately I'm a little busy with the NixOS 14.11 release and other stuff, and I'm also looking toward migrating from blogger to a more coder-oriented blogging platform. So sorry for the delayed and shorter pills :)
## Next pill

34
pills/17-nixpkgs-overriding-packages.md
View file

@ -1,8 +1,8 @@
# Nixpkgs Overriding Packages
Welcome to the 17th Nix pill. In the previous [16th](16-nixpkgs-parameters.md) pill we have started to dive into the [nixpkgs repository](http://github.com/NixOS/nixpkgs). `Nixpkgs` is a function, and we\'ve looked at some parameters like `system` and `config`.
Welcome to the 17th Nix pill. In the previous [16th](16-nixpkgs-parameters.md) pill we have started to dive into the [nixpkgs repository](http://github.com/NixOS/nixpkgs). `Nixpkgs` is a function, and we've looked at some parameters like `system` and `config`.
Today we\'ll talk about a special attribute: `config.packageOverrides`. Overriding packages in a set with fixed point can be considered another design pattern in nixpkgs.
Today we'll talk about a special attribute: `config.packageOverrides`. Overriding packages in a set with fixed point can be considered another design pattern in nixpkgs.
## Overriding a package
@ -10,16 +10,16 @@ Recall the override design pattern from the [nix pill 14](14-override-design-pat
We put the override function in the returned attribute set of the original function call.
Take for example graphviz. It has an input parameter xorg. If it\'s null, then graphviz will build without X support.
Take for example graphviz. It has an input parameter xorg. If it's null, then graphviz will build without X support.
$ nix repl
nix-repl> :l <nixpkgs>
Added 4360 variables.
nix-repl> :b graphviz.override { withXorg = false; }
This will build graphviz without X support, it\'s as simple as that.
This will build graphviz without X support, it's as simple as that.
However, let\'s say a package `P` depends on graphviz, how do we make `P` depend on the new graphviz without X support?
However, let's say a package `P` depends on graphviz, how do we make `P` depend on the new graphviz without X support?
## In an imperative world\...
@ -29,11 +29,11 @@ However, let\'s say a package `P` depends on graphviz, how do we make `P` depend
pkgs.graphviz = pkgs.graphviz.override { withXorg = false; };
build(pkgs.P)
Given `pkgs.P` depends on `pkgs.graphviz`, it\'s easy to build `P` with the replaced graphviz. In a pure functional language it\'s not that easy because you can assign to variables only once.
Given `pkgs.P` depends on `pkgs.graphviz`, it's easy to build `P` with the replaced graphviz. In a pure functional language it's not that easy because you can assign to variables only once.
## Fixed point
The fixed point with lazy evaluation is crippling but about necessary in a language like Nix. It lets us achieve something similar to what we\'d do imperatively.
The fixed point with lazy evaluation is crippling but about necessary in a language like Nix. It lets us achieve something similar to what we'd do imperatively.
Follows the definition of fixed point in [nixpkgs](https://github.com/NixOS/nixpkgs/blob/f224a4f1b32b3e813783d22de54e231cd8ea2448/lib/fixed-points.nix#L19):
@ -47,9 +47,9 @@ Follows the definition of fixed point in [nixpkgs](https://github.com/NixOS/nixp
result;
}
It\'s a function that accepts a function `f`, calls `f result` on the result just returned by `f result` and returns it. In other words it\'s `f(f(f(....`
It's a function that accepts a function `f`, calls `f result` on the result just returned by `f result` and returns it. In other words it's `f(f(f(....`
At first sight, it\'s an infinite loop. With lazy evaluation it isn\'t, because the call is done only when needed.
At first sight, it's an infinite loop. With lazy evaluation it isn't, because the call is done only when needed.
nix-repl> fix = f: let result = f result; in result
nix-repl> pkgs = self: { a = 3; b = 4; c = self.a+self.b; }
@ -64,13 +64,13 @@ Without the `rec` keyword, we were able to refer to `a` and `b` of the same set.
- The `pkgs` function gets called again to get the value of `a` and `b`.
The trick is that `c` is not needed to be evaluated in the inner call, thus it doesn\'t go in an infinite loop.
The trick is that `c` is not needed to be evaluated in the inner call, thus it doesn't go in an infinite loop.
Won\'t go further with the explanation here. A good post about fixed point and Nix can be [found here](http://r6.ca/blog/20140422T142911Z.html).
Won't go further with the explanation here. A good post about fixed point and Nix can be [found here](http://r6.ca/blog/20140422T142911Z.html).
### Overriding a set with fixed point
Given that `self.a` and `self.b` refer to the passed set and not to the literal set in the function, we\'re able to override both `a` and `b` and get a new value for `c`:
Given that `self.a` and `self.b` refer to the passed set and not to the literal set in the function, we're able to override both `a` and `b` and get a new value for `c`:
nix-repl> overrides = { a = 1; b = 2; }
nix-repl> let newpkgs = pkgs (newpkgs // overrides); in newpkgs
@ -82,7 +82,7 @@ In the first case we computed pkgs with the overrides, in the second case we als
## Overriding nixpkgs packages
We\'ve seen how to override attributes in a set such that they get recursively picked by dependent attributes. This approach can be used for derivations too, after all `nixpkgs` is a giant set of attributes that depend on each other.
We've seen how to override attributes in a set such that they get recursively picked by dependent attributes. This approach can be used for derivations too, after all `nixpkgs` is a giant set of attributes that depend on each other.
To do this, `nixpkgs` offers `config.packageOverrides`. So `nixpkgs` returns a fixed point of the package set, and `packageOverrides` is used to inject the overrides.
@ -102,9 +102,9 @@ Now we can build e.g. asciidoc-full and it will automatically use the overridden
nix-repl> pkgs = import <nixpkgs> { config = import ./config.nix; }
nix-repl> :b pkgs.asciidoc-full
Note how we pass the `config` with `packageOverrides` when importing `nixpkgs`. Then `pkgs.asciidoc-full` is a derivation that has graphviz input (`pkgs.asciidoc` is the lighter version and doesn\'t use graphviz at all).
Note how we pass the `config` with `packageOverrides` when importing `nixpkgs`. Then `pkgs.asciidoc-full` is a derivation that has graphviz input (`pkgs.asciidoc` is the lighter version and doesn't use graphviz at all).
Since there\'s no version of asciidoc with graphviz without X support in the binary cache, Nix will recompile the needed stuff for you.
Since there's no version of asciidoc with graphviz without X support in the binary cache, Nix will recompile the needed stuff for you.
## The \~/.config/nixpkgs/config.nix file
@ -114,9 +114,9 @@ Instead of passing it explicitly whenever we import `nixpkgs`, it will be automa
## Conclusion
We\'ve learned about a new design pattern: using fixed point for overriding packages in a package set.
We've learned about a new design pattern: using fixed point for overriding packages in a package set.
Whereas in an imperative setting, like with other package managers, a library is installed replacing the old version and applications will use it, in Nix it\'s not that straight and simple. But it\'s more precise.
Whereas in an imperative setting, like with other package managers, a library is installed replacing the old version and applications will use it, in Nix it's not that straight and simple. But it's more precise.
Nix applications will depend on specific versions of libraries, hence the reason why we have to recompile asciidoc to use the new graphviz library.

20
pills/18-nix-store-paths.md
View file

@ -1,14 +1,14 @@
# Nix Store Paths
Welcome to the 18th Nix pill. In the previous [17th](17-nixpkgs-overriding-packages.md) pill we have scratched the surface of the `nixpkgs` repository structure. It is a set of packages, and it\'s possible to override such packages so that all other packages will use the overrides.
Welcome to the 18th Nix pill. In the previous [17th](17-nixpkgs-overriding-packages.md) pill we have scratched the surface of the `nixpkgs` repository structure. It is a set of packages, and it's possible to override such packages so that all other packages will use the overrides.
Before reading existing derivations, I\'d like to talk about store paths and how they are computed. In particular we are interested in fixed store paths that depend on an integrity hash (e.g. a sha256), which is usually applied to source tarballs.
Before reading existing derivations, I'd like to talk about store paths and how they are computed. In particular we are interested in fixed store paths that depend on an integrity hash (e.g. a sha256), which is usually applied to source tarballs.
The way store paths are computed is a little contrived, mostly due to historical reasons. Our reference will be the [Nix source code](https://github.com/NixOS/nix/blob/07f992a74b64f4376d5b415d0042babc924772f3/src/libstore/store-api.cc#L197).
## Source paths
Let\'s start simple. You know nix allows relative paths to be used, such that the file or directory is stored in the nix store, that is `./myfile` gets stored into `/nix/store/.......` We want to understand how is the store path generated for such a file:
Let's start simple. You know nix allows relative paths to be used, such that the file or directory is stored in the nix store, that is `./myfile` gets stored into `/nix/store/.......` We want to understand how is the store path generated for such a file:
$ echo mycontent > myfile
@ -79,9 +79,9 @@ Finally the comments tell us to compute the base-32 representation of the first
## Output paths
Output paths are usually generated for derivations. We use the above example because it\'s simple. Even if we didn\'t build the derivation, nix knows the out path `hs0yi5n5nw6micqhy8l1igkbhqdkzqa1`. This is because the out path only depends on inputs.
Output paths are usually generated for derivations. We use the above example because it's simple. Even if we didn't build the derivation, nix knows the out path `hs0yi5n5nw6micqhy8l1igkbhqdkzqa1`. This is because the out path only depends on inputs.
It\'s computed in a similar way to source paths, except that the .drv is hashed and the type of derivation is `output:out`. In case of multiple outputs, we may have different `output:<id>`.
It's computed in a similar way to source paths, except that the .drv is hashed and the type of derivation is `output:out`. In case of multiple outputs, we may have different `output:<id>`.
At the time nix computes the out path, the .drv contains an empty string for each out path. So what we do is getting our .drv and replacing the out path with an empty string:
@ -96,7 +96,7 @@ The `myout.drv` is the .drv state in which nix is when computing the out path fo
$ nix-hash --type sha256 --truncate --base32 --flat myout.str
hs0yi5n5nw6micqhy8l1igkbhqdkzqa1
Then nix puts that out path in the .drv, and that\'s it.
Then nix puts that out path in the .drv, and that's it.
In case the .drv has input derivations, that is it references other .drv, then such .drv paths are replaced by this same algorithm which returns a hash.
@ -110,7 +110,7 @@ A derivation can take three special attributes: `outputHashMode`, `outputHash` a
The builder must create the out path and make sure its hash is the same as the one declared with `outputHash`.
Let\'s say our builder should create a file whose contents is `mycontent`:
Let's say our builder should create a file whose contents is `mycontent`:
$ echo mycontent > myfile
$ sha256sum myfile
@ -118,7 +118,7 @@ Let\'s say our builder should create a file whose contents is `mycontent`:
nix-repl> derivation { name = "bar"; system = "x86_64-linux"; builder = "none"; outputHashMode = "flat"; outputHashAlgo = "sha256"; outputHash = "f3f3c4763037e059b4d834eaf68595bbc02ba19f6d2a500dce06d124e2cd99bb"; }
«derivation /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv»
Inspect the .drv and see that it also stored the fact that it\'s a fixed-output derivation with sha256 algorithm, compared to the previous examples:
Inspect the .drv and see that it also stored the fact that it's a fixed-output derivation with sha256 algorithm, compared to the previous examples:
```
$ nix derivation show /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv
@ -135,7 +135,7 @@ $ nix derivation show /nix/store/ymsf5zcqr9wlkkqdjwhqllgwa97rff5i-bar.drv
}
```
It doesn\'t matter which input derivations are being used, the final out path must only depend on the declared hash.
It doesn't matter which input derivations are being used, the final out path must only depend on the declared hash.
What nix does is to create an intermediate string representation of the fixed-output content:
@ -155,7 +155,7 @@ Hence, the store path only depends on the declared fixed-output hash.
There are other types of store paths, but you get the idea. Nix first hashes the contents, then creates a string description, and the final store path is the hash of this string.
Also we\'ve introduced some fundamentals, in particular the fact that Nix knows beforehand the out path of a derivation since it only depends on the inputs. We\'ve also introduced fixed-output derivations which are especially used by the nixpkgs repository for downloading and verifying source tarballs.
Also we've introduced some fundamentals, in particular the fact that Nix knows beforehand the out path of a derivation since it only depends on the inputs. We've also introduced fixed-output derivations which are especially used by the nixpkgs repository for downloading and verifying source tarballs.
## Next pill

30
pills/19-fundamentals-of-stdenv.md
View file

@ -4,11 +4,11 @@ Welcome to the 19th Nix pill. In the previous [18th](18-nix-store-paths.md) pill
This time we will instead look into `nixpkgs`, in particular one of its core derivations: `stdenv`.
The `stdenv` is not treated as a special derivation by Nix, but it\'s very important for the `nixpkgs` repository. It serves as a base for packaging software. It is used to pull in dependencies such as the GCC toolchain, GNU make, core utilities, patch and diff utilities, and so on: basic tools needed to compile a huge pile of software currently present in `nixpkgs`.
The `stdenv` is not treated as a special derivation by Nix, but it's very important for the `nixpkgs` repository. It serves as a base for packaging software. It is used to pull in dependencies such as the GCC toolchain, GNU make, core utilities, patch and diff utilities, and so on: basic tools needed to compile a huge pile of software currently present in `nixpkgs`.
## What is stdenv?
First of all, `stdenv` is a derivation, and it\'s a very simple one:
First of all, `stdenv` is a derivation, and it's a very simple one:
$ nix-build '<nixpkgs>' -A stdenv
/nix/store/k4jklkcag4zq4xkqhkpy156mgfm34ipn-stdenv
@ -19,9 +19,9 @@ First of all, `stdenv` is a derivation, and it\'s a very simple one:
result/nix-support:
propagated-user-env-packages
It has just two files: `/setup` and `/nix-support/propagated-user-env-packages`. Don\'t worry about the latter. It\'s empty, in fact. The important file is `/setup`.
It has just two files: `/setup` and `/nix-support/propagated-user-env-packages`. Don't worry about the latter. It's empty, in fact. The important file is `/setup`.
How can this simple derivation pull in all of the toolchain and basic tools needed to compile packages? Let\'s look at the runtime dependencies:
How can this simple derivation pull in all of the toolchain and basic tools needed to compile packages? Let's look at the runtime dependencies:
$ nix-store -q --references result
/nix/store/3a45nb37s0ndljp68228snsqr3qsyp96-bzip2-1.0.6
@ -43,13 +43,13 @@ Remember our generic `builder.sh` in [Pill 8](08-generic-builders.md)? It sets u
The [stdenv setup file](https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh) is exactly that. It sets up several environment variables like `PATH` and creates some helper bash functions to build a package. I invite you to read it.
The hardcoded toolchain and utilities are used to initially fill up the environment variables so that it\'s more pleasant to run common commands, similar to what we did with our builder with `baseInputs` and `buildInputs`.
The hardcoded toolchain and utilities are used to initially fill up the environment variables so that it's more pleasant to run common commands, similar to what we did with our builder with `baseInputs` and `buildInputs`.
The build with `stdenv` works in phases. Phases are like `unpackPhase`, `configurePhase`, `buildPhase`, `checkPhase`, `installPhase`, `fixupPhase`. You can see the default list in the `genericBuild` function.
What `genericBuild` does is just run these phases. Default phases are just bash functions. You can easily read them.
Every phase has hooks to run commands before and after the phase has been executed. Phases can be overwritten, reordered, whatever, it\'s just bash code.
Every phase has hooks to run commands before and after the phase has been executed. Phases can be overwritten, reordered, whatever, it's just bash code.
How to use this file? Like our old builder. To test it, we enter a fake empty derivation, source the `stdenv` `setup`, unpack the hello sources and build it:
@ -69,11 +69,11 @@ So we ran the `configurePhase` function and `buildPhase` function and they worke
## How the setup file is built
Until now we worked with plain bash scripts. What about the Nix side? The `nixpkgs` repository offers a useful function, like we did with our old builder. It is a wrapper around the raw derivation function which pulls in the `stdenv` for us, and runs `genericBuild`. It\'s [stdenv.mkDerivation](https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/make-derivation.nix).
Until now we worked with plain bash scripts. What about the Nix side? The `nixpkgs` repository offers a useful function, like we did with our old builder. It is a wrapper around the raw derivation function which pulls in the `stdenv` for us, and runs `genericBuild`. It's [stdenv.mkDerivation](https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/make-derivation.nix).
Note how `stdenv` is a derivation but it\'s also an attribute set which contains some other attributes, like `mkDerivation`. Nothing fancy here, just convenience.
Note how `stdenv` is a derivation but it's also an attribute set which contains some other attributes, like `mkDerivation`. Nothing fancy here, just convenience.
Let\'s write a `hello.nix` expression using this newly discovered `stdenv`:
Let's write a `hello.nix` expression using this newly discovered `stdenv`:
with import <nixpkgs> { };
stdenv.mkDerivation {
@ -81,7 +81,7 @@ Let\'s write a `hello.nix` expression using this newly discovered `stdenv`:
src = ./hello-2.10.tar.gz;
}
Don\'t be scared by the `with` expression. It pulls the `nixpkgs` repository into scope, so we can directly use `stdenv`. It looks very similar to the hello expression in [Pill 8](08-generic-builders.md).
Don't be scared by the `with` expression. It pulls the `nixpkgs` repository into scope, so we can directly use `stdenv`. It looks very similar to the hello expression in [Pill 8](08-generic-builders.md).
It builds, and runs fine:
@ -93,7 +93,7 @@ It builds, and runs fine:
## The stdenv.mkDerivation builder
Let\'s take a look at the builder used by `mkDerivation`. You can read the code [here in nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/make-derivation.nix):
Let's take a look at the builder used by `mkDerivation`. You can read the code [here in nixpkgs](https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/make-derivation.nix):
{
# ...
@ -114,7 +114,7 @@ You can open [default-builder.sh](https://github.com/NixOS/nixpkgs/blob/master/p
source $stdenv/setup
genericBuild
It\'s what we did in [Pill 10](10-developing-with-nix-shell.md) to make the derivations `nix-shell` friendly. When entering the shell, the setup file only sets up the environment without building anything. When doing `nix-build`, it actually runs the build process.
It's what we did in [Pill 10](10-developing-with-nix-shell.md) to make the derivations `nix-shell` friendly. When entering the shell, the setup file only sets up the environment without building anything. When doing `nix-build`, it actually runs the build process.
To get a clear understanding of the environment variables, look at the .drv of the hello derivation:
@ -171,7 +171,7 @@ warning: you did not specify '--add-root'; the result might be removed by the ga
}
```
It\'s so short I decided to paste it entirely above. The builder is bash, with `-e default-builder.sh` arguments. Then you can see the `src` and `stdenv` environment variables.
It's so short I decided to paste it entirely above. The builder is bash, with `-e default-builder.sh` arguments. Then you can see the `src` and `stdenv` environment variables.
The last bit, the `unpackPhase` in the setup, is used to unpack the sources and enter the directory. Again, like we did in our old builder.
@ -189,9 +189,9 @@ The overall process is simple:
- `genericBuild`
That\'s it. Everything you need to know about the stdenv phases is in the [setup file](https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh).
That's it. Everything you need to know about the stdenv phases is in the [setup file](https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh).
Really, take your time to read that file. Don\'t forget that juicy docs are also available in the [nixpkgs manual](http://nixos.org/nixpkgs/manual/#chap-stdenv).
Really, take your time to read that file. Don't forget that juicy docs are also available in the [nixpkgs manual](http://nixos.org/nixpkgs/manual/#chap-stdenv).
## Next pill\...

24
pills/20-basic-dependencies-and-hooks.md
View file

@ -1,8 +1,8 @@
# Basic Dependencies and Hooks
Welcome to the 20th Nix pill. In the previous [19th](19-fundamentals-of-stdenv.md) pill we introduced Nixpkgs\' stdenv, including `setup.sh` script, `default-builder.sh` helper script, and `stdenv.mkDerivation` builder. We focused on how stdenv is put together, and how it\'s used, and a bit about the phases of `genericBuild`.
Welcome to the 20th Nix pill. In the previous [19th](19-fundamentals-of-stdenv.md) pill we introduced Nixpkgs' stdenv, including `setup.sh` script, `default-builder.sh` helper script, and `stdenv.mkDerivation` builder. We focused on how stdenv is put together, and how it's used, and a bit about the phases of `genericBuild`.
This time, we\'ll focus on the interaction of packages built with `stdenv.mkDerivation`. Packages need to depend on each other, of course. For this we have `buildInputs` and `propagatedBuildInputs` attributes. We\'ve also found that dependencies sometimes need to influence their dependents in ways the dependents can\'t or shouldn\'t predict. For this we have setup hooks and env hooks. Together, these 4 concepts support almost all build-time package interactions.
This time, we'll focus on the interaction of packages built with `stdenv.mkDerivation`. Packages need to depend on each other, of course. For this we have `buildInputs` and `propagatedBuildInputs` attributes. We've also found that dependencies sometimes need to influence their dependents in ways the dependents can't or shouldn't predict. For this we have setup hooks and env hooks. Together, these 4 concepts support almost all build-time package interactions.
<div class="info">
@ -12,7 +12,7 @@ Note: The complexity of the dependencies and hooks infrastructure has increased,
## The `buildInputs` Attribute
For the simplest dependencies where the current package directly needs another, we use the `buildInputs` attribute. This is exactly the pattern used in our builder in [Pill 8](08-generic-builders.html). To demo this, let\'s build GNU Hello, and then another package which provides a shell script that `exec`s it.
For the simplest dependencies where the current package directly needs another, we use the `buildInputs` attribute. This is exactly the pattern used in our builder in [Pill 8](08-generic-builders.html). To demo this, let's build GNU Hello, and then another package which provides a shell script that `exec`s it.
let
@ -150,7 +150,7 @@ The `buildInputs` covers direct dependencies, but what about indirect dependenci
See how the intermediate package has a `propagatedBuildInputs` dependency, but the wrapper only needs a `buildInputs` dependency on the intermediary.
How does this work? You might think we do something in Nix, but actually it\'s done not at eval time but at build time in bash. let\'s look at part of the `fixupPhase` of stdenv:
How does this work? You might think we do something in Nix, but actually it's done not at eval time but at build time in bash. let's look at part of the `fixupPhase` of stdenv:
fixupPhase() {
@ -179,7 +179,7 @@ This dumps the propagated build inputs in a so-named file in `$out/nix-support/`
fi
}
See how `findInputs` is actually recursive, looking at the propagated build inputs of each dependency, and those dependencies\' propagated build inputs, etc.
See how `findInputs` is actually recursive, looking at the propagated build inputs of each dependency, and those dependencies' propagated build inputs, etc.
We actually simplified the `findInputs` call site from before; `propagatedBuildInputs` is also looped over in reality:
@ -188,13 +188,13 @@ We actually simplified the `findInputs` call site from before; `propagatedBuildI
findInputs $i
done
This demonstrates an important point. For the *current* package alone, it doesn\'t matter whether a dependency is propagated or not. It will be processed the same way: called with `findInputs` and `addToEnv`. (The packages discovered by `findInputs`, which are also accumulated in `pkgs` and passed to `addToEnv`, are also the same in both cases.) Downstream however, it certainly does matter because only the propagated immediate dependencies are put in the `$out/nix-support/propagated-build-inputs`.
This demonstrates an important point. For the *current* package alone, it doesn't matter whether a dependency is propagated or not. It will be processed the same way: called with `findInputs` and `addToEnv`. (The packages discovered by `findInputs`, which are also accumulated in `pkgs` and passed to `addToEnv`, are also the same in both cases.) Downstream however, it certainly does matter because only the propagated immediate dependencies are put in the `$out/nix-support/propagated-build-inputs`.
## Setup Hooks
As we mentioned above, sometimes dependencies need to influence the packages that use them in ways other than just *being* a dependency. [^1] `propagatedBuildInputs` can actually be seen as an example of this: packages using that are effectively \"injecting\" those dependencies as extra `buildInputs` in their downstream dependents. But in general, a dependency might affect the packages it depends on in arbitrary ways. *Arbitrary* is the key word here. We could teach `setup.sh` things about upstream packages like `pkg/nix-support/propagated-build-inputs`, but not arbitrary interactions.
Setup hooks are the basic building block we have for this. In nixpkgs, a \"hook\" is basically a bash callback, and a setup hook is no exception. Let\'s look at the last part of `findInputs` we haven\'t covered:
Setup hooks are the basic building block we have for this. In nixpkgs, a \"hook\" is basically a bash callback, and a setup hook is no exception. Let's look at the last part of `findInputs` we haven't covered:
findInputs() {
local pkg=$1
@ -211,13 +211,13 @@ Setup hooks are the basic building block we have for this. In nixpkgs, a \"hook\
If a package includes the path `pkg/nix-support/setup-hook`, it will be sourced by any stdenv-based build including that as a dependency.
This is strictly more general than any of the other mechanisms introduced in this chapter. For example, try writing a setup hook that has the same effect as a *propagatedBuildInputs* entry. One can almost think of this as an escape hatch around Nix\'s normal isolation guarantees, and the principle that dependencies are immutable and inert. We\'re not actually doing something unsafe or modifying dependencies, but we are allowing arbitrary ad-hoc behavior. For this reason, setup-hooks should only be used as a last resort.
This is strictly more general than any of the other mechanisms introduced in this chapter. For example, try writing a setup hook that has the same effect as a *propagatedBuildInputs* entry. One can almost think of this as an escape hatch around Nix's normal isolation guarantees, and the principle that dependencies are immutable and inert. We're not actually doing something unsafe or modifying dependencies, but we are allowing arbitrary ad-hoc behavior. For this reason, setup-hooks should only be used as a last resort.
## Environment Hooks
As a final convenience, we have environment hooks. Recall in [Pill 12](12-inputs-design-pattern.md) how we created `NIX_CFLAGS_COMPILE` for `-I` flags and `NIX_LDFLAGS` for `-L` flags, in a similar manner to how we prepared the `PATH`. One point of ugliness was how anti-modular this was. It makes sense to build the `PATH` in a generic builder, because the `PATH` is used by the shell, and the generic builder is intrinsically tied to the shell. But `-I` and `-L` flags are only relevant to the C compiler. The stdenv isn\'t wedded to including a C compiler (though it does by default), and there are other compilers too which may take completely different flags.
As a final convenience, we have environment hooks. Recall in [Pill 12](12-inputs-design-pattern.md) how we created `NIX_CFLAGS_COMPILE` for `-I` flags and `NIX_LDFLAGS` for `-L` flags, in a similar manner to how we prepared the `PATH`. One point of ugliness was how anti-modular this was. It makes sense to build the `PATH` in a generic builder, because the `PATH` is used by the shell, and the generic builder is intrinsically tied to the shell. But `-I` and `-L` flags are only relevant to the C compiler. The stdenv isn't wedded to including a C compiler (though it does by default), and there are other compilers too which may take completely different flags.
As a first step, we can move that logic to a setup hook on the C compiler; indeed that\'s just what we do in CC Wrapper. [^2] But this pattern comes up fairly often, so somebody decided to add some helper support to reduce boilerplate.
As a first step, we can move that logic to a setup hook on the C compiler; indeed that's just what we do in CC Wrapper. [^2] But this pattern comes up fairly often, so somebody decided to add some helper support to reduce boilerplate.
The other half of `addToEnv` is:
@ -246,8 +246,8 @@ and if one dependency has that setup hook then all of them will be so `echo`ed.
## Next pill\...
\...I\'m not sure! We could talk about the additional dependency types and hooks which cross compilation necessitates, building on our knowledge here to cover stdenv as it works today. We could talk about how nixpkgs is bootstrapped. Or we could talk about how `localSystem` and `crossSystem` are elaborated into the `buildPlatform`, `hostPlatform`, and `targetPlatform` each bootstrapping stage receives. Let us know which most interests you!
\...I'm not sure! We could talk about the additional dependency types and hooks which cross compilation necessitates, building on our knowledge here to cover stdenv as it works today. We could talk about how nixpkgs is bootstrapped. Or we could talk about how `localSystem` and `crossSystem` are elaborated into the `buildPlatform`, `hostPlatform`, and `targetPlatform` each bootstrapping stage receives. Let us know which most interests you!
[^1]: We can now be precise and consider what `addToEnv` does alone the minimal treatment of a dependency: i.e. a package that is *just* a dependency would *only* have `addToEnv` applied to it.
[^2]: It was called [GCC Wrapper](https://github.com/NixOS/nixpkgs/tree/6675f0a52c0962042a1000c7f20e887d0d26ae25/pkgs/build-support/gcc-wrapper) in the version of nixpkgs suggested for following along in this pill; Darwin and Clang support hadn\'t yet motivated the rename.
[^2]: It was called [GCC Wrapper](https://github.com/NixOS/nixpkgs/tree/6675f0a52c0962042a1000c7f20e887d0d26ae25/pkgs/build-support/gcc-wrapper) in the version of nixpkgs suggested for following along in this pill; Darwin and Clang support hadn't yet motivated the rename.