Guix Deploy failed to update my guix server — GNUcode.me

Guix Deploy failed to update my guix server

by Joshua Branson — May 08, 2024

tl;dr You may run into trouble updating Guix System, if you rarely update it. You may also discover that using guix deploy to update your server will silently fail and not tell you. When you use guix deploy…you may want to do something like this:

joshua@laptopT400 $ guix deploy gnucode.me-current-config.scm -x -- \
                    guix describe && guix system describe; \
                    sudo herd restart nginx;

On with the blog post!

So my server that runs https://gnucode.me is a guix system linode server!  Woo hoo! Well lately the server is being a right pain, and it refuses to update. Most likely my server is lonely and just wants a hug. Let’s see how outdated the server is eh?

joshua@gnucode.me $ guix system describe

Generation 20   Aug 17 2023 07:45:06    (current)
  file name: /var/guix/profiles/system-20-link
  canonical file name: /gnu/store/g10d07ab2hfspvj0wglwlicmphi9ly07-system
  label: GNU with Linux-Libre 6.3.13
  bootloader: grub
  root device: /dev/sda
  kernel: /gnu/store/224j20diq0qgxqzmh1hgsp35zi3qn9jc-linux-libre-6.3.13/bzImage
  channels:
    guix:
      repository URL: https://git.savannah.gnu.org/git/guix.git
      branch: master
      commit: 210bbf9c59355218fec1484a686cc2ecdc218506
    guixrus:
      repository URL: https://git.sr.ht/~whereiseveryone/guixrus
      branch: master
      commit: d70a9ea65eba43aba3ec7560d797452c01171b85
    nonguix:
      repository URL: https://gitlab.com/nonguix/nonguix
      branch: master
      commit: 1a3082c574077aaf6771ff927e725fc390a2d531
  configuration file: /gnu/store/8xi1rqijhlafnrvzrgla7k3qlkl9jsfw-configuration.scm

joshua@gnucode.me $ guix describe

Generation 2    Aug 17 2023 07:11:45    (current)
  guix 1b2d43f
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: 1b2d43fe016848ea2ec16ff18cbc14340944fc4e

Yikes! My guix system server is 6 or 7 months outdated! That’s not good! Let’s fix that!

I have used guix deploy in the past to update my linode server, which works super awesomely well! But lately, guix deploy no longer updates my linode server. Let’s go ahead and try to update my linode server via guix deploy, just to ensure that guix deploy is not working. :)

joshua@laptopT400 $ guix deploy gnucode.me-current-config.scm -x -- sudo herd restart nginx
guix deploy: sending 12 store items (133 MiB) to '45.56.66.20'...
guix deploy: copertino: command succeeded
guix deploy: command output on copertino:
Service user-homes has been started.
Service nginx has been started.

Let’s see if gnucode.me has been updated:

joshua@gnucode.me $ guix describe
                    Aug 2023  # (the same date as before)
joshua@gnucode.me $ guix system describe
                    Aug 2023  # (the same date as before)

So it appears that guix deploy is not updating my server. I’ve been guix deploying my server for a while now, and I didn’t even realize that the server was failing to update (because my services all stayed the same between updates).

Well, I recently fixed a similar issue, where my laptop was really outdated and guix pull failed. To update my laptop I built guix from source (I even made a bug report for it). So I might end up rebuilding guix on my server from source too.

Well, since guix deploy failed to update my linode server, let’s try updating with guix pull. First I powered down the linode and updated the vm so that it had 4GB of RAM (guix really needs at least 2GB of RAM to function well).

Anyway, at 2pm, I ran guix pull. It started building stuff, and I let it sit.

joshua@gnucode.me $ guix pull
  # more output above
building /gnu/store/lb5b7svdmfj1ijnzrripsjcv0bhqzpwb-Python-3.5.9.tar.xz.drv...
building /gnu/store/z8kqgsx6yh2mpm0n2jhzk1phihdscazf-datefudge_1.23.tar.xz.drv...
building /gnu/store/9cl9w041lwjz607f5l6pzjzgixdl1jjm-Python-3.5.9.tar.xz.drv...
building /gnu/store/8r73jwy621pvmbfyk34w0gqf65sff9di-elfutils-0.187.tar.bz2.drv...
building /gnu/store/0innk69jqi1zadpwyi8p9szdl95l2lk0-expat-2.5.0.tar.xz.drv...
building /gnu/store/3zysg963rlp01rpkjwsi8qzfr84kwd9d-gcc-11.3.0.tar.xz.drv...
building /gnu/store/lpkc0wsvgbfz0qwxn1nckh88n4xrg1kg-gettext-0.19.8.1.tar.gz.drv...
building /gnu/store/g6qb58plai65zdps620ny2rfif1z42s0-gettext-0.21.tar.gz.drv...
building /gnu/store/dc2y0rldz6wadknqz4jl6zjb7l4xsay5-glibc-2.35.tar.xz.drv...
building /gnu/store/gb6lfdzqprh1fmw91kpkclywg8sbrwg3-glibc-mesboot0-2.2.5.drv...
/ 'install' phase^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[A^[[|

building /gnu/store/5s4cfyc3w6jn958jn1i09g6my4g7l9l4-glibc-2.35.tar.xz.drv...
building /gnu/store/qm63mzpmzriksa9vi0gl977h7qhcmgkg-gcc-mesboot0-2.95.3.drv...
building /gnu/store/1hrskysmxkyblhw8mpkkz1m2jn4g0f8s-gmp-6.0.0a.tar.xz.drv...
building /gnu/store/cfbs6ysgyrqwfgbkd15gkbf7brgfgvk3-binutils-mesboot1-2.20.1a.drv...
building /gnu/store/b4xy58rqb3cqw2wiip8l0ibf9crk13zd-gmp-6.0.0a.tar.xz.drv...
building /gnu/store/1vp4cjnwxr55basnbjr4vfd5l5pmf41c-gnutls-3.8.3.tar.xz.drv...
building /gnu/store/j9vzm9yrkzfsgjpliab737fllq6kkm7q-guile-lib-0.2.8.tar.gz.drv...
building /gnu/store/wh7yqmqk07p7xby179razdbp6ysgqygl-isl-0.24.tar.xz.drv...
building /gnu/store/c3ds6y32xq9xck076apm525qqvp649ia-libffi-3.4.4.tar.gz.drv...
building /gnu/store/nn8hm6242hfwv906v1dy2m7sq3yv3hsh-libunistring-1.0.tar.xz.drv...
building /gnu/store/05nznjzl8qfsdps7wd51852babfvrndw-m4-1.4.19.tar.xz.drv...
building /gnu/store/6hqxh7hlcarn7l3y32pn4l3gkjbpkjia-make-mesboot-3.82.drv...
building /gnu/store/jxg62mw1619fznmnq3b7if5nv08s5q1i-module-import-compiled.drv...
building /gnu/store/gmw2a5zxc6liilr8jwpdw51isfrbrkis-gcc-mesboot1-4.6.4.drv...
- 'build' phas-
\
\ 'build' phas/^C

I started the guix pull command at 2pm. At 5pm…the server was still trying to build gcc-mesboot0. That sounds like it’s trying to build guix from source, which will take a long time. I like my server, and I don’t want to make it that busy.

Let’s check guix weather. Maybe I don’t have a lot of substitutes available.

joshua@gnucode.me $ guix weather
https://ci.guix.gnu.org ☀
  88.0% substitutes available (25,358 out of 28,805)
  at least 102,748.6 MiB of nars (compressed)
  177,798.9 MiB on disk (uncompressed)
  0.009 seconds per request (32.5 seconds in total)
  106.0 requests per second

  0.0% (0 out of 3,447) of the missing items are queued
  at least 1,000 queued builds
      aarch64-linux: 974 (97.4%)
      i686-linux: 14 (1.4%)
      x86_64-linux: 11 (1.1%)
      powerpc64le-linux: 1 (.1%)
  build rate: 22.87 builds per hour
      powerpc64le-linux: 2.10 builds per hour
      i686-linux: 10.75 builds per hour
      x86_64-linux: 9.42 builds per hour
      aarch64-linux: 0.53 builds per hour
      armhf-linux: 0.03 builds per hour
      i586-gnu: 0.05 builds per hour
looking for 28,805 store items on https://bordeaux.guix.gnu.org...
https://bordeaux.guix.gnu.org ☀
  96.9% substitutes available (27,923 out of 28,805)
  at least 72,788.8 MiB of nars (compressed)
  216,904.7 MiB on disk (uncompressed)
  0.005 seconds per request (135.6 seconds in total)
  191.3 requests per second
  (continuous integration information unavailable)

Ok, I had plenty of substitutes available. It looks like guix pull is not really going to work.

Now that the “easy” way to update my server has failed (via guix pull), let’s try to update it by building guix from source! Since my linode server now has 4 GB of RAM, this should work.

joshua@gnucode.me $ guix package --delete-generations 
joshua@gnucode.me $ sudo guix system delete-generations
joshua@gnucode.me $ guix gc 
joshua@gnucode.me $ guix shell -D guix help2man git strace -CPW
joshua@gnucode.me $ ./bootstrap
joshua@gnucode.me $ ./configure --localstatedir=/var --sysconfdir=/etc
joshua@gnucode.me $ make
joshua@gnucode.me $ exit  # you have to leave the container environment, or
                          # your substitutes will not work.
joshua@gnucode.me $ sudo -E ./pre-inst-env guix system reconfigure \
                    ../guix-config/linode-guix-system-configuration/gnucode.me-current-config.scm

The above command failed a few times, because I had to tweak the gnucode.me-current-config.scm file. First I had to change dovecot-service to dovecot-service-type. Then I had to get rid of the deploy stuff in the file. Then I was able to rerun the command and it started to work:

Woo hoo! It worked!

And a new guix pull invocation also worked!

I should also probably create a gnucode.me-current-config-deploy.scm file. I should also probably email bug-guix@gnu.org to have guix deploy output the results of guix describe and guix system describe. Or perhaps before guix deploy deploys the update, it should record the results of guix describe and guix system describe. If one or both of those are the same after guix deploy, then it should issue a warning to the user. Until next time!