Go to file
Sandro - 2621bd671f
web: add mastodon webfinger
2022-12-02 02:11:16 +01:00
doc README: update list-upgradable documentation 2021-09-08 01:45:28 +02:00
hosts web: add mastodon webfinger 2022-12-02 02:11:16 +01:00
keys Move sops secrets into nix-config, move hosts under hosts/containers to hosts 2022-07-31 17:52:24 +02:00
modules modules/cluster/deployment: add operator to needForSpeed nomad affinity 2022-11-28 01:16:54 +01:00
overlay overlay: remove tracer-game for now 2022-09-28 23:59:34 +02:00
.git-blame-ignore-revs Ignore formatting in blames 2022-06-12 17:27:07 +02:00
.gitignore Add "result" to .gitignore 2019-11-29 14:21:52 +01:00
.sops.yaml mastodon: enable ldap 2022-11-30 02:28:56 +01:00
README.md README: doc skyflake deployment 2022-11-30 21:00:36 +01:00
deploy-flake.sh deploy-flake: always use target hostname for safety 2022-01-07 00:15:54 +01:00
flake.lock Switch to nixpkgs stable fork for cherry-picked fixes 2022-11-30 22:49:55 +01:00
flake.nix Switch to nixpkgs stable fork for cherry-picked fixes 2022-11-30 22:49:55 +01:00
host-registry.nix host-registry: add dacbert, pulsebert ip4 addresses 2022-09-14 00:19:19 +02:00
users.nix leon, leoncloud: reuse sshKeys 2022-11-21 03:46:11 +01:00



Add this repo to your local Nix registry

As an alternative to a local checkout, always pull the latest code from this repo.

nix registry add c3d2 git+https://gitea.c3d2.de/C3D2/nix-config

This enables nix commands to find this Flake given the c3d2# prefix in some arguments.

Working with this repo

If you checked out this git repository for working on the code, replace c3d2# with .# and run commands from the repository root.

Don't forget to git add new files! Flakes require that.

The secrets repo

Make sure you have access.

Install Nix Flakes

Nix Flakes ist gegenwärtig bei Nix (Version 20.09) noch keine standardmäßige Funktionalität für Nix. Die Bereitstellung der Kommandos für Nix Flakes müssen als experimentelle Funktionalität für das Kommando ''nix'' festgelegt werden, um sie verfügbar zu machen.

Set some configuration (do this only once):

echo 'experimental-features = nix-command flakes' >> ~/.config/nix/nix.conf

Permanent System with Nix Flakes

set this to your NixOS configuration:

{ pkgs, ... }: {
  nix = {
    extraOptions = "experimental-features = nix-command flakes";


Deploy a NixOS system from this Flake locally

Running nixos-rebuild --flake c3d2 switch on a machine should be sufficient to update that machine to the current configuration and Nixpkgs revision.

Deploy to a remote NixOS system with this Flake

For every host that has a nixosConfiguration in our Flake, there are two scripts that can be run for deployment via ssh.

  • nix run .#glotzbert-nixos-rebuild switch

    Copies the current state to build on the target system. This may fail due to eg. container resource limits.

    The target must already be a nixFlakes system.

  • nix run .#glotzbert-nixos-rebuild-local switch

    Builds locally, then uses nix copy to transfer the new NixOS system to the target.

    Help! It's needlessly rebuilding stuff that already runs on the target? If so, use nix copy to transfer where /run/current-system points to to your build machine.

Remote deployment from non-NixOS

A shell script that copies the current working tree, and runs nixos-rebuild switch on the target:

./deploy-flake.sh hydra.hq.c3d2.de

It cannot not lookup hostnames in host-registry.nix. To avoid deploying the wrong container on the unrelated DNS records, the script always uses the hostname that is already configured on the target system.

Checking for updates

nix run .#list-upgradable

list-upgradable output

Checks all hosts with a nixosConfiguration in flake.nix.

Update from Hydra build

The fastest way to update a system, a manual alternative to setting c3d2.autoUpdate = true;

Just run:


Deploy a MicroVM

Building spaceapi remotely, and deploy

nix run .#microvm-update-spaceapi

Building spaceapi locally, and deploy

nix run .#microvm-update-spaceapi-local

Update MicroVM from our Hydra

Our Hydra runs nix flake update daily in the updater.timer, pushing it to the flake-update branch so that it can build fresh systems. This branch is setup as the source flake in all the MicroVMs, so the following is all that is needed on a MicroVM-hosting server:

microvm -Ru $hostname

High Availability Deployment on Nomad

First, stop and delete /var/lib/microvm/$NAME where the systemd-managed MicroVMs live, or move the state to /glusterfs/fast/microvms/$NAME.

nix run .#nomad-$NAME

Cluster deployment with Skyflake


Skyflake provides Hyperconverged Infrastructure to run NixOS MicroVMs on a cluster. Our setup unifies networking with one bridge per VLAN. Persistent storage is replicated with Glusterfs.

Recognize MicroVMs for Skyflake by modules containing self.nixosModules.cluster-options.


Push our repo to any machine on the cluster, preferably to Hydra because it just builds stuff and probably has most packages already in store.

You don't deploy all MicroVMs at once. Instead, Skyflake allows you to select NixOS systems by the branches you push to.

Example: deploy hosts mucbot and sdrweb

git push c3d2@hydra.serv.zentralwerk.org:config HEAD:mucbot HEAD:sdrweb



gluster volume info
gluster volume status

Restart glusterd

systemctl restart glusterd

Remount volumes

systemctl restart /glusterfs/fast
systemctl restart /glusterfs/big


Check the cluster state

nomad server members

Browse in the terminal

Use wander and damon

Browse with a browser

First, tunnel TCP port :4646 from a cluster server:

ssh -L 4646:localhost:4646 root@server10.cluster.zentralwerk.org

Then, visit https://localhost:4646 for for full klickibunti.

Reset the Nomad state on a node

After upgrades, Nomad servers may fail rejoining the cluster. Do:

systemctl stop nomad
rm -rf /var/lib/nomad/server/raft/
systemctl start nomad

Secrets management

Secrets managment with PGP

Add your gpg-id to the .gpg-id file in secrets and let somebody reencrypt it for you. Maybe this works for you, maybe not. I did it somehow:

PASSWORD_STORE_DIR=`pwd` tr '\n' ' ' < .gpg-id | xargs -I{} pass init {}

Your gpg key has to have the Authenticate flag set. If not update it and push it to a keyserver and wait. This is necessary, so you can login to any machine with your gpg key.

Secrets Management Using sops-nix

Adding a new host

Edit secrets/.sops.yaml:

  1. Add an AGE key for this host. Comments in this file tell you how to do it.
  2. Add a creation_rules section for host/$host/*yaml files

Editing a hosts secrets

Edit secrets/.sops.yaml to add files for a new host and its SSH pubkey.

# Enter the secrets flake
cd secrets
# Get sops
nix develop
# Decrypt, start en EDITOR, encrypt
sops hosts/.../secrets.yaml
# Push
git commit -a -m YOLO
git push origin HEAD:master
# Go back to this flake
cd ..
# Update flake.lock file
nix flake lock . --update-input secrets

Laptops / Desktops

This repository contains a NixOS module that can be used with personal machines as well. This module appends /etc/ssh/ssh_known_hosts with the host keys of registered HQ hosts, and optionally appends /etc/hosts with static IPv6 addresses local to HQ. Simply import the lib directory to use the module. As an example:

# /etc/nixos/configuration.nix
{ config, pkgs, lib, ... }:
  c3d2Config =
    builtins.fetchGit { url = "https://gitea.c3d2.de/C3D2/nix-config.git"; };
in {
  imports = [
    # ...

  c3d2 = {
    isInHq = false; # not in HQ, this is the default.
    mergeHostsFile = true; # Make entries in /etc/hosts form hosts.nix
    enableMotd = true; # Set the login shell message to the <<</>> logo.

  # ...