From a9e71931ba2141cf942c0dd0a772c9072b5aa460 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Sandro=20J=C3=A4ckel?= Date: Thu, 22 Dec 2022 01:25:02 +0100 Subject: [PATCH] Update README --- README.md | 195 +++++++++++++++++++++--------------------------------- 1 file changed, 77 insertions(+), 118 deletions(-) diff --git a/README.md b/README.md index 315c7e8e..58d2d41f 100644 --- a/README.md +++ b/README.md @@ -5,81 +5,53 @@ include_toc: yes lang: en --- -# Setup +# C3D2 infrastructure based on NixOS -## Add this repo to your local Nix registry +## Setup -As an alternative to a local checkout, always pull the latest code -from this repo. +## Enable nix flakes user wide -```bash -nix registry add c3d2 git+https://gitea.c3d2.de/C3D2/nix-config -``` - -This enables `nix` commands to find this Flake given the `c3d2#` -prefix in some arguments. - -## Working with this repo - -If you checked out this git repository for working on the code, -replace `c3d2#` with `.#` and run commands from the repository root. - -Don't forget to `git add` new files! Flakes require that. - -## The secrets repo - -Make sure you have access. - -## Install Nix Flakes - -> Nix Flakes ist gegenwärtig bei Nix (Version 20.09) noch keine standardmäßige Funktionalität für Nix. Die Bereitstellung der Kommandos für Nix Flakes müssen als experimentelle Funktionalität für das Kommando ''nix'' festgelegt werden, um sie verfügbar zu machen. - -Set some configuration (do this only once): +Add the setting to the user nix.conf. Only do this once! ```bash echo 'experimental-features = nix-command flakes' >> ~/.config/nix/nix.conf ``` -### Permanent System with Nix Flakes +## Enable nix flakes system wide (preferred for NixOS) -set this to your NixOS configuration: +add this to your NixOS configuration: ```nix -{ pkgs, ... }: { - nix = { - extraOptions = "experimental-features = nix-command flakes"; - }; -} +nix.settings.experimental-features = "nix-command flakes"; ``` -# Deployment +### The secrets repo -## Deploy a NixOS system from this Flake locally +is deprecated. Everything should be done through sops. +If you don't have secrets access ask sandro or astro to get onboarded. -Running `nixos-rebuild --flake c3d2 switch` on a machine should be sufficient -to update that machine to the current configuration and Nixpkgs revision. +## Deployment -## Deploy to a remote NixOS system with this Flake +### Deploy to a remote NixOS system -For every host that has a `nixosConfiguration` in our Flake, there are -two scripts that can be run for deployment via ssh. +For every host that has a `nixosConfiguration` in our Flake, there are two scripts that can be run for deployment via ssh. -- `nix run .#glotzbert-nixos-rebuild switch` +- `nix run .#HOSTNAME-nixos-rebuild switch` - Copies the current state to build on the target system. This may - fail due to eg. container resource limits. + Copies the current state to build on the target system. + This may fail due to resource limits on eg. Raspberry Pis. + +- `nix run .#HOSTNAME-nixos-rebuild-local switch` + + Builds everything locally, then uses `nix copy` to transfer the new NixOS system to the target. - The target must already be a nixFlakes system. + To use the cache from hydra set the following nix options similar to enabling flakes: -- `nix run .#glotzbert-nixos-rebuild-local switch` + ``` + trusted-public-keys = nix-serve.hq.c3d2.de:KZRGGnwOYzys6pxgM8jlur36RmkJQ/y8y62e52fj1ps= + trusted-substituters = https://nix-serve.hq.c3d2.de + ``` - Builds locally, then uses `nix copy` to transfer the new NixOS - system to the target. - - **Help!** It's needlessly rebuilding stuff that already runs on the - target? If so, use `nix copy` to transfer where - `/run/current-system` points to to your build machine. - -## Checking for updates +### Checking for updates ```shell nix run .#list-upgradable @@ -89,7 +61,7 @@ nix run .#list-upgradable Checks all hosts with a `nixosConfiguration` in `flake.nix`. -## Update from [Hydra build](https://hydra.hq.c3d2.de/jobset/c3d2/nix-config#tabs-jobs) +### Update from [Hydra build](https://hydra.hq.c3d2.de/jobset/c3d2/nix-config#tabs-jobs) The fastest way to update a system, a manual alternative to setting `c3d2.autoUpdate = true;` @@ -100,21 +72,21 @@ Just run: update-from-hydra ``` -## Deploy a MicroVM +### Deploy a MicroVM -### Building spaceapi remotely, and deploy +#### Build a microvm remotely and deploy ```shell -nix run .#microvm-update-spaceapi +nix run .#microvm-update-HOSTNAME ``` -### Building spaceapi locally, and deploy +#### Build microvm locally and deploy ```shell -nix run .#microvm-update-spaceapi-local +nix run .#microvm-update-HOSTNAME-local ``` -### Update MicroVM from our Hydra +#### Update MicroVM from our Hydra Our Hydra runs `nix flake update` daily in the `updater.timer`, pushing it to the `flake-update` branch so that it can build fresh @@ -125,9 +97,9 @@ so the following is all that is needed on a MicroVM-hosting server: microvm -Ru $hostname ``` -# Cluster deployment with Skyflake +## Cluster deployment with Skyflake -## About +### About [Skyflake](https://github.com/astro/skyflake) provides Hyperconverged Infrastructure to run NixOS MicroVMs on a cluster. Our setup unifies @@ -137,7 +109,7 @@ with Glusterfs. Recognize nixosConfiguration for our Skyflake deployment by the `self.nixosModules.cluster-options` module being included. -## User interface +### User interface We use the less-privileged `c3d2@` user for deployment. This flake's name on the cluster is `config`. Other flakes can coexist in the same @@ -145,7 +117,7 @@ user so that we can run separately developed projects like *dump-dvb*. *leon* and potentially other users can deploy Flakes and MicroVMs without name clashes. -### Deploying +#### Deploying **git push** this repo to any machine in the cluster, preferably to Hydra because there building won't disturb any services. @@ -170,19 +142,19 @@ This will: *Deleting* a nixosConfiguration's branch will **stop** the MicroVM in Nomad. -### Updating +#### Updating **TODO:** how would you like it? -### MicroVM status +#### MicroVM status ```bash ssh c3d2@hydra.serv.zentralwerk.org status ``` -## Debugging for cluster admins +### Debugging for cluster admins -### Glusterfs +#### Glusterfs Glusterfs holds our MicroVMs' state. They *must always be mounted* or brains are split. @@ -192,22 +164,22 @@ gluster volume info gluster volume status ``` -#### Restart glusterd +##### Restart glusterd ```bash systemctl restart glusterd ``` -#### Remount volumes +##### Remount volumes ```bash systemctl restart /glusterfs/fast systemctl restart /glusterfs/big ``` -### Nomad +#### Nomad -#### Check the cluster state +##### Check the cluster state ```shell nomad server members @@ -217,13 +189,13 @@ Nomad *servers* **coordinate** the cluster. Nomad *clients* **run** the tasks. -#### Browse in the terminal +##### Browse in the terminal [wander](https://github.com/robinovitch61/wander) and [damon](https://github.com/hashicorp/damon) are nice TUIs that are preinstalled on our cluster nodes. -#### Browse with a browser +##### Browse with a browser First, tunnel TCP port `:4646` from a cluster server: @@ -233,7 +205,7 @@ ssh -L 4646:localhost:4646 root@server10.cluster.zentralwerk.org Then, visit https://localhost:4646 for for full klickibunti. -#### Reset the Nomad state on a node +##### Reset the Nomad state on a node After upgrades, Nomad servers may fail rejoining the cluster. Do this to make a *Nomad server* behave like a newborn: @@ -244,9 +216,32 @@ rm -rf /var/lib/nomad/server/raft/ systemctl start nomad ``` -# Secrets management +## Secrets management -## Secrets managment with PGP +### Secrets Management Using `sops-nix` + +#### Adding a new host + +Edit `.sops.yaml`: + +1. Add an AGE key for this host. Comments in this file tell you how to do it. +2. Add a `creation_rules` section for `host/$host/*.yaml` files + +#### Editing a hosts secrets + +Edit `.sops.yaml` to add files for a new host and its SSH pubkey. + +```bash +# Get sops +nix develop +# Decrypt, start en EDITOR, encrypt +sops hosts/.../secrets.yaml +# Push +git commit -a -m Adding new secrets +git push origin +``` + +### Secrets management with PGP Add your gpg-id to the .gpg-id file in secrets and let somebody reencrypt it for you. Maybe this works for you, maybe not. I did it somehow: @@ -258,39 +253,7 @@ PASSWORD_STORE_DIR=`pwd` tr '\n' ' ' < .gpg-id | xargs -I{} pass init {} Your gpg key has to have the Authenticate flag set. If not update it and push it to a keyserver and wait. This is necessary, so you can login to any machine with your gpg key. -## Secrets Management Using `sops-nix` - -### Adding a new host - -Edit `secrets/.sops.yaml`: - -1. Add an AGE key for this host. Comments in this file tell you how to - do it. -2. Add a `creation_rules` section for `host/$host/*yaml` files - -### Editing a hosts secrets - -Edit `secrets/.sops.yaml` to add files for a new host and its SSH pubkey. - -```bash -# Enter the secrets flake -cd secrets -# Get sops -nix develop -# Decrypt, start en EDITOR, encrypt -sops hosts/.../secrets.yaml -# Push -git commit -a -m YOLO -git push origin HEAD:master -# Go back to this flake -cd .. -# Update flake.lock file -nix flake lock . --update-input secrets -``` - -# Laptops / Desktops - -Note: currently being rebuild! +## Laptops / Desktops This repository contains a NixOS module that can be used with personal machines as well. This module appends `/etc/ssh/ssh_known_hosts` with the host keys of @@ -302,19 +265,15 @@ an example: # /etc/nixos/configuration.nix { config, pkgs, lib, ... }: let - c3d2Config = - builtins.fetchGit { url = "https://gitea.c3d2.de/C3D2/nix-config.git"; }; + # Using a flake is recommended instead + c3d2Config = builtins.fetchGit { url = "https://gitea.c3d2.de/C3D2/nix-config.git"; }; in { imports = [ - # ... "${c3d2Config}/modules/c3d2.nix" ]; c3d2 = { - enableMotd = true; # Set the login shell message to the <<> logo. + ... }; - - # ... } - ```