c3d2/nix-config
c3d2
/
nix-config
24
7
Fork 3
Code Issues 12 Pull Requests 2 Releases Wiki Activity

Update README

This commit is contained in:
Sandro - 2022-12-22 01:25:02 +01:00
parent b9a827fa31
commit a9e71931ba
Signed by: sandro
GPG Key ID: 3AF5A43A3EECC2E5
1 changed files with 77 additions and 118 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

195
README.md
View File

@ -5,81 +5,53 @@ include_toc: yes
lang: en
---
# Setup
# C3D2 infrastructure based on NixOS
## Add this repo to your local Nix registry
## Setup
As an alternative to a local checkout, always pull the latest code
from this repo.
## Enable nix flakes user wide
```bash
nix registry add c3d2 git+https://gitea.c3d2.de/C3D2/nix-config
```
This enables `nix` commands to find this Flake given the `c3d2#`
prefix in some arguments.
## Working with this repo
If you checked out this git repository for working on the code,
replace `c3d2#` with `.#` and run commands from the repository root.
Don't forget to `git add` new files! Flakes require that.
## The secrets repo
Make sure you have access.
## Install Nix Flakes
> Nix Flakes ist gegenwärtig bei Nix (Version 20.09) noch keine standardmäßige Funktionalität für Nix. Die Bereitstellung der Kommandos für Nix Flakes müssen als experimentelle Funktionalität für das Kommando ''nix'' festgelegt werden, um sie verfügbar zu machen.
Set some configuration (do this only once):
Add the setting to the user nix.conf. Only do this once!
```bash
echo 'experimental-features = nix-command flakes' >> ~/.config/nix/nix.conf
```
### Permanent System with Nix Flakes
## Enable nix flakes system wide (preferred for NixOS)
set this to your NixOS configuration:
add this to your NixOS configuration:
```nix
{ pkgs, ... }: {
nix = {
extraOptions = "experimental-features = nix-command flakes";
};
}
nix.settings.experimental-features = "nix-command flakes";
```
# Deployment
### The secrets repo
## Deploy a NixOS system from this Flake locally
is deprecated. Everything should be done through sops.
If you don't have secrets access ask sandro or astro to get onboarded.
Running `nixos-rebuild --flake c3d2 switch` on a machine should be sufficient
to update that machine to the current configuration and Nixpkgs revision.
## Deployment
## Deploy to a remote NixOS system with this Flake
### Deploy to a remote NixOS system
For every host that has a `nixosConfiguration` in our Flake, there are
two scripts that can be run for deployment via ssh.
For every host that has a `nixosConfiguration` in our Flake, there are two scripts that can be run for deployment via ssh.
- `nix run .#glotzbert-nixos-rebuild switch`
- `nix run .#HOSTNAME-nixos-rebuild switch`
Copies the current state to build on the target system. This may
fail due to eg. container resource limits.
Copies the current state to build on the target system.
This may fail due to resource limits on eg. Raspberry Pis.
- `nix run .#HOSTNAME-nixos-rebuild-local switch`
Builds everything locally, then uses `nix copy` to transfer the new NixOS system to the target.
The target must already be a nixFlakes system.
To use the cache from hydra set the following nix options similar to enabling flakes:
- `nix run .#glotzbert-nixos-rebuild-local switch`
```
trusted-public-keys = nix-serve.hq.c3d2.de:KZRGGnwOYzys6pxgM8jlur36RmkJQ/y8y62e52fj1ps=
trusted-substituters = https://nix-serve.hq.c3d2.de
```
Builds locally, then uses `nix copy` to transfer the new NixOS
system to the target.
**Help!** It's needlessly rebuilding stuff that already runs on the
target? If so, use `nix copy` to transfer where
`/run/current-system` points to to your build machine.
## Checking for updates
### Checking for updates
```shell
nix run .#list-upgradable
@ -89,7 +61,7 @@ nix run .#list-upgradable
Checks all hosts with a `nixosConfiguration` in `flake.nix`.
## Update from [Hydra build](https://hydra.hq.c3d2.de/jobset/c3d2/nix-config#tabs-jobs)
### Update from [Hydra build](https://hydra.hq.c3d2.de/jobset/c3d2/nix-config#tabs-jobs)
The fastest way to update a system, a manual alternative to setting
`c3d2.autoUpdate = true;`
@ -100,21 +72,21 @@ Just run:
update-from-hydra
```
## Deploy a MicroVM
### Deploy a MicroVM
### Building spaceapi remotely, and deploy
#### Build a microvm remotely and deploy
```shell
nix run .#microvm-update-spaceapi
nix run .#microvm-update-HOSTNAME
```
### Building spaceapi locally, and deploy
#### Build microvm locally and deploy
```shell
nix run .#microvm-update-spaceapi-local
nix run .#microvm-update-HOSTNAME-local
```
### Update MicroVM from our Hydra
#### Update MicroVM from our Hydra
Our Hydra runs `nix flake update` daily in the `updater.timer`,
pushing it to the `flake-update` branch so that it can build fresh
@ -125,9 +97,9 @@ so the following is all that is needed on a MicroVM-hosting server:
microvm -Ru $hostname
```
# Cluster deployment with Skyflake
## Cluster deployment with Skyflake
## About
### About
[Skyflake](https://github.com/astro/skyflake) provides Hyperconverged
Infrastructure to run NixOS MicroVMs on a cluster. Our setup unifies
@ -137,7 +109,7 @@ with Glusterfs.
Recognize nixosConfiguration for our Skyflake deployment by the
`self.nixosModules.cluster-options` module being included.
## User interface
### User interface
We use the less-privileged `c3d2@` user for deployment. This flake's
name on the cluster is `config`. Other flakes can coexist in the same
@ -145,7 +117,7 @@ user so that we can run separately developed projects like
*dump-dvb*. *leon* and potentially other users can deploy Flakes and
MicroVMs without name clashes.
### Deploying
#### Deploying
**git push** this repo to any machine in the cluster, preferably to
Hydra because there building won't disturb any services.
@ -170,19 +142,19 @@ This will:
*Deleting* a nixosConfiguration's branch will **stop** the MicroVM in Nomad.
### Updating
#### Updating
**TODO:** how would you like it?
### MicroVM status
#### MicroVM status
```bash
ssh c3d2@hydra.serv.zentralwerk.org status
```
## Debugging for cluster admins
### Debugging for cluster admins
### Glusterfs
#### Glusterfs
Glusterfs holds our MicroVMs' state. They *must always be mounted* or
brains are split.
@ -192,22 +164,22 @@ gluster volume info
gluster volume status
```
#### Restart glusterd
##### Restart glusterd
```bash
systemctl restart glusterd
```
#### Remount volumes
##### Remount volumes
```bash
systemctl restart /glusterfs/fast
systemctl restart /glusterfs/big
```
### Nomad
#### Nomad
#### Check the cluster state
##### Check the cluster state
```shell
nomad server members
@ -217,13 +189,13 @@ Nomad *servers* **coordinate** the cluster.
Nomad *clients* **run** the tasks.
#### Browse in the terminal
##### Browse in the terminal
[wander](https://github.com/robinovitch61/wander) and
[damon](https://github.com/hashicorp/damon) are nice TUIs that are
preinstalled on our cluster nodes.
#### Browse with a browser
##### Browse with a browser
First, tunnel TCP port `:4646` from a cluster server:
@ -233,7 +205,7 @@ ssh -L 4646:localhost:4646 root@server10.cluster.zentralwerk.org
Then, visit https://localhost:4646 for for full klickibunti.
#### Reset the Nomad state on a node
##### Reset the Nomad state on a node
After upgrades, Nomad servers may fail rejoining the cluster. Do this
to make a *Nomad server* behave like a newborn:
@ -244,9 +216,32 @@ rm -rf /var/lib/nomad/server/raft/
systemctl start nomad
```
# Secrets management
## Secrets management
## Secrets managment with PGP
### Secrets Management Using `sops-nix`
#### Adding a new host
Edit `.sops.yaml`:
1. Add an AGE key for this host. Comments in this file tell you how to do it.
2. Add a `creation_rules` section for `host/$host/*.yaml` files
#### Editing a hosts secrets
Edit `.sops.yaml` to add files for a new host and its SSH pubkey.
```bash
# Get sops
nix develop
# Decrypt, start en EDITOR, encrypt
sops hosts/.../secrets.yaml
# Push
git commit -a -m Adding new secrets
git push origin
```
### Secrets management with PGP
Add your gpg-id to the .gpg-id file in secrets and let somebody reencrypt it for you.
Maybe this works for you, maybe not. I did it somehow:
@ -258,39 +253,7 @@ PASSWORD_STORE_DIR=`pwd` tr '\n' ' ' < .gpg-id | xargs -I{} pass init {}
Your gpg key has to have the Authenticate flag set. If not update it and push it to a keyserver and wait.
This is necessary, so you can login to any machine with your gpg key.
## Secrets Management Using `sops-nix`
### Adding a new host
Edit `secrets/.sops.yaml`:
1. Add an AGE key for this host. Comments in this file tell you how to
do it.
2. Add a `creation_rules` section for `host/$host/*yaml` files
### Editing a hosts secrets
Edit `secrets/.sops.yaml` to add files for a new host and its SSH pubkey.
```bash
# Enter the secrets flake
cd secrets
# Get sops
nix develop
# Decrypt, start en EDITOR, encrypt
sops hosts/.../secrets.yaml
# Push
git commit -a -m YOLO
git push origin HEAD:master
# Go back to this flake
cd ..
# Update flake.lock file
nix flake lock . --update-input secrets
```
# Laptops / Desktops
Note: currently being rebuild!
## Laptops / Desktops
This repository contains a NixOS module that can be used with personal machines
as well. This module appends `/etc/ssh/ssh_known_hosts` with the host keys of
@ -302,19 +265,15 @@ an example:
# /etc/nixos/configuration.nix
{ config, pkgs, lib, ... }:
let
c3d2Config =
builtins.fetchGit { url = "https://gitea.c3d2.de/C3D2/nix-config.git"; };
# Using a flake is recommended instead
c3d2Config = builtins.fetchGit { url = "https://gitea.c3d2.de/C3D2/nix-config.git"; };
in {
imports = [
# ...
"${c3d2Config}/modules/c3d2.nix"
];
c3d2 = {
enableMotd = true; # Set the login shell message to the <<</>> logo.
...
};
# ...
}
```