SSH Jump Server once again

The previously presented idea of the jump server was ok, but it doesn’t scale well. The poor scaling is due to passing your public key somewhere like the git repository and around this repo you build your automation like ansible. What about a more self-support solution and a more “enterprise” one.

Now comes with help Vault - a tool that can manages your secrets. It contains a concept of secret engines a kind of plugin that extends vault. One of the plugins is SSH which lets you sign your SSH public key with CA and the SSH server can be instructed to authorize your public key with CA public key. Whenever you sign your public key you can pick extensions to it, so ie. you can sign a key that is only allowed to setup port-forwarding and nothing else. It’s not a vault feature but openssh one. Unfortunately, it’s not feature-rich as in “authorized_keys” file ie. lacks permitopen option (but can be applied later phase), the full list of options is here.

So how it can look like from a 1000-foot view:

• vault can authenticate users through OIDC (like Google, Okta)
• users have a policy bound to sign their SSH public keys with selected extensions and short TTL
• configure OpenSSH server to validate public keys with CA and optionally configure AuthorizedPrincipalsFile to get Principals from the signed public key and match it to the selected user, so you have a single principal and login as different users in different hosts
• all the stuff around Vault can be audited to get info on who is asking for sign public key

It’s already been written here how to setup things up, so don’t want to copy and paste, but I will mention a few steps:

• prepare role for port-forwarding only:

$ vault write ssh-client-signer/roles/my-role -<<"EOH"
{
  "algorithm_signer": "rsa-sha2-256",
  "allow_user_certificates": true,
  "allowed_users": "ubuntu",
  "allowed_extensions": "",
  "default_extensions": {
    "no-user-rc": "",
    "no-pty": "",
    "permit-port-forwarding": ""
  },
  "key_type": "ca",
  "default_user": "ubuntu",
  "ttl": "30m0s"
}

• user’s policy is being configured to not change any extension:

$ vault write -field=signed_key ssh-client-signer/sign/my-role -<<"EOH"
{
  "public_key": "<pubkey>",
  "valid_principals": "ubuntu",
  "extensions": {
    "permit-pty": ""
  }
}
EOH
...
* extensions [permit-pty] are not on allowed list

• AuthorizedPrincipalsFile has the same format as authorized_keys file, so you can further limit port-forwarding option with permitopen ie.

$ cat /etc/ssh/auth_principals/ubuntu
permitopen="1.2.3.4:22" ubuntu

• ssh client can automate requesting for sign public key ie.

$ cat ~/.ssh/config
...
Match host <host> exec "~/sign-key.sh > ~/.ssh/signed.pem"
    HostName <hostname>
    User ubuntu
    IdentityFile ~/.ssh/id_ed25519
    IdentityFile ~/.ssh/signed.pem
...

• unfortunately vault doesn’t provide a certificate revocation list, so the only workaround is to choose short TTL

To sum up vault or probably any other around signing ssh public keys can significantly scale up your solution even to some point that you can omit using some identity management solutions like FreeIPA.