Dynamic SSH ports with Hashicorp Vault TOTP

In this post I’m demonstrating how you could use Hashicorp’s Vault TOTP generator and an authenticator to connect to a dynamic ssh port. This is mostly a proof of concept and not a production grade solution, so use at your own risk.

What’s TOTP? It’s a Time-based One Time Password. TBOTP is probably too ugly of an acronym to have gained widespread use. I don’t know, I didn’t coin it.

¯\_(ツ)_/¯

Why?

Honestly, just for fun and as a mental exercise.

How does it work?

In essence, the process works as follows:

VAULT creates TOTP token, SSH creates listener on compatible TOTP provided port, user opens SSH session on port provided by Authenicator.

Requirements

I currently run a HashiStack on (in?) my homelab. My Hashistack consists of Consul, Vault and Nomad by Hashicorp. I run dozens of other infra services and apps, but I’ve found these to be core for many reasons.

This demo assumes a few things:

• You have a working vault server instance
• You have consul working with vault
• A working instance of Vault

Consul has a lot of useful features, but for this I’m using the service discovery, dynamic dns and dns steering. I have multiple vault instances and without consul I would have to manually reference one vault server or another. With consul integration, I can simply access active.vault.service.chrisbergeron.com for all TOTP requests and consul will direct the request to the active vault node. This saves me from having to muck with DNS entries and/or environment variables.

A TOTP Generator

I’m using Hashicorp Vault for the TOTP generator since it’s already a core piece of my home infrastructure. Vault provides a highly available secrets manager with an extensible modular architecture. For this demo, I’m using the totp secrets engine.

An OTP Authenticator

A One-Time-Password Authenticator is a device or app that provides Time-based One Time Passwords. It’s synchronized with the OTP Auth App during initial configuration. The purpose of having TOTP tokens is to provide a part of Multi-Factor Authentication (MFA). In a nutshell, MFA provides a good balance of security with usability.

It’s based on the principle of:

• Something you have (physically, like a phone, keyfob, etc)
• Something you know (ex. a pin number or similar)

An SSH Server

Obviously, you’ll need access rights to start/stop SSH on a server (or be able to start an SSH reverse proxy).

Show and Tell

We’ll start by enabling the TOTP secrets engine in vault:

user@hostname:/etc/vault.d$ vault secrets enable totp
Success! Enabled the totp secrets engine at: totp/

And checking that the engine is indeed enabled:

cbergeron@cb-mbp-max ~ $ vault secrets list
Path          Type         Accessor              Description
----          ----         --------              -----------
aws/          aws          aws_deadbeef          n/a
... snip ...
totp/         totp         totp_4be571e5         n/a

Here you see the totp/ path of type totp. This is where we’ll be sourcing our TOTP Code.

Let’s configure the TOTP engine to generate codes for a key. For this demo, I’ll be using monkey¹ as the key.

cbergeron@cb-mbp-max ~ $ vault write totp/keys/monkey generate=true issuer="CB\ Vault" account_name=monkey@chrisbergeron.com | grep barcode | tr -s " " | cut -f2 -d " " | base64 -d | tee > ./monkey-vault-totp.png

cbergeron@cb-mbp-max ~ $ ls -l monkey-vault-totp.png
.rw-r--r-- 1.7k cbergeron  7 Feb 22:10 monkey-vault-totp.png

cbergeron@cb-mbp-max ~ $ file monkey-vault-totp.png
monkey-vault-totp.png: PNG image data, 200 x 200, 16-bit grayscale, non-interlaced

cbergeron@cb-mbp-max ~ $ imgcat monkey-vault-totp.png

Okay, there’s a bit to unpack here. This first part:

vault write totp/keys/monkey generate=true issuer="CB\ Vault" account_name=monkey@chrisbergeron.com

Invokes the vault command, write ‘s a key to totp/keys/monkey, and tells the engine to generate a token. It has some parameters passed along, such as issuer="CB Vault" and the account_name to associate with the key.

When this command runs, it produces a base64 string and an OIDC authentication endpoint. For my purpose here, I’m not using using the OIDC uri. I just care about the QR code base64 string, which is what this part of the command is:

base64 -d | tee > ./monkey-vault-totp.png

This part of the command takes the base64 encoded string and decodes it into a .PNG image. This image is the QR code that an OTP Authenicator App uses for account setup.

Now we have a TOTP generator (vault) and a QR code that we can use to set up an OTP Authenticator app. It would have been trivial to use any standalone TOTP generator for this, but I wanted to use my existing infrastructure and tools for this demo.

Setting up the Authenticator App

I use an iPhone OTP App which is aptly named OTP Auth. It can be found on the Apple App Store here. TODOOOOOOOOOOO ***

Note: It’s important to ensure your TOTP Authentication App works with your

Once the app is installed you’ll need to add an account to it.

The app will prompt you to scan a barcode and open your phone’s camera. Simply point the camera at the QR code what was created by vault and it should import the configuration.

At this point, you should be able to query the vault and get the same value from the OTP App.

Query vault for a totp token:

cbergeron@cb-mbp-max ~ $ vault read totp/code/monkey
Key     Value
---     -----
code    088027

For our purpose, we only need the value, so we can use this instead:

cbergeron@cb-mbp-max ~ $ vault read -format=json totp/code/monkey | jq -r .data.code

614536

Syncronizing the App with SSH

Wasn’t sure what the interval was, so I watched the vault generator every second with watch -d “vault read ...”

At exactly 00 and 30 seconds, the number changes. This was an aha! moment for me. It makes perfect sense when I think about it though. So now, in theory all I need to do is have ssh change every 30 seconds.

Based on this, I’ve confirmed a few things:

• My Authenticator App is synchronized to the vault TOTP generator
• At :00 and :30 of each minute vault generates a new code
• SSH or the routing to SSH will have to be updated accordingly

Now it’s time to configure SSH to listen on a port that aligns with the TOTP Code.

Routing SSH

I can think of a few ways to get ssh to listen on a port dynamically:

• put a SOCKS5 or similar proxy in front of SSH
• change the SSH listening port and restart sshd
• use IPTables to use kernel network routing
• start an ssh reverse proxy to a fixed port ssh listening instance

For this demo, I’ve decided to start an ssh reverse proxy every 30 seconds. This can be done with a simple bash script:

#!/usr/bin/env bash

# Wait until the first occurrence of :00 seconds on the clock
date
echo "Sleeping until the next minute ..."
sleep $(echo 60 - $(date +%S) | bc)
date

# Loop indefinitely and start an ssh listener every 30 seconds
while true; do
	ssh -D 8080 -fNq -p $(echo "$(vault read -format=json totp/code/monkey | jq -r .data.code) % (10 ^ 5)" | bc)
	mypid=$!
	sleep 30
	kill ${mypidf}
done

This script works by first waiting until the next :00 second occurence, then starting a while loop. While in this loop, if the seconds past the minute is :00 or :30, it will start a new ssh listener and kill the previous listener.

Consiering that there are 1 million combinations of a 6 digit number (000000 - 999999), I need to do some math to only return 5 digits. This is done with a simple modulo math operator:

123456 % (10 ^ 5 )

Let’s break this down:

123456 modulo the product of 10 to the power of 5

I’m using modulo to get the remainder of 123456 divided by ten thousand.

123456 / 10000 = 23456

But wait, why not just use a bash builtin like ${vaultcode: -5}?

I think I’m going to have to some more math magic with this number later on and just getting the last 5 characters might limit my options. Let’s carry on…

So, now I have the last 5 digits.

cbergeron@cb-mbp-max tmp $ ./vault-ssh-demo.sh
Tue Feb  8 00:21:25 EST 2022
Sleeping until the next minute ...
Tue Feb  8 00:22:00 EST 2022
ssh -D 8080 -fNq -p 47590
ssh -D 8080 -fNq -p 62891
ssh -D 8080 -fNq -p 32116
ssh -D 8080 -fNq -p 23820

Wunderbar! But hark! There’s a new problem:

There are only 65534 ports available on modern (2022) computers and networking equipment. Of these, the first 1,024 are privileged ports.

But that’s not the issue.

The issue is that ports 65535-99999 are unassignable to SSH, but they do exist in the universe of numbers that could be generated by the TOTP generator!

To get around this, I’ll use an administrative hack instead of a technical one. If the first number is between 6 and 9, I won’t spawn a new ssh reverse proxy.

Problems / Reader Exercises

This was done as a fun exercise in my homelab. It’s not production grade for several reasons:

• Relying on a shell script ssh wrapper isn’t very robust
• The likelihood of getting a code that is out of bounds is higher than acceptable for many non-homelab use cases
• Passing unsanitized input directly to SSH could pose a security risk

That said, this was a fun little experiement.

If you have questions or comments, please post below :arrowdown:. If you hate it, that’s fine. I get it, we can’t all be as awesome as you are.

FAQ - Frequently Asked Questions

¹ “Why did you choose monkey as the username in your example?”

• Because everyone likes monkeys. They’re funny.

“Does this mean your ssh sessions can only last for up to 30 seconds?”

• Nope. By using a reverse proxy, once the session is established it will remain connected even after a new listener is started.

“Couldn’t someone just hack this by brute force connecting to all the ports?”

Yes and No for a few reasons:

• This is just a demo / PoC
• I run fail2ban, to firewall failed ssh attempts
• I use ssh keys.
• I could configure a backoff period to prevent rolling attacks

“Why don’t you just use an existing PAM/TOTP/ssh module?”

• This is just a demo / PoC

“By doing this, you’ve just reduced the combinatorix from 1 million to less than 65535!

• You right.

“Could this be hardened / made more secure?”

• I can think of a few ways this could be hardened and made more secure. I could couple port-knocking based on the TOTP code, or a similar strategy.

But wait, there’s more!

I’m not affiliated with nor sponsored by this OTP Auth app but I really like it.

It has an iOS widget available and when configured it looks like this:

Additionally, if you don’t have a modern mobile phone, the OTP Auth application that I mentioned here also has a macos version. This puts the OTP access right in the macos menubar: