SSH Integration
This section provides an overview of the Salt SSH integration with SUSE Manager. This integration adds support for both ssh-push and ssh-push-tunnel connections for Salt minions.
SSH Push Overview
Like the traditional stack, Salt minions may use an ssh connection to manage minions in place of Zeromq. This additional functionality is based on Salt SSH. Salt SSH enables you to execute salt commands and states via ssh without ever needing to install a salt-minion.
When the server executes an action on a minion an ssh connection is made on demand. This connection differs from the always-connected mode used by minions managed via Zeromq.
In SUSE Manager there are two ssh-push methods. In both use cases the server initiates an ssh connection to the minion in order to execute a Salt call using salt-ssh
. The difference in the two methods is how zypper/yum
initially connects to the server repositories:
- ssh-push
-
zypper works as usual. The http(s) connection to the server is created directly.
- ssh-push-tunnel
-
The server creates an http(s) connection through an ssh tunnel. The http(s) connection initiated by
zypper
is redirected through the tunnel by means of/etc/hosts
aliasing (see below). This method should be used for in place firewall setups that block http(s) connections from a minion to the server.
Salt SSH Integration
As with all Salt calls, SUSE Manager invokes salt-ssh
via the salt-api
.
Salt SSH relies on a Roster to obtain details such as hostname, ports, and ssh parameters of an ssh minion. SUSE Manager keeps these details in the database and makes them available to Salt by generating a temporary Roster file for each salt-ssh call. The location of the temporary Roster file is supplied to salt-ssh using the --roster-file= option
.
Authentication
salt-ssh supports both password and key authentication. SUSE Manager uses both methods:
- Bootstrapping Authentication
-
Password authentication is used only when bootstrapping. During the bootstrap step the key of the server is not authorized on the minion and therefore a password must be utilized for a connection to be made. The password is used transiently in a temporary roster file used for bootstrapping. This password is not stored.
- Common Salt Call Authentication
-
All other common salt calls use key authentication. During the bootstrap step the ssh key of the server is authorized on the minion (added to a minion’s
~/.ssh/authorized_keys
file). Therefore subsequent calls no longer require a password.
User Account for salt-ssh Calls
The user for salt-ssh
calls made by SUSE Manager is taken from the ssh_push_sudo_user
setting. The default value of this is root
.
If the value of ssh_push_sudo_user
is not root
then the --sudo
options of salt-ssh
are used.
SSH Push Tunnel HTTP(s) Redirection
For the ssh-push-tunnel
method the traffic originating from zypper/yum has to be redirected through an ssh tunnel in order to bypass any firewall blocking a direct connection from the minion to the server.
This is achieved by using port 1233
in the repo url:
https://suma-server:1233/repourl...
Next alias the suma-server hostname to localhost in /etc/hosts:
127.0.0.1 localhost suma-server
The server creates a reverse ssh tunnel that connects localhost:1233
on the minion to suma-server:443
(ssh … -R 1233:suma-server:443
)
The result is that zypper/yum will actually connect to localhost:1233
which is then forwarded to suma-server:443
via the ssh tunnel.
This implies that zypper can contact the server only if the tunnel is open. This happens only when the servers executes an action on the minion. Manual zypper operations that require server connectivity are not possible in this case.
SUSE Manager Salt SSH Call Sequence
-
Prepare the Salt Roster for the call
-
Create remote port forwarding option IF the contact method is ssh-push-tunnel
-
Compute the ProxyCommand IF the minion is connected through a proxy
-
create Roster content:
-
hostname
-
user
-
port
-
remote_port_forwards
: The remote port forwarding ssh option -
ssh_options
: other ssh options:-
ProxyCommand
: If the minion connects through a SUMA proxy
-
-
timeout
: default 180s -
minion_opts
:-
master
: set to the minion id if contact method is ssh-push-tunnel
-
-
-
-
create a temporary Roster file
-
execute a synchronous salt-ssh call via the API
-
remove the temporary Roster file
Additional Information:
Bootstrap Process Sequence
Bootstrapping minions uses salt-ssh under the hood. This happens for both regular and ssh minion.
The bootstrap sequence is a bit different than the regular salt-ssh call:
-
For a regular minion generate and pre-authorize the Salt key of the minion
-
If this is an ssh minion and a proxy was selected retrieve the ssh public key of the proxy using the mgrutil.chain_ssh_cmd runner. The runner copies the public key of the proxy to the server using ssh. If needed it can chain multiple ssh commands to reach the proxy across multiple hops.
-
Generate pillar data for bootstrap. Pillar data contains:
- mgr_server
-
The hostname of the SUSE Manager server
- minion_id
-
The hostname of the minion to bootstrap
- contact_method
-
The connection type
- mgr_sudo_user
-
The user for salt-ssh
- activation_key
-
If selected
- minion_pub
-
The public minion key that was pre-authorized
- minion_pem
-
The private minion key that was pre-authorized
- proxy_pub_key
-
The public ssh key that was retrieved from the proxy if the target is an ssh minion and a proxy was selected
-
If contact method is
ssh-push-tunnel
fill the remote port forwarding option -
if the minion connects through a SUMA proxy compute the
ProxyCommand
option. This depends on the path used to connect to the proxy, e.g. server → proxy1 → proxy2 → minion -
generate the roster for bootstrap into a temporary file. This contains:
-
hostname
-
user
-
password
-
port
-
remote_port_forwards
: the remote port forwarding ssh option -
ssh_options
: other ssh options:-
ProxyCommand
if the minion connects through a SUMA proxy
-
-
timeout
: default 180s
-
-
Via the Salt API execute:
salt-ssh --roster-file=<temporary_bootstrap_roster> minion state.apply certs,<bootstrap_state>`
<bootstrap_state> replaceable by bootstrap for regular minions or ssh_bootstrap for ssh minions.
The following image provides an overview of the Salt SSH bootstrap process.

Additional Information:
Proxy Support
In order to make salt-ssh work with SUSE Managers proxies the ssh connection is chained from one server/proxy to the next. This is also know as multi-hop or multi gateway ssh connection.

The ProxyCommand
In order to redirect the ssh connection through the proxies the ssh ProxyCommand
option is used. This options invokes an arbitrary command that is expected to connect to the ssh port on the target host. The standard input and output of the command is used by the invoking ssh process to talk to the remote ssh daemon.
The ProxyCommand basically replaces the TCP/IP connection. It doesn’t do any authorization, encryption, etc. Its role is simply to create a byte stream to the remote ssh daemon’s port.
E.g. connecting to a server behind a gateway:

In this example netcat (nc) is used to pipe port 22 of the target host into the ssh std i/o. |
Salt SSH Call Sequence via Proxy
Salt SSH Call sequence via a proxy.
-
SUSE Manager initates the ssh connections as described above.
-
Additionally the ProxyCommand uses ssh to create a connection from the server to the minion through the proxies.
Twin Proxies and SSH Push
The following example uses the ProxyCommand option with two proxies and the usual ssh-push method
This is a test.
# 1 /usr/bin/ssh -i /srv/susemanager/salt/salt_ssh/mgr_ssh_id -o StrictHostKeyChecking=no -o User=mgrsshtunnel proxy1 # 2 /usr/bin/ssh -i /var/lib/spacewalk/mgrsshtunnel/.ssh/id_susemanager_ssh_push -o StrictHostKeyChecking=no -o User=mgrsshtunnel -W minion:22 proxy2
-
Connect from the server to the first proxy
-
Connect from the first proxy to the second and forward standard input/output on the client to minion:22 using the -W option.

Twin Proxies and SSH Push Tunnel
The following example uses the ProxyCommand option with two proxies over an ssh-push-tunnel connection:
# 1 /usr/bin/ssh -i /srv/susemanager/salt/salt_ssh/mgr_ssh_id -o User=mgrsshtunnel proxy1 # 2 /usr/bin/ssh -i /home/mgrsshtunnel/.ssh/id_susemanager_ssh_push -o User=mgrsshtunnel proxy2 # 3 /usr/bin/ssh -i /home/mgrsshtunnel/.ssh/id_susemanager_ssh_push -o User=root -R 1233:proxy2:443 minion # 4 /usr/bin/ssh -i /root/.ssh/mgr_own_id -W minion:22 -o User=root minion
-
Connect from the server to the first proxy.
-
Connect from the first proxy to the second.
-
connect from the second proxy to the minion and open an reverse tunnel (-R 1233:proxy2:443) from the minion to the https port on the second proxy.
-
Connect from the minion to itself and forward the std i/o of the server to the ssh port of the minion (-W minion:22). This is equivalent to ssh … proxy2 netcat minion 22 and is needed because ssh doesn’t allow to have both the reverse tunnel (-R 1233:proxy2:443) and the standard i/o forwarding (-W minion:22) in the same command.

Additional Information:
Users and SSH Key Management
In order to connect to a proxy the parent server/proxy uses a specific user called mgrsshtunnel
.
The ssh config /etc/ssh/sshd_config
of the proxy will force the execution of `/usr/sbin/mgr-proxy-ssh-force-cmd
when mgrsshtunnel
connects.
`/usr/sbin/mgr-proxy-ssh-force-cmd
is a simple shell script that allows only the execution of scp
, ssh
or cat
commands.
The connection to the proxy or minion is authorized using ssh keys in the following way:
-
The server connects to the minion and to the first proxy using the key in
`/srv/susemanager/salt/salt_ssh/mgr_ssh_id
. -
Each proxy has its own key pair in
`/home/mgrsshtunnel/.ssh/id_susemanager_ssh_push
. -
Each proxy authorizes the key of the parent proxy or server.
-
The minion authorized its own key.

Additional Information:
Repository access via proxy
For both ssh-push and ssh-push-tunnel the minion connects to the proxy to retrieve packages and repo data.
The difference is how the connection works:
-
In case of ssh-push, zypper or yum connect directly to the proxy using http(s). This assumes there’s not firewall between the minion and the proxy that would block http connections initiated by the minion.

-
In case of ssh-push-tunnel, the http connection to the proxy is redirected through a reverse ssh tunnel.

Proxy setup
When the spacewalk-proxy
package is installed on the proxy the user mgrsshtunnel
is created if it doesn’t already exist.
During the initial configuration with configure-proxy.sh
the following happens:
-
Generate a ssh key pair or import an existing one
-
Retrieve the ssh key of the parent server/proxy in order to authorize it on the proxy
-
Configure the
sshd
of the proxy to restrict the usermgrsshtunnel
This configuration is done by the mgr-proxy-ssh-push-init
script. This is called from configure-proxy.sh
and the user doesn’t have to invoke it manually.
Retrieving the parent key is done by calling an HTTP endpoint on the parent server or proxy.
-
First
https://$PARENT/pub/id_susemanager_ssh_push.pub
is tried. If the parent is proxy this will return the public ssh key of that proxy. -
If a
404
is received then it’s assumed the parent is a server not a proxy andhttps://$PARENT/rhn/manager/download/saltssh/pubkey
is tried.-
If
/srv/susemanager/salt/salt_ssh/mgr_ssh_id.pub
already exists on the server it’s returned. -
If the public key doesn’t exist (because
salt-ssh
has not been invoked yet) generate the key by calling themgrutil.ssh_keygen
runner.
-
salt-ssh generates a key pair the first time it is invoked in
/srv/susemanager/salt/salt_ssh/mgr_ssh_id . The previous sequence is needed in case a proxy is configured before salt-ssh was invoked for the first time.
|
Additional Information: