------------------------------------------------------------------- Wed Apr 3 11:36:09 UTC 2019 - David Cassany - Commit 5d274fb by Birger Schmidt on 2019-04-03: Merge pull request #36 from cduch/patch-1 fixed kubelet param for system reservation (bsc#1128863) - Commit baf1472 by lcavajani on 2019-04-01: Merge pull request #37 from SUSE/disable-transactional-update-timer [bsc#1113518] Don't always reenable the transaction-update timer on update - Commit cf94b9a by lcavajani on 2019-04-01: Merge pull request #33 from SUSE/reactor Remove sync-modules from reactor(fix bsc#1128491) - Commit 45c9da7 by lcavajani on 2019-03-21: Merge pull request #38 from SUSE/release-3.0-disable_systemwide_proxy [3.0] Remove the proxy config files when `proxy:systemwide` is not set (bsc#1116572) - Commit 0a0d7d2 by lcavajani on 2019-03-21: Merge pull request #29 from SUSE/backport [3.0] Remove custom module caasp_http fix bsc#1127804 - Commit bb611d5 by George Gkioulis on 2019-03-19: Merge pull request #39 from SUSE/release-3.0-update_mine_before_orch [3.0] Synchronize everything before starting an orchestration (bsc#1124784) - Commit 78d3337 by Miquel Sabaté Solà on 2019-03-12: Don't always reenable the transaction-update timer on update When executing the update orchestration, it's not always desired to reenable the transactional-update. Moreover, sometimes the timer was on a weird state if the update failed. This commit refines the update orchestration following this logic: 1. Check first which nodes have explicitely disabled the timer, so they are not reenabled in the end. 2. Disable the timer at the very beginning. 3. If this is a migration, reenable it always. 4. Otherwise, reenable the timer only for these nodes which did not explicitely disable it before the update. Last but not least, I've also tweaked the `transactional-update/init.sls` file so the timer is not touched by this file when updating. bsc#1113518 Signed-off-by: Miquel Sabaté Solà ------------------------------------------------------------------- Thu Mar 14 20:39:08 UTC 2019 - Daniel Orf - Corrected the tar.gz file name and contained folder to match .spec file ------------------------------------------------------------------- Wed Mar 13 17:45:35 UTC 2019 - Daniel Orf - Fixes included in this change: * bsc#1098664: Shutdown of first master in the cluster causes dex pods to "CrashLoopBackOff" * bsc#1127326: CaaSP 3.0 cAdvisor API locked down anonymous access to kubelet customer need if for Cluster-Monitoring * bsc#1117942: Third party volume plugins configuration is wrong for kubelet/controller-manager * bsc#1121346: Velum – X-Frame-Options header set to “sameorigin” * bsc#1120717: Upgrade v2 to v3 broken * bsc#1111173: Stop.sls might set 'should_uncordon' incorrectly after failed orchestrations ------------------------------------------------------------------- Tue Mar 5 18:05:58 UTC 2019 - Daniel Orf - Commit e151518 by Rafael Fernández López ereslibre@ereslibre.es Make nodename appear first on the /etc/hosts file Salt will pick the first name on the current default interface to determine the hostname of the machine. Since we are sorting with all entries for each machine there's a high change that a salt minion id will win the first position, affecting certain grains that we use to determine the hostname of the node. Fixes: bsc#1117339 (cherry picked from commit 4f75ad33fdffc3ea88c91c56e56fb5af9275714d) ------------------------------------------------------------------- Fri Feb 22 17:50:55 UTC 2019 - jmassaguerpla@suse.com - Commit 29c9641 by Vítor Avelino vavelino@suse.com haproxy: block requests to /internal-api endpoint Internal api endpoints exposes sensitive data and this cannot be accessed via internet. This internal api was developed inside velum project and haproxy was allowing the request to that endpoint. Velum listens on 0.0.0.0 and needs to block for that specific path. With this patch we are blocking any request to anything that starts with /internal-api. Signed-off-by: Vítor Avelino bsc#1121162 ------------------------------------------------------------------- Wed Feb 13 16:56:26 UTC 2019 - jmassaguerpla@suse.com - Commit a11a668 by Florian Bergmann fbergmann@suse.de Fix bsc#1121147: Add missing import of 'os'. Must have been dropped during cherry-picking. ------------------------------------------------------------------- Fri Feb 8 08:12:46 UTC 2019 - jmassaguerpla@suse.com - Commit f5843ea by Florian Bergmann fbergmann@suse.de Force basename on the system certificate name to prevent path traversal fix bsc#1121147 ------------------------------------------------------------------- Thu Feb 7 16:49:15 UTC 2019 - jmassaguerpla@suse.com - Commit ace6498 by Panos Georgiadis drpaneas@gmail.com Disable insecure port in kube-apiserver (bsc#1121148) * Fixes bsc#1121148 - Critical Security issue for KubeAPI Insecure API port exposed to all Master Node guest containers In older versions of Kubernetes, you could run kube-apiserver with an API port that does not have any protections around it. This PR disables insecure port by passing the --insecure-port=0 In recent versions, this has been disabled by default with the intention of completely deprecating it ------------------------------------------------------------------- Wed Feb 6 11:52:52 UTC 2019 - jmassaguerpla@suse.com - Fixes included in this change: * bsc#1121146 - Kubernetes – Kubelet Service allows unauthenticated access to Kubelet API * bsc#1122439 - failed to parse bool none * bsc#1123291 - CaasP 3.0 Update Admin node, worker and master failed * bsc#1123650 - ExperimentalCriticalPodAnnotation feature not enabled * bsc#1114832 - Running supportconfig on any node can take lots of resources, even fill the hard disk on big/long-running clusters ------------------------------------------------------------------- Tue Jan 29 12:05:52 UTC 2019 - Containers Team - Commit 526f9dd by David Helkowski dhelkowski@suse.com Add support for OIDC connectors to dex configmap (cherry picked from commit 9ef0f58a26a90eecbdb5a93425c0b94f8cc25581) ------------------------------------------------------------------- Tue Jan 22 08:00:35 UTC 2019 - Containers Team - Commit 1827f13 by Michal Jura mjura@suse.com update etcdctl sysconfig with ENDPOINTS flag (bsc#1120047) (cherry picked from commit 129466a842eea437639f13015d12341929d417d0) ------------------------------------------------------------------- Thu Jan 17 11:41:00 UTC 2019 - Containers Team - Commit bdc3f3b by Kiall Mac Innes kiall@macinnes.ie Run tox based tests using a pre-baked tox container [mchandras: resolved conflicts] (cherry picked from commit 309adf413929a6d924773e9c34f63eaf4cc2e85f) ------------------------------------------------------------------- Thu Dec 20 13:22:58 UTC 2018 - Containers Team - Commit 30c898d by Michal Jura mjura@suse.com [CPI] Add self-signed certificate to CPI configuration, bsc#1101973 (cherry picked from commit 67e276cd10066918d7590d29d485215770795208) ------------------------------------------------------------------- Fri Dec 7 12:55:21 UTC 2018 - Containers Team - Commit c60bd07 by Florian Bergmann fbergmann@suse.de Changes has to be dictionary. When using a boolean it will fail the state in salt-2018.3.0. bsc#1098334 (cherry picked from commit 84a115ffaf30bb552b59178b86e7762638352cf2) Commit f27ffe1 by Alvaro Saurin alvaro.saurin@gmail.com Generate the /etc/hosts file from a state, merging our entries with previously found entries. bsc#1098334 (cherry picked from commit 5bcafd25baf1c622efc69997928b56899fc8be16) ------------------------------------------------------------------- Fri Dec 7 12:38:53 UTC 2018 - Containers Team - Commit 529cc99 by Michal Jura mjura@suse.com [CPI] Add option to ignore OpenStack Cinder availability zone, bsc#1095572 Ignore OpenStack Cinder avability zone when attaching volumes. When Nova and Cinder have different availability zones, this should be set to true. Default is false. (cherry picked from commit 8ded363da94c017cad364b0efc08da6f0fc77c22) ------------------------------------------------------------------- Fri Dec 7 12:36:48 UTC 2018 - Containers Team - Commit e39f138 by Michal Jura mjura@suse.com [CPI] Fix and remove empty lines in OpenStack cpi config, bsc#1101973 (cherry picked from commit f93b74b9e7a77561c7e431f3a8c82667611da071) ------------------------------------------------------------------- Fri Dec 7 12:34:38 UTC 2018 - Containers Team - Commit 94cfac1 by Michal Jura mjura@suse.com [CPI] Add self-signed certificate to CPI configuration, bsc#1101973 (cherry picked from commit 67e276cd10066918d7590d29d485215770795208) ------------------------------------------------------------------- Fri Dec 7 12:29:21 UTC 2018 - Containers Team - Commit babb2ff by Florian Bergmann fbergmann@suse.de Fix bsc#1115236: Use the correct key to access the etcd_version from pillars (cherry picked from commit bf5feaa15f5f9adae14ebf61db6b4ae38ddb04ee) Commit 7a8cd18 by Florian Bergmann fbergmann@suse.de Fix bsc#1115236: Only add a new etcd member if no alias is already a member When adding a new member to etcd, it might happen that it is already part of the cluster using one of the aliases - when migrating from v2 to v3 it seems common that the default nodename changes. If this is the case it should not be added again with the new nodename, as one node can not have 2 etcd members. (cherry picked from commit 962a830f98a1300be23b63bfd78b4e3847eae2ab) ------------------------------------------------------------------- Tue Dec 4 16:50:07 UTC 2018 - Containers Team - Commit ef642a1 by Florian Bergmann fbergmann@suse.de Fix bsc#1116933: Add a dummy state to prevent empty state in orch This is a workaround for https://github.com/saltstack/salt/issues/14553 when upgrading crio 1.9 to 1.10. (cherry picked from commit 1e20516f8e67faddfb5c2dead773cdef2abe5331) ------------------------------------------------------------------- Tue Dec 4 12:45:55 UTC 2018 - Containers Team - Commit 54013dc by Florian Bergmann fbergmann@suse.de Workaround bsc#1116933: remove /var/lib/container on crio installation (cherry picked from commit 0d30835a426e7d3caa8e1b096552122b21d2d014) ------------------------------------------------------------------- Fri Nov 30 12:49:30 UTC 2018 - jmassaguerpla@suse.com - Workaround bsc#1116933: remove /var/lib/container on crio installation ------------------------------------------------------------------- Fri Nov 16 09:05:58 UTC 2018 - containers-bugowner@suse.de - Commit 2a0c0d0 by Maximilian Meister mmeister@suse.de don't run haproxy states when not really needed in case of a kubernetes update from 1.9 to 1.10 we can't afford to stop kubernetes through the haproxy states, because it will not be able to restart as the --config file flag has changed between those releases the update orchestration fails in the sanity check of the state all-workers-3.0-pre-clean-shutdown because the new kubelet configuration is already applied, but the old kubernetes version is still running before the reboot This is a corner case and our other states would have to be adapted as well to re-run configs when a node gets accidentally rebooted and the config hasn't been applied yet. Furthermore this is only an issue coming from v2 during migration to v3 - so the case that this happens is even rarer. Trying to run this state on each worker would require a check for /etc/caasp/haproxy/haproxy.cfg to safely determine if it needs to be run or not, but it is not possible to use salt runners with a target to determine if this file exists on all worker nodes. salt.runners.salt.cmd doesn't accept targets salt.runners.salt.execute only exists since salt2017.7.0 which might not be present yet for a user that hasn't installed the salt upgrade yet. bsc#1114645 Signed-off-by: Maximilian Meister (cherry picked from commit 11c82a549ea9284374507e86319a4d0c71fa6b78) ------------------------------------------------------------------- Mon Nov 12 09:40:05 UTC 2018 - containers-bugowner@suse.de - Commit 4337a9e by Rafael Fernández López ereslibre@ereslibre.es Add a whitelist for returned events so we only save events that we care about Fixes: bsc#1112967 (cherry picked from commit 793d856721ad1d7cf990622342b49415180e928f) ------------------------------------------------------------------- Mon Nov 12 09:36:03 UTC 2018 - containers-bugowner@suse.de - Commit 6ed3236 by Ludovic Cavajani lcavajani@suse.com bsc#1108195 Aggregation layer needs configuration Signed-off-by: Ludovic Cavajani (cherry picked from commit 081d260d60a2e542af7418c026d9c55908abe10b) ------------------------------------------------------------------- Tue Nov 6 11:44:53 UTC 2018 - containers-bugowner@suse.de - Commit c2260b2 by Michal Jura mjura@suse.com Move deprecated flags to kubelet config.yaml (cherry picked from commit c02c3ec409576ec03b590d74cdf113106aa288e1) bsc#1114645 ------------------------------------------------------------------- Mon Oct 29 11:28:33 UTC 2018 - containers-bugowner@suse.de - Commit 215213e by Maximilian Meister mmeister@suse.de fix for bsc#1111333 we tried to run zypper from within the ca container which tried to fetch from the sles repos Signed-off-by: Maximilian Meister ------------------------------------------------------------------- Thu Oct 18 12:09:31 UTC 2018 - jmassaguerpla@suse.com - Commit 117cbb5 by by Kiall Mac Innes kiall@macinnes.ie Configure addon pod affinity Sometimes, Kubernetes will schedule all replicas of an addon to the same machine. Defeating much of the purpose of running multiple replicas. Configure all addons with affinity rules to encourage Kubernetes to spread these pods around the available machines. bsc#1101805 ------------------------------------------------------------------- Thu Oct 18 12:07:11 UTC 2018 - jmassaguerpla@suse.com - Commit 32c965b by Ludovic Cavajani lcavajani@suse.com Fix bsc#1105910 CAdvisor is publicly exposed on the kubernetes nodes(:::4194) - Commit 5cf1b92 by Ludovic Cavajani lcavajani@suse.com bsc#1105910 disable read-only kubelet port ------------------------------------------------------------------- Thu Oct 18 12:00:21 UTC 2018 - jmassaguerpla@suse.com - Commit 08c508d by Rafael Fernández López ereslibre@ereslibre.es and Alvaro Saurin alvaro.saurin@gmail.com Perform some checks before starting the node removal feature#node_removal Fixes: bsc#1098433 Fixes: bsc#1098064 Fixes: bsc#1098161 - Commit 97fbfd5 by Maximilian Meister mmeister@suse.de and Rafael Fernández López ereslibre@ereslibre.es switch to etcd3 as a storage back-end upgrade#etcdctl Fixes: bsc#1098433 Fixes: bsc#1098064 Fixes: bsc#1098161 - Commit 9fc144 by Rafael Fernández López ereslibre@ereslibre.es Use etcd api v2 on the 3.0 release branch This commit can be reverted when we want to migrate to the etcd api v3. Fixes: bsc#1098433 Fixes: bsc#1098064 Fixes: bsc#1098161 - Commit b60fa54 by Rafael Fernández López ereslibre@ereslibre.e Allow `etcd` to grow as required and shrink to optimal etcd cluster sizes on corner cases. Improve `etcd` configuration handling to allow it to grow as needed. This change includes: * Adding several masters at the same time ** `etcd` will grow instance by instance still, as recommended by the `etcd` administration best practices. * Try to use the current endpoints reported by `etcd`. This makes much easier to grow several instances one by one without having to relay on internal hacks to properly set up `ETCD_INITIAL_CLUSTER` environment variable. * Add helper methods that allow us to list current members (active and unstarted) * Differentiate between the first bootstrap (`ETCD_INITIAL_CLUSTER_STATE` defaults to `new`) and *any* other run, where `ETCD_INITIAL_CLUSTER_STATE` will be `existing`, as the `etcd` cluster is already running. When we grow, we take into account the golden ratio; however, when shrinking the cluster we don't. It might happen that a cluster ends up with not recommended etcd number of nodes (2, 4, 6...) depending on how it grew before and how it shrank. This logic makes sure that we are always on an etcd golden ratio, also on corner cases when removing nodes. Fixes: bsc#1098433 Fixes: bsc#1098064 Fixes: bsc#1098161 - Commit 6f2e00e by Rafael Fernández López ereslibre@ereslibre.e Add more test cases for `caasp_etcd` * Update to allow `etcdctl` API version 3 Fixes: bsc#1098433 Fixes: bsc#1098064 Fixes: bsc#1098161 ------------------------------------------------------------------- Thu Oct 18 11:02:35 UTC 2018 - containers-bugowner@suse.de - Commit b8fea6c by Vicente Zepeda Mas vzepedamas@suse.com Fix bsc#1099045 adds annotation to use docker/default seccomp profile Signed-off-by: Vicente Zepeda Mas ------------------------------------------------------------------- Tue Oct 16 08:02:58 UTC 2018 - containers-bugowner@suse.de - Commit 2a8325e by Maximilian Meister mmeister@suse.de Fix bsc#1111168: Do not expect masters to always need to be updated If the masters already updated, but workers failed to update this state will not have any minions to run on and fail if 'execpt_minions: false' is not set. Signed-off-by: Maximilian Meister (cherry picked from commit 6c552b98817d9c1c1496197f877e8e29c00110c7) ------------------------------------------------------------------- Tue Oct 9 14:08:43 UTC 2018 - containers-bugowner@suse.de - Commit 9f195bc by Florian Bergmann fbergmann@suse.de Fix bsc#1111168: Do not expect masters to always need to be updated If the masters already updated, but workers failed to update this state will not have any minions to run on and fail if 'execpt_minions: false' is not set. (cherry picked from commit 9786217d2e68d9d130522eb3aa8a43d4007f685e) ------------------------------------------------------------------- Mon Oct 8 09:10:20 UTC 2018 - containers-bugowner@suse.de - Commit 3fef004 by Rafael Fernández López ereslibre@ereslibre.es Always wait for haproxy to be serving requests before continuing. We could do the wait on the different places to avoid a generic piece like haproxy having to wait for a specific component like the apiserver, but we are already writing specific components in its configuration, and a future reordering of states could trigger this error again. So, when we kill haproxy, wait for it to be serving requests again before continuing with the next state. On the 2 to 3 upgrade this was causing a failure because right after restarting haproxy we were trying to drain the node. Since we run this operation on the very same machine that is being targeted, this `kubectl` command cannot reach the apiserver (because haproxy is still initializing), causing the whole update orchestration to fail. Fixes: bsc#1109661 (cherry picked from commit 95c1980e99e0e2a9787caab02b86056db8e199c0) ------------------------------------------------------------------- Mon Sep 17 09:30:07 UTC 2018 - containers-bugowner@suse.de - Commit d39411e by David Helkowski dhelkowski@suse.com Add configmap from pillar data to dex ldap connectors (fate#324601) ------------------------------------------------------------------- Fri Aug 17 12:10:56 UTC 2018 - containers-bugowner@suse.de - Commit bbf18cd by Kiall Mac Innes kiall@macinnes.ie Reintroduce kubelet drain timeout and abort if draining fails This is a partial revert of 03d371fc489f4bd0e15da348b60390aa558daf76. We reintroduce the --timeout flag, leaving --grace-period unset (thus, inheriting from from the Pods terminationGracePeriodSeconds value). Without this, kubectl drain can hang forever in certain circumstances. Additionally, should the drain fail, then fail the orchestration. This ensures that we do not reboot a node which has, for example, SES/Ceph mounts active, which would in turn cause systemd to hang as the machine is rebooted. bsc#1104217 (cherry picked from commit 1d5c83010f0179193b936826a291d718c37050ea) - Commit e5e046 by Kiall Mac Innes kiall@macinnes.ie Create RoleBinding to allow dex discovery This RoleBinding allows unauthenticated users (such as those using caasp-cli) to find the Dex service endpoint. This was dropped in 3cdcfae bsc#1104658 (cherry picked from commit 904eac6) ------------------------------------------------------------------- Fri Aug 17 12:08:46 UTC 2018 - containers-bugowner@suse.de - Commit 281beef by Rafael Fernández López ereslibre@ereslibre.es HAProxy will refuse to start if it cannot resolve any name. In a context in which cloud-init could be updating the hostnames after machines are continuing with the update orchestration, we could be writing one thing to `/etc/hosts` and another one in the `haproxy` configuration, refusing this one to start because it cannot resolve the new name. This easily fixable in a newer HAProxy version by using the `init-addr` configuration, so HAProxy won't refuse to start if it cannot resolve any backend -- it will just ignore it --. For now, let's make the temporal window as small as possible, making the `haproxy` init.sls depend on the `etc-hosts` SLS, as it's *so* dependant on it. However, this is not in any way an ideal fix; rather a way to make this problematic window as small as possible. Fixes: bsc#1097478 (cherry picked from commit 54e4891ee95ced02f19d00484dcde2a76360026b) ------------------------------------------------------------------- Mon Aug 6 11:09:20 UTC 2018 - containers-bugowner@suse.de - Commit aada6c8 by Florian Bergmann fbergmann@suse.de Fix bsc#1103699: Allow states targeting specific versions of caasp to have no nodes. Otherwise the states would fail if no nodes are returned in the `tgt` expression. (cherry picked from commit 3c67ad3d89c44a2c428cfdafc90be8fba65e3fc8) Commit 1c8abfd by Maximilian Meister mmeister@suse.de Fix bsc#1103699: explicitly pass unix_socket this affects only kubic for now where we use PyMySQL we cant use the MYSQL_UNIX_PORT workaround anymore as we could do with MySQLdb salt#mysql-unix-socket Signed-off-by: Maximilian Meister (cherry picked from commit 45b8f7b54511f38135d7fdbbd36cc262349f9d45) Commit b205ed0 by Florian Bergmann fbergmann@suse.de Fix bsc#1103699: Adjust network_settings config format for salt 2018.3.0. Before this release the format did not use the 'interfaces' key. (cherry picked from commit 8250ea887c0f5f26b25aa5ffc0e9947a46d7774f) Commit 274e952 by Florian Bergmann fbergmann@suse.de Fix bsc#1103699: Use a reactor to sync modules and update mine on minion start. (cherry picked from commit e950dd112f0e8e74ebbc30a5d6d1d58d228f94e3) Commit 2719f3c by Florian Bergmann fbergmann@suse.de Fix bsc#1103699: Force the 15-secret.yaml file to be created first in dex. Otherwise the kubectl_apply_dir_template macro will fail, as the file does not exist when it tries to run `salt.hashutil.digest` on it. (cherry picked from commit 38a274502ea134b81305cd59a79f7c7cb8059618) Commit 17f93a1 by Florian Bergmann fbergmann@suse.de Fix bsc#1103699: Add missing __virtual__ functions to execution modules. (Attempt to make the automatic synchronization work for custom execution modules - seems not to work) (cherry picked from commit ceb7689ebc36ac572d52447a8b7429ab270263d5) ------------------------------------------------------------------- Tue Jul 17 13:48:28 UTC 2018 - containers-bugowner@suse.de - Commit 01568e5 by Maximilian Meister mmeister@suse.de override volume plugin dir (bsc#1084766) kubernetes 1.10 uses /usr/libexec by default which doesnt exist, and we want to stick with /usr/lib Signed-off-by: Maximilian Meister (cherry picked from commit de8bd66cebf33dc1e06e4204e8b8211feef2709a) ------------------------------------------------------------------- Mon Jul 16 07:15:13 UTC 2018 - containers-bugowner@suse.de - Commit bcfe4e9 by Rafael Fernández López ereslibre@ereslibre.es Batch potentially dangerous and massive operations. Fixes: bsc#1101124 (cherry picked from commit f0a0ac1bd1190ee1989eaa9d06fc9da272b3e2ea) ------------------------------------------------------------------- Thu Jul 12 08:01:50 UTC 2018 - containers-bugowner@suse.de - Commit 71b20b6 by Rafael Fernández López ereslibre@ereslibre.es Add haproxy migration sls to apply during upgrade During an upgrade from 2.0 to 3.0, workers will lose communication with the apiservers on the master nodes because of an auth change. After we have applied all the master nodes, and before we start looping over the workers, apply haproxy system-wide on all the workers, allowing their haproxy to update its configuration, thus, being able to authenticate against the apiservers again. This patch includes a new tree structure, meant to be destroyed between versions, but that allows to not poison the main structure of states with transient migration logic. The structure is as follows: - migrations - - - overriden-sls/* - * (direct actions that can spawn other migration tasks) Fixes: bsc#1100212 (cherry picked from commit d03c2fadd5b3e72476a09b9ab9722495da091905) Commit aa9b2e5 by Rafael Fernández López ereslibre@ereslibre.es Migrate all labels when renaming a node (builtin and user-defined labels). Fixes: bsc#1100891 (cherry picked from commit f190a7a9994075d969c3e5e042e3d5ff259f0f53) Commit 49b391a by Rafael Fernández López ereslibre@ereslibre.es Only perform migrations on machines that are going to be updated. On an upgrade process we are going to perform different migrations; only perform these migrations on machines that are part of the current subset of machines to be updated. Fixes: bsc#1100115 (cherry picked from commit a7e1b723690ee98a1c9cf3589607d652f3caa02e) Commit a34af40 by Kiall Mac Innes kiall@macinnes.ie Stop kubelet before any other services Explicitly stop kubelet before any other services. If cri.stop is ran in parallel to or before kubelet.stop, kubelet will be unable to successfully drain. bsc#1085980 (cherry picked from commit fd3507f50aa95a90856ce3d4a9e721ff28a0ea6f) ------------------------------------------------------------------- Thu Jun 21 09:38:51 UTC 2018 - containers-bugowner@suse.de - Commit 1bf2ec1 by Rafael Fernández López ereslibre@ereslibre.es Call to `mine.update` after `saltutil.sync_pillar` has been called. During an upgrade we want to call to `mine.update` after `saltutil.sync_pillar` has been called, because the `mine_functions` reside on the pillar, we first want to make sure to sync that, and update the mine afterwards. Otherwise, we risk doing this in a race condition when the salt minion starts, and it could or could not lead to update orchestration failure. Fixes: bsc#1097478 (cherry picked from commit 97d81781d8bb7ad7586caa5c613f6b5003106873) ------------------------------------------------------------------- Wed Jun 20 15:36:40 UTC 2018 - containers-bugowner@suse.de - Commit 5505237 by Kiall Mac Innes kiall@macinnes.ie During upgrade, ensure masters always have the correct taints When migrating from the "old" to "new" names for the kubelets, we pre-create the new node so that we can clone the network config. This means the kubelet is NOT self-registering, and the "single use options" like --register-with-taints are ignored. This means the kubelet is connected from the period of time where it starts, to where salt later forcefully adds the taint. Any pods created during this window could end up scheduled to the master. bsc#1098383 ------------------------------------------------------------------- Mon Jun 18 10:52:27 UTC 2018 - containers-builds@suse.de - Commit 01b9c4e by Alvaro Saurin alvaro.saurin@gmail.com Move the early services setup even before updating the masters (we can do this by removing some unnecessary dependencies). bsc#1096992 ------------------------------------------------------------------- Fri Jun 15 19:09:55 UTC 2018 - containers-builds@suse.de - Commit 515c677 by Alvaro Saurin alvaro.saurin@gmail.com Try to load the manifests once we have at least one updated master. bsc#1096992 (cherry picked from commit ba205822e858d33a627e8717de4c1779d31f4c63) ------------------------------------------------------------------- Fri Jun 15 15:19:45 UTC 2018 - containers-builds@suse.de - Commit 32c85ce by Alvaro Saurin alvaro.saurin@gmail.com Early setup some services on updates Removed "allowedFlexVolumes" in PSP (as it doesn't pass the API verification in 2.1) bsc#1096992 (cherry picked from commit 180e54580e3d0066b5f73d6e342f366140c9cd4a) ------------------------------------------------------------------- Wed Jun 13 11:10:26 UTC 2018 - containers-builds@suse.de - Commit 4734ae5 by Alvaro Saurin alvaro.saurin@gmail.com Do not set the `bootstrap_complete` flag in all the nodes: do it only in the nodes that had some role assigned. Remove the `bootstrap_in_progress` even if the orchestration fails. Fixed typo in target. bsc#1094078 (cherry picked from commit a4480ed33b2b980ff13523c8c7aaa66591431a9d) ------------------------------------------------------------------- Tue Jun 12 18:04:29 UTC 2018 - containers-builds@suse.de - Commit 5e13ca5 by Rafael Fernández López ereslibre@ereslibre.es Remove mine information when removing a node This will avoid to render stale information about critical components, like `etcd` endpoints in the `etcd` configuration. `etcd` is very sensitive to this kind of misleading (stale) information, if more endpoints are provided in `ETCD_INITIAL_CLUSTER` than the ones that actually exist in the cluster, a new instance of etcd will refuse to start. Fixes: bsc#1097001 Fixes: bsc#1097147 (cherry picked from commit cf5b83bb8bbb867178945cf60155378dee657bae) ------------------------------------------------------------------- Mon Jun 11 14:17:08 UTC 2018 - containers-builds@suse.de - Commit 6dfab39 by Rafael Fernández López ereslibre@ereslibre.es Force `etc-hosts` sls to be run before `etcd` Before the real update orchestration happens we are updating etcd certificates, so this machine isn't left isolated. However, in this process, the configuration for etcd might refer to the new machine names if this happens during the upgrade of 2.0 to 3.0. This might leave the etcd instances in a state in which they cannot resolve other etcd peer names (because their `/etc/hosts` file is outdated). In order to prevent this, force the `etc-hosts` sls to be run before we execute the `etcd` sls, so we are sure that `/etc/hosts` will contain both the old and the new names during the upgrade, and etcd will be able to refer to other peers using the new hostnames. Fixes: bsc#1096750 (cherry picked from commit 23ce1f28cc1c35b12ac43c57ec265dcb19a53611) ------------------------------------------------------------------- Mon Jun 11 11:57:34 UTC 2018 - containers-builds@suse.de - Commit 5719ff2 by Rafael Fernández López ereslibre@ereslibre.es Also stop `kubelet` on masters when performing an upgrade If some important change lands between Kubernetes updates, it might happen that since we don't disable the `kubelet` service on the master nodes, when the machine gets rebooted, `systemd` will try to start the `kubelet` service, failing in a burst mode. This will prevent our salt states from trying to start it again, because the service will be in a failed state. Stop the service and disable it on the masters too when we are performing an upgrade, this way we are sure that we'll try to start and enable it when we have performed the required changes for it to succeed. Fixes: bsc#1096768 (cherry picked from commit ec6238cb4d43983cce7c708b677c9e99e508d787) ------------------------------------------------------------------- Thu Jun 7 08:48:42 UTC 2018 - containers-builds@suse.de - Commit 49c4721 by Alvaro Saurin alvaro.saurin@gmail.com Use the cache whenever something bad happens when refreshing the Pillar from Velum. (cherry picked from commit c77b0ee846350e7f60abedb011a29649d29131a5) bsc#1093123 ------------------------------------------------------------------- Wed Jun 6 13:47:01 UTC 2018 - containers-builds@suse.de - Commit 402cbfb by Rafael Fernández López ereslibre@ereslibre.es Uncordon node in a explicit sls action This way we don't try to uncordon the node in the `kubelet/init.sls` file, required for example by `haproxy`, that will end up in the machine trying to early uncordon itself (when `haproxy` configuration hasn't been written yet, and leading to early failure). Splitting this action and called only when required (this is: the update process) is safer. Fixes: bsc#1080978 (cherry picked from commit 02f063385e3a8cd435a76280ca246a87099a01d5) ------------------------------------------------------------------- Tue Jun 5 15:26:23 UTC 2018 - containers-builds@suse.de - Commit 694a8ce by Alvaro Saurin alvaro.saurin@gmail.com Skip nodes that are being removed in the list of servers in haproxy. bsc#1095330 (cherry picked from commit 33b39b3e8d670c5ce7a77f49b6ff7ddd7da37151) Commit c00f6a4 by Alvaro Saurin alvaro.saurin@gmail.com Fix the "targets" priorities for getting nodes for replacements. Minor: use the same pattern for targeting nodes in removals.sls as in kubernetes.sls. Do not use "unassigned" nodes when looking for replacements. Minor improvements bsc#1095336 bsc#1095330 bsc#1094078 (cherry picked from commit 8484c28fac28a072791c3cc5e01ef7c7d16d4bcb) Commit 75d3e00 by Alvaro Saurin alvaro.saurin@gmail.com Minor cleanups and "beautifications" feature#cleanups (cherry picked from commit b80c8f1a223ca235fd2d62f0e21e1365ee0af643) bsc#1095330 bsc#1095336 ------------------------------------------------------------------- Fri May 25 14:08:53 UTC 2018 - containers-bugowner@suse.de - Commit 34d9f0 by Ty Daines and Florian Bergmann fix bsc#1091809: pillar and openstack config can use project and domain ids (cherry picked from commit 37556bb) ------------------------------------------------------------------- Fri May 25 13:05:21 UTC 2018 - containers-bugowner@suse.de - Commit 205b7db by Rafael Fernández López ereslibre@ereslibre.es Remove unsupported `--require-kubeconfig` argument deprecated in Kubernetes (and removed in 1.10) Fixes: bsc#1094217 (cherry picked from commit 2a6eb071814732eaa8aa3d29970b9b5689f7963f) ------------------------------------------------------------------- Thu May 24 12:04:39 UTC 2018 - containers-bugowner@suse.de - Commit d025704 by Maximilian Meister mmeister@suse.de fix crio reload and drop a duplicated reload watcher fix#reload Signed-off-by: Maximilian Meister (cherry picked from commit ccde36b6ef0c03d1aa419219e9fe03ea63da6d08) Commit a302482 by Maximilian Meister mmeister@suse.de fix docker reload again it apparently doesnt work to use service.running to do the reload. using cmd.run is reliable fix#reload-cert Signed-off-by: Maximilian Meister (cherry picked from commit 9a47960237bc54c0f6f711fbcd1dfcba4b358f11) ------------------------------------------------------------------- Tue May 22 19:05:18 UTC 2018 - containers-bugowner@suse.de - Commit 08d471b by Kiall Mac Innes kiall@macinnes.ie Fix module tests on python3 Commit 920a824 by Kiall Mac Innes kiall@macinnes.ie Allow salt tests to be ran via tox and Jenkins Example to run them locally: tox -e tests-salt-2016.11.4-py27 or: tox -e tests-salt-2016.11.4-py34 (cherry picked from commit 987f865b2123b90cd558e26caa563f1f0783b565) ------------------------------------------------------------------- Tue May 22 17:23:51 UTC 2018 - containers-bugowner@suse.de - Commit 57b4664 by Florian Bergmann fbergmann@suse.de Install system wide certificates from pillars. `cert`-state will install the certificates as trust anchors. (cherry picked from commit 22a3b2373757a51cb740a3ff71564f80092b1cdc) Fixes bsc#1090067 ------------------------------------------------------------------- Tue May 22 16:54:45 UTC 2018 - containers-bugowner@suse.de - Commit 7e91362 by Rafael Fernández López ereslibre@ereslibre.es Log all CRI issues as we go, and show them if we really timeout Related: bsc#1093918 (cherry picked from commit 8b75460a77f682f315bd8ad4bbbfd409a6f185a1) ------------------------------------------------------------------- Tue May 22 08:43:02 UTC 2018 - containers-bugowner@suse.de - Commit 5cc699f by Maximilian Meister mmeister@suse.de skip removed etcd servers (bsc#1093305) Signed-off-by: Maximilian Meister (cherry picked from commit 6c4ec0c05fdfd9991869bb21f51c2d0ec3afab18) ------------------------------------------------------------------- Tue May 22 08:23:25 UTC 2018 - containers-bugowner@suse.de - Commit effa069 by Maximilian Meister mmeister@suse.de also reload docker when certificates change fix#reload-certs Signed-off-by: Maximilian Meister (cherry picked from commit b5a6432afa710a4a276fe3afd26b029edebfa882) ------------------------------------------------------------------- Mon May 21 22:54:11 UTC 2018 - containers-bugowner@suse.de - Commit 7c0fd6d by Kiall Mac Innes kiall@macinnes.ie Add Collaborator Check to flake8 job (cherry picked from commit becdf82e8d437342d05fbcdaf36bbab674c34ff4) ------------------------------------------------------------------- Sat May 19 17:25:46 UTC 2018 - containers-bugowner@suse.de - Commit a14ef0e by Flavio Castelli fcastelli@suse.com Remove unneeded state The registries state is something from the early days of caasp. Something we don't need (and use) anymore. feature#remove-unneeded-code-registries Signed-off-by: Flavio Castelli (cherry picked from commit 4497dac531a960e237eec34eedea0887669cf42a) ------------------------------------------------------------------- Sat May 19 17:24:48 UTC 2018 - containers-bugowner@suse.de - Commit 38975b8 by Flavio Castelli fcastelli@suse.com Add support for kube API auditing Allow users to enable kubernetes API server auditing feature. The auditing will produce an audit log file locally that can then be pushed to a central logging solution (eg: by using a fluentd daemonset running on the master nodes). By default there's no auditing in place. This is enabled only when the user provides a value for each one of the new pillars introduced by this commit. feature#kube-api-audit fate#325337 Signed-off-by: Flavio Castelli (cherry picked from commit 8fa612827897b78fe6ec4129e179a2fb410b4f07) ------------------------------------------------------------------- Sat May 19 10:12:07 UTC 2018 - containers-bugowner@suse.de - Commit c7e0bd0 by Flavio Castelli fcastelli@suse.com Provide configuration to transactional-update Fixes bsc#1088675 Signed-off-by: Flavio Castelli (cherry picked from commit b501d9fe5600ab711139dab215bb3e55c5de6fd7) ------------------------------------------------------------------- Fri May 18 07:41:35 UTC 2018 - containers-bugowner@suse.de - Commit f9117fe by Rafael Fernández López ereslibre@ereslibre.es Remove default grace period and timeout when draining a node. By default, the grace period is -1, or whatever the pod specifies on its `terminationGracePeriodSeconds` spec. The pod can know better than us what it needs to cleanly stop, and we don't need to apply arbitrary timeouts. If this is not specified, the default `terminationGracePeriodSeconds` value is 30 seconds. After this grace termination period, a SIGKILL will be sent to the process when evicting pods. Aside from this, we should have an "inifinite" timeout. Given that this process doesn't stall, it's safer to perform this operation until it succeeds. If we have proof that this is causing problems we should add a timeout, but in general the draining process should not hang. The alternative is in reality the real problem: if we timeout the draining process, it can happen that certain pods with remote volumes (nfs, rbd...) are never evicted, and when we go to restart the machine it hangs, because systemd fails to kill the processes when there are active mounts. Since there are no sensible defaults for the grace period and for the global timeout is better to let the first one to the pod definition, and the second one to just "infinite" until we really hit an issue because of this. Fixes: bsc#1085980 (cherry picked from commit 03d371fc489f4bd0e15da348b60390aa558daf76) ------------------------------------------------------------------- Thu May 17 21:32:13 UTC 2018 - containers-bugowner@suse.de - Commit a211c00 by Rafael Fernández López ereslibre@ereslibre.es Lower the per-request timeout when we are checking for successful query When we are waiting for some service to be up, if the request hangs for some reason, we want to retry at least several times. Without setting this value explicitly, it takes the default (`http_request_timeout` as 3600), what is way over our `wait_for` argument set at 300 seconds. By setting the default `http_request_timeout` to a more reasonable default when doing this kind of checks we can ensure that the request itself will timeout several times before we call it done. Fixes: bsc#1093540 Fixes: bsc#1093685 (cherry picked from commit 876f7c7f03c3c6c970ba6f81fa4c676d5ea43b03) ------------------------------------------------------------------- Thu May 17 14:59:12 UTC 2018 - containers-bugowner@suse.de - Commit d9a12a6 by Rafael Fernández López ereslibre@ereslibre.es Only remove the master grains if there are any masters to be updated. The `salt.function` call will be marked as failed if there were no minions to target. Make sure that we only run this step if we know that we'll have some targets available. Fixes: bsc#1093491 (cherry picked from commit b13d89a67142849ec3f40f56876a39dc5feba3f4) ------------------------------------------------------------------- Wed May 16 12:54:11 UTC 2018 - containers-bugowner@suse.de - Commit 252aa1b by Rafael Fernández López ereslibre@ereslibre.es Make HAProxy work as an http proxy instead of a tcp proxy. This allows us to add fine-grained timeouts depending on the endpoint being accessed or with what parameters (e.g. /log?follow=true should have no timeout as happens on the apiserver). /exec is another example, but in this case the protocol is upgraded to spdy. Fixes: bsc#1071994 (cherry picked from commit 442a76cad214f2308f6a1de0ddef8febca8074c8) ------------------------------------------------------------------- Tue May 15 10:23:17 UTC 2018 - containers-bugowner@suse.de - Commit b2d5f0a by Maximilian Meister mmeister@suse.de fix eviction-hard path feature#compute-resources bsc#1086185 Signed-off-by: Maximilian Meister (cherry picked from commit 4b37cb948e548be4e6f651d4281c6ea4b17e1b25) ------------------------------------------------------------------- Tue May 15 01:50:06 UTC 2018 - containers-bugowner@suse.de - Commit 1385b59 by Kiall Mac Innes kiall@macinnes.ie Add JUnit output (cherry picked from commit 177f7746a041b6573479925bb10626ca1ff4cb9e) Commit 07ab8b2 by Kiall Mac Innes kiall@macinnes.ie Update README with style check steps (cherry picked from commit 28e522e552becb4ca879b9c27e258224a1e5ec8d) Commit 73bb377 by Kiall Mac Innes kiall@macinnes.ie Fixup python code style issues (cherry picked from commit 248c2286093eceaef048bd29498c0b36ad7a572f) Commit 9cddf08 by Kiall Mac Innes kiall@macinnes.ie Add flake8 job (cherry picked from commit 4712a69ce28cf4e11796bc2b4d573e02ade64f4f) ------------------------------------------------------------------- Tue May 15 00:52:11 UTC 2018 - containers-bugowner@suse.de - Commit 21cd26b by Kiall Mac Innes kiall@macinnes.ie Add Housekeeping Job ------------------------------------------------------------------- Fri May 11 12:00:37 UTC 2018 - containers-bugowner@suse.de - Commit a21ae7d by Flavio Castelli fcastelli@suse.com Add missing cri-o removal states This is required to fix node removal on clusters using CRI-O as CRI. Fixes bsc#1092614 Signed-off-by: Flavio Castelli (cherry picked from commit 1657de5abbaec9734dbc5388c1403d361a006824) ------------------------------------------------------------------- Thu May 10 10:05:51 UTC 2018 - containers-bugowner@suse.de - Commit dde3f41 by Flavio Castelli fcastelli@suse.com kubelet: allow resource reservation Allow kubelet to take into account resource reservation and eviction threshold. == Resource reservation == It's possible to reserve resources for the `kube` and the `system` components. The `kube` component is the one including the kubernetes components: api server, controller manager, scheduler, proxy, kubelet and the container engine components (docker, containerd, cri-o, runc). The `system` component is the `system.slice`, basically all the system services: sshd, cron, logrotate,... By default don't specify any kind of resource reservation. Note well: when the resource reservations are in place kubelet will reduce the amount or resources allocatable by the node. However **no** enforcement will be done neither on the `kube.slice` nor on the `system.slice`. This is not happening because: * Resource enforcement is done using cgroups. * The slices are created by systemd. * systemd doesn't manage all the available cgroups yet. * kubelet tries to manage cgroups that are not handled by systemd, resulting in the kubelet failing at startup. * Changing the cgroup driver to `systemd` doesn't fix the issue. Moreover enforcing limits on the `system` and the `kube` slices can lead to resource starvation of core components of the system. As advised even by the official kubernetes docs, this is something that only expert users should do only after extensive profiling of their nodes. Finally, even if we wanted to enforce the limits, the right place would be systemd (by tuning the slice settings). For more information see the official documentation: https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ == Eviction threshold == By default no eviction threshold is set. bsc#1086185 Signed-off-by: Flavio Castelli (cherry picked from commit bcf54151819aba34b9535650dc3787031a41d742) ------------------------------------------------------------------- Thu May 10 10:04:51 UTC 2018 - containers-bugowner@suse.de - Commit ccafda4 by Flavio Castelli fcastelli@suse.com Make crictl handling more robust Some of our states are now depending on `crictl` tool. All these states have to depend on the `kubelet service.running` one, otherwise the `crictl` socket won't be available and the state will fail. Also, with these changes, the "blame" of a failure should point directly to the guilty (`kubelet` service not running for whatever reason) instead of falling on the `haproxy` one. Finally, the check looking for `crictl` socket has been changed to ensure the socket file exists and the service is actually listening. This will help with bugs like bsc#1091419 Signed-off-by: Flavio Castelli (cherry picked from commit e286f9bae8d5b0d3510e712cce4bd9dc24129d90) ------------------------------------------------------------------- Wed May 9 10:42:46 UTC 2018 - containers-bugowner@suse.de - Commit 4034199 by Maximilian Meister mmeister@suse.de add condition to KUBE_ADMISSION_CONTROL bsc#1092140 Signed-off-by: Maximilian Meister (cherry picked from commit 964deeee89594ebfab76ecb18a032fc84e2ef2e2) Commit 3381aff by Maximilian Meister mmeister@suse.de fix conflicting sls id's they need to be globally unique orch error happened when setting psp to false in params.sls partially fixes https://bugzilla.suse.com/show_bug.cgi?id=1092140 bsc#1092140 Signed-off-by: Maximilian Meister (cherry picked from commit eaab500fef59dc8908f86724676a6088e8cff133) ------------------------------------------------------------------- Wed May 9 10:41:25 UTC 2018 - jmassaguerpla@suse.com - Remove master.tar.gz tarball and use release-3.0.tar.gz release#3.0 ------------------------------------------------------------------- Wed May 9 10:32:13 UTC 2018 - containers-bugowner@suse.de - Commit a637496 by Maximilian Meister mmeister@suse.de make VERSION stable release#3.0 Signed-off-by: Maximilian Meister ------------------------------------------------------------------- Mon May 7 09:18:12 UTC 2018 - containers-bugowner@suse.de - Commit 8388498 by Alvaro Saurin alvaro.saurin@gmail.com Try to resist existent data in the mine https://bugzilla.suse.com/show_bug.cgi?id=1091361 bsc#1091361 ------------------------------------------------------------------- Thu May 3 10:02:57 UTC 2018 - containers-bugowner@suse.de - Commit 0294ed9 by Alvaro Saurin alvaro.saurin@gmail.com Do not try to use the mine when we can get the same information with a module. (cherry picked from commit dfd3b8a6a65c7d969466b09a1f20536a525ae42a) bsc#1091077 ------------------------------------------------------------------- Wed May 2 11:57:18 UTC 2018 - containers-bugowner@suse.de - Commit 17e9533 by Kiall Mac Innes kiall@macinnes.ie Harden the waiting for CRI socket to become active * Allow more time for the CRI socket to become active - 20 seconds * Explicitly fail if the socket does not become active within this time. Related to bsc#1091419 ------------------------------------------------------------------- Sun Apr 29 13:31:42 UTC 2018 - containers-bugowner@suse.de - Commit c03b41d by Alvaro Saurin alvaro.saurin@gmail.com Retry the `wait_for_http` when waiting for the API server. Use the same cleanup.post-orchestration that tyhe forces removal uses. Some other removal orchestration fixes and improvements. feature#node_removal ------------------------------------------------------------------- Fri Apr 27 15:15:36 UTC 2018 - containers-bugowner@suse.de - Commit 03242db by Kiall Mac Innes kiall@macinnes.ie Fix caasp_etcd.get_member_id error handling caasp_etcd.get_member_id was referencing a variable that doesn't exist. ------------------------------------------------------------------- Thu Apr 26 09:56:06 UTC 2018 - containers-bugowner@suse.de - Commit c3b81a6 by Flavio Castelli fcastelli@suse.com Ensure swap is disabled before kubelet is started We have to ensure the swap state is executed before the kubelet service is started, otherwise kubelt won't run and this will lead to issues like the ones causing bsc#1090337 Signed-off-by: Flavio Castelli ------------------------------------------------------------------- Wed Apr 25 12:10:02 UTC 2018 - containers-bugowner@suse.de - Commit 24bea3d by Nirmoy Das ndas@suse.de cni: add cilium as alternate to flannel plugin ------------------------------------------------------------------- Tue Apr 24 15:58:27 UTC 2018 - containers-bugowner@suse.de - Commit 1fd2a98 by Alvaro Saurin alvaro.saurin@gmail.com Remove leftover file feature#node_removal ------------------------------------------------------------------- Tue Apr 24 09:21:14 UTC 2018 - containers-bugowner@suse.de - Commit e1b9c75 by Kiall Mac Innes kiall@macinnes.ie Update tiller tag to 2.8.2 This matches the tag used in the updated image via SR#162727. ------------------------------------------------------------------- Tue Apr 24 08:42:18 UTC 2018 - containers-bugowner@suse.de - Commit 3e70e4f by Alvaro Saurin alvaro.saurin@gmail.com Use get_with_expr() feature#node_removal Commit b4d09dd by Alvaro Saurin alvaro.saurin@gmail.com Convert integers in the pillar to real integers. Unit tests for the get_pillar() function. See https://trello.com/c/O7daOErL feature#node_removal Commit 0d65d79 by Alvaro Saurin alvaro.saurin@gmail.com Fix: do not include the current node in the list of endpoints when adding a new member. Unit tests for the etcd modoule. See https://trello.com/c/O7daOErL feature#node_removal Commit 399f7ea by Alvaro Saurin alvaro.saurin@gmail.com Try to resist unresponsive nodes when removing a node. * the replacement will not be chosen from the unresponsive nodes * affected nodes will exclude them too. Possibility to skip any action on the target (with the `skip` pillar), so we can remove unresponsive targets while still looking for replacements. See https://trello.com/c/O7daOErL feature#node_removal ------------------------------------------------------------------- Tue Apr 24 07:58:31 UTC 2018 - containers-bugowner@suse.de - Commit f80f752 by Alvaro Saurin alvaro.saurin@gmail.com Don't to remove some things that are not so important. feature#node_removal ------------------------------------------------------------------- Mon Apr 23 12:03:59 UTC 2018 - containers-bugowner@suse.de - Commit 44798f4 by Rafael Fernández López ereslibre@ereslibre.es Use `expr_form` instead of `tgt_type` until we update salt This is producing an error on our current salt version: `Rendering SLS 'base:cleanup.remove-post-orchestration' failed: Jinja error: get() got an unexpected keyword argument 'tgt_type'` Go back to using `expr_form` until we update. feature#deployment-stability ------------------------------------------------------------------- Mon Apr 23 10:16:50 UTC 2018 - containers-bugowner@suse.de - Commit 352e4f5 by Rafael Fernández López ereslibre@ereslibre.es Always remove the "we are removing a machine" grain from the cluster Even if the `removal` orchestration has failed, we want to remove this grain from the cluster, or the subsequent `etc-hosts` orchestrations won't be executed if a removal failed. feature#deployment-stability ------------------------------------------------------------------- Mon Apr 23 07:43:54 UTC 2018 - containers-bugowner@suse.de - Commit f2190ca by Alvaro Saurin alvaro.saurin@gmail.com Instead of running things on the forced-removal orchestration, move actions to SLS files (so they can be shared with the regular removal orchestration). feature#node_removal ------------------------------------------------------------------- Sat Apr 21 10:16:14 UTC 2018 - containers-bugowner@suse.de - Commit 6d5dcda by Federico Ceratto federico.ceratto@suse.de Stop using __opts__ and os_data() bsc#1087115 ------------------------------------------------------------------- Fri Apr 20 08:47:24 UTC 2018 - containers-bugowner@suse.de - Commit ec9c37c by Flavio Castelli fcastelli@suse.com Introduce feature-gates pillar Allow feature gates to be toggled via a dedicated pillar. feature#feature-gates ------------------------------------------------------------------- Thu Apr 19 09:00:00 UTC 2018 - containers-bugowner@suse.de - Commit 165baf2 by Federico Ceratto federico.ceratto@suse.de Switch caasp_nodename to using __opts__ bsc#1087115 ------------------------------------------------------------------- Wed Apr 18 15:23:47 UTC 2018 - containers-bugowner@suse.de - Commit 52b61c2 by Flavio Castelli fcastelli@suse.com crio: fix upgrade orchestration Ensure everything is fine on the admin node feature#crio Signed-off-by: Flavio Castelli Commit 33256f0 by Flavio Castelli fcastelli@suse.com crio: cleanup code Several changes to reflect the feedback got on the pull request. feature#crio Signed-off-by: Flavio Castelli Commit f62aaec by Flavio Castelli fcastelli@suse.com Do not rely on salt virtual_subtype grain The `virtual_subtype` grain cannot be used to identify salt minions that are running inside of containers started by kubernetes. The salt core code sets this grain to `Docker` by looking at the cgroup hierarchy of PID 1 on the minion. On regular docker container (not managed by kubernetes!) the cgroup hierarchy includes a `docker` slice. However all the containers started by kubelet are placed under the `kubepods` slice. Right now the only salt minion running inside of a container is the `ca` one, which can be easily identified by looking at its roles. This commit changes our salt states to use roles instead of the unreliable `virtual_subtype` grain. feature#crio Signed-off-by: Flavio Castelli Commit 569c9aa by Flavio Castelli fcastelli@suse.com Extend motd Show information about the container runtime used on the node. feature#crio Signed-off-by: Flavio Castelli Commit 1bae9eb by Flavio Castelli fcastelli@suse.com Remove unused cri abstractions cri-o doesn't have yet a way to copy files from the host into its running containers. Fortunately this feature is required only on the admin node, which is still using docker. This commit removes some of the abstractions introduced to be able to copy files into running containers. We will revert this commit later on, once we migrate the admin node to use cri-o. feature#crio Signed-off-by: Flavio Castelli Commit 0c7a2b2 by Flavio Castelli fcastelli@suse.com Fix issue caused by velum pillar override Pillars set by velum are going to override what is set via the `salt/pillars` files. That caused all the nodes to be using cri-o. The following code enforces 'docker' to be used for all the nodes with a certain role (eg: the admin and the ca ones). feature#crio Signed-off-by: Flavio Castelli Commit 72e93b8 by Flavio Castelli fcastelli@suse.com Full support of cri-o Allow to deploy new SUSE CaaS Platform clusters using cri-o as a container runtime instead of docker. The cluster will keep using docker on the admin node, while all the other nodes are going to use cri-o. It's not possible to have mixed environments, all nodes have to use the same container runtime. The CRI can be chosen by setting the value of the `cri:name` pillar, which is defined inside of the `pillar/cri.sls` file. By default `docker` is being used. feature#crio Signed-off-by: Flavio Castelli Commit 8bc9d1b by Flavio Castelli fcastelli@suse.com Remove e2e image puller manifest This is no longer used. Commit e4b586a by Alvaro Saurin alvaro.saurin@gmail.com Added support for the CRIO containers runtime ------------------------------------------------------------------- Wed Apr 18 08:17:13 UTC 2018 - containers-bugowner@suse.de - Commit 902cc67 by Kiall Mac Innes kiall@macinnes.ie Ensure salt master and api configs are complete This moves the external_auth section over to 50-master.conf, as this is needed by the salt-master process, and duplicates `user: root` from 50-master.conf to 50-api.conf - which allows salt-api to start and function without it reading 50-master.conf ------------------------------------------------------------------- Wed Apr 18 08:12:11 UTC 2018 - containers-bugowner@suse.de - Commit 24835c2 by Alvaro Saurin alvaro.saurin@gmail.com Fix: always remove the "we-are-removing-a-node" cluster-wide grain. Make sure we flush the mine (for the target) after removing the target's key. feature#node_removal ------------------------------------------------------------------- Wed Apr 18 07:47:02 UTC 2018 - containers-bugowner@suse.de - Commit 9d782ee by Michal Jura mjura@suse.com Add cinder volume type to cluster user policy, bsc#1089863 ------------------------------------------------------------------- Wed Apr 18 07:40:33 UTC 2018 - containers-bugowner@suse.de - Commit 32b868a by Rafael Fernández López ereslibre@ereslibre.es Remove unneeded variables feature#code-cleanup ------------------------------------------------------------------- Tue Apr 17 16:44:28 UTC 2018 - containers-bugowner@suse.de - Commit 2355abd by Rafael Fernández López ereslibre@ereslibre.es Add force removal orchestration This orchestration will try to unregister a node on a best-effort basis, and is considered to always succeed. feature#force-node-removal ------------------------------------------------------------------- Tue Apr 17 08:27:34 UTC 2018 - containers-bugowner@suse.de - Commit 009516d by Federico Ceratto federico.ceratto@suse.de Lowercase hostnames bsc#1087115 ------------------------------------------------------------------- Mon Apr 16 07:39:56 UTC 2018 - containers-bugowner@suse.de - Commit 5e89e09 by Thorsten Kukuk kukuk@thkukuk.de Add pyroute2 and etcd python modules as Requires (moved from patterns) Commit 026ea39 by Thorsten Kukuk kukuk@thkukuk.de Use python3 for post SLE12 and kubic as image name for Factory ------------------------------------------------------------------- Fri Apr 13 16:20:21 UTC 2018 - containers-bugowner@suse.de - Commit 236835f by Alvaro Saurin alvaro.saurin@gmail.com Code cleanup: use `caasp_grains.get` instead of a local version. feature#code_cleanup ------------------------------------------------------------------- Fri Apr 13 12:11:18 UTC 2018 - containers-bugowner@suse.de - Commit 0e7d745 by Alvaro Saurin alvaro.saurin@gmail.com Configure taints/labels on the replacement node Fix typo feature#node_removal ------------------------------------------------------------------- Fri Apr 13 11:44:45 UTC 2018 - containers-bugowner@suse.de - Commit 69d271d by Rafael Fernández López ereslibre@ereslibre.es Remove unneeded includes `ca-cert` and `cert` for `velum/init.sls` and `ldap/init.sls` feature#deployment-stability ------------------------------------------------------------------- Fri Apr 13 11:04:32 UTC 2018 - containers-bugowner@suse.de - Commit 1de5846 by Kiall Mac Innes kiall@macinnes.ie Add PodSecurityPolicy Support Add support for PodSecurityPolicy's, allowing us to disable use of the hostPath volume type. This change adds 2 PSP's: * unprivileged (Default assigned to all users) The unprivileged PodSecurityPolicy is intended to be a reasonable compromise between the reality of Kubernetes workloads, and suse:caasp:psp:privileged. By default, we'll grant this PSP to all users and service accounts. * privileged The privileged PodSecurityPolicy is intended to be given only to trusted workloads. It provides for as few restrictions as possible and should only be assigned to highly trusted users. Fixes bsc#1047535 ------------------------------------------------------------------- Wed Apr 11 07:39:47 UTC 2018 - containers-bugowner@suse.de - Commit 489cbef by Alvaro Saurin alvaro.saurin@gmail.com Fix race condition on update-etc-hosts fix#update-etc-hosts ------------------------------------------------------------------- Tue Apr 10 11:15:39 UTC 2018 - containers-bugowner@suse.de - Commit 0ef0581 by Alvaro Saurin alvaro.saurin@gmail.com * Do some code cleanups in caasp_etcd.py by using the same logic for getting etcd replacements as for getting additional etcd servers when bootstrapping. * Move most of the removal logic to a caasp_nodes.py Python module, as Jinja is not a proper language... * Add the corresponding unit tests for this new Python code. * Do not be so strict when finding a replacement: if the replacement is not valid for a k8s master, do not make it unsuitable for etcd too. * Use some basic k8s master replacement finder. * Try to use some common logging functions * Refactor out the grains.get code to a new caasp_grains.py module (as it is shared by several custom modules) See https://trello.com/c/O7daOErL feature#node_removal ------------------------------------------------------------------- Tue Apr 10 07:54:00 UTC 2018 - containers-bugowner@suse.de - Commit c189bca by Alvaro Saurin alvaro.saurin@gmail.com Try to resist to transient node failures on updates See https://trello.com/c/irviWd1m feature#update_on_node_failures ------------------------------------------------------------------- Mon Apr 9 08:55:51 UTC 2018 - containers-bugowner@suse.de - Commit caa100b by Alvaro Saurin alvaro.saurin@gmail.com Change the meaning of some grains: * removal_in_progress -> node_removal_in_progress (only for the node that is being removed) * addition_in_progress -> node_addition_in_progress (only for the node that is being added) * removal_in_progress: cluster-wide grain for marking that a removal is being done. This should avoid conflicts with the etc-hosts-update orchestration... https://bugzilla.suse.com/show_bug.cgi?id=1087108 bsc#1087108 ------------------------------------------------------------------- Fri Apr 6 07:57:48 UTC 2018 - containers-bugowner@suse.de - Commit 3a529ab by Alvaro Saurin alvaro.saurin@gmail.com Reject keys of removed nodes instead of just deleting them. https://bugzilla.suse.com/show_bug.cgi?id=1087062 bsc#1087062 ------------------------------------------------------------------- Thu Apr 5 08:41:48 UTC 2018 - containers-bugowner@suse.de - Commit ae4018a by Rafael Fernández López ereslibre@ereslibre.es Force drain when trying to drain a node When trying to drain a node we can get an error if the kubelet is running a pod created by local manifests (manifests living in the local filesystem): ``` caasp-admin:~ # kubectl drain --ignore-daemonsets caasp-worker-1 node "caasp-worker-1" cordoned error: unable to drain node "caasp-worker-1", aborting command... There are pending nodes to be drained: caasp-worker-1 error: pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet (use --force to override): haproxy-caasp-worker-1 ``` As opposed to: ``` caasp-admin:~ # kubectl drain --force --ignore-daemonsets caasp-worker-1 node "caasp-worker-1" already cordoned WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: haproxy-caasp-worker-1; Ignoring DaemonSet-managed pods: kube-flannel-vklfc node "caasp-worker-1" drained ``` Related: bsc#1085980 ------------------------------------------------------------------- Tue Apr 3 10:26:04 UTC 2018 - containers-bugowner@suse.de - Commit c7ee6be by Rafael Fernández López ereslibre@ereslibre.es Wait for deployments during the orchestration time. Additionally to other checks, we should also consider the orchestration done once that the expected pods are running. feature#deployment-stability ------------------------------------------------------------------- Tue Mar 27 10:03:43 UTC 2018 - containers-bugowner@suse.de - Commit 043a686 by Kiall Mac Innes kiall@macinnes.ie Extend certificates to one year lifespan 100 days is a very short lifespan, lets bump this to one year - a much more common value for certificate lifetime. Related to bsc#1082722 ------------------------------------------------------------------- Thu Mar 22 16:53:56 UTC 2018 - containers-bugowner@suse.de - Commit 0901ff0 by Kiall Mac Innes kiall@macinnes.ie Increase Kube-DNS replicas to 3 Having only a single Kube-DNS replica means that, during upgrades or other failure scenarios, Kube-DNS will not be functional. A value of 3 matches what we use for Dex. Commit 2c42773 by Kiall Mac Innes kiall@macinnes.ie Dex should not have cluster-admin Dex does not require cluster admin access. Instead, it should use a new role defined with just the permissions Dex requires. Commit 38e654d by Kiall Mac Innes kiall@macinnes.ie Kube-DNS should not have cluster-admin Kubernetes DNS service does not require cluster admin access. Instead, it should use the build in system:kube-dns role. Commit 9dec359 by Kiall Mac Innes kiall@macinnes.ie Remove duplicated Dex ClusterRoleBinding The ClusterRoleBinding's for Dex were duplicated - this removes the extra copy. Commit 0aebc0d by Kiall Mac Innes kiall@macinnes.ie Match addons/{dns,tiller} patterns to addons/dex This pattern is cleaner, and lets Kubernetes do more of the hard work related to applying and updating manifests changes. This will be further extended to CNI/flannel soon. ------------------------------------------------------------------- Thu Mar 22 11:54:08 UTC 2018 - containers-bugowner@suse.de - Commit 3b3f0ae by Rafael Fernández López ereslibre@ereslibre.es Refresh modules before we call to any `sls`, they might use undiscovered modules Commit 8b49308 by Rafael Fernández López ereslibre@ereslibre.es When we explicitly run `haproxy` sls in the update, run `etc-hosts` too. During a rename, it might happen that `haproxy` refuses to start because it cannot resolve the new names `nodename.infra.caasp.local` in the configuration because its `/etc/hosts` file hasn't been updated yet. ------------------------------------------------------------------- Wed Mar 21 17:10:41 UTC 2018 - containers-bugowner@suse.de - Commit 0926982 by Kiall Mac Innes kiall@macinnes.ie Add flannel readiness/liveness probe This makes sure flannel has at least reached the point where it starts the healthz API endpoint. However, that point in the flannel code is *very* early and not all that useful for actual health checking. Additionally, as long as the HTTP gorouting is running, healthz will *always* respond with a 200. It performs no actual health checking. Even still, lets include the probe. If flannel gets better health checking, it will be enabled for us, on the other hand, if flannel doesn't get better health checking, it's still *very slightly* useful to know that flannel has at least reached this point in it's code. ------------------------------------------------------------------- Wed Mar 21 17:06:31 UTC 2018 - containers-bugowner@suse.de - Commit 4259116 by Rafael Fernández López ereslibre@ereslibre.es Wait for dex on the admin node before calling the orchestration done When we finish the orchestration all bits and pieces should be working as expected. Wait for the haproxy on the admin node to be correctly pointing to dex before finishing the orchestration. ------------------------------------------------------------------- Wed Mar 21 08:43:52 UTC 2018 - containers-bugowner@suse.de - Commit 113a807 by Rafael Fernández López ereslibre@ereslibre.es If no replacement provided do not ask for nonexistent states. If no replacement is provided, `sync-all` was trying to refer to states that didn't exist because those states also were wrapped with a `replacement` guard. Commit f6d8787 by Rafael Fernández López ereslibre@ereslibre.es Always set `replacement_provided` variable Salt was complaining that this variable didn't exist in the `orch.removal` orchestration when removing a master when no replacement was provided. ------------------------------------------------------------------- Fri Mar 16 17:05:34 UTC 2018 - containers-bugowner@suse.de - Commit 30b9ae5 by Kiall Mac Innes kiall@macinnes.ie Dex: Delay liveness probe in addition to readiness probe Delay the liveness probe by 30 seconds, matching the readiness probe. ------------------------------------------------------------------- Fri Mar 16 16:57:20 UTC 2018 - containers-bugowner@suse.de - Commit 753978f by Rafael Fernández López ereslibre@ereslibre.es Use complete host references on haproxy configuration This avoids an incompatibility on the admin node in which if the external fqdn field matched any of the master nodes host, haproxy would be checking 127.0.0.1:6444 for the apiserver for healthchecks. Now, we are using the internal infra domain suffix so we are sure we are referring to the real /etc/hosts entry with the ip address of the target machines. ------------------------------------------------------------------- Fri Mar 16 16:56:19 UTC 2018 - containers-bugowner@suse.de - Commit 9f06d7d by Rafael Fernández López ereslibre@ereslibre.es PCRE grain expressions only allow the regexp on the value side. Fix PCRE grain query expressions so they are matching what we expect. ``` caasp-admin:~ # docker exec -it 06bf salt -P 'bootstrap_complete:.*' cmd.run hostname admin: caasp-admin 6b5cb85d20f94f6eb813449b228cfe13: caasp-worker-1 4c0e4d31bc754369940ffcbae28e2f0a: caasp-worker-0 cb92123fa85d4170807e0aa24573501b: caasp-master-0 66d5844bc5f14d1480896b1bc234dd92: caasp-master-1 3f3f505c6eb3464e8a08cc0ae6fbc8f4: caasp-master-2 caasp-admin:~ # docker exec -it 06bf salt -P 'bootstrap_.*:true' cmd.run hostname No minions matched the target. No command was sent, no jid was assigned. ERROR: No return received ``` ------------------------------------------------------------------- Thu Mar 15 21:22:34 UTC 2018 - containers-bugowner@suse.de - Commit afc91fe by Kiall Mac Innes kiall@macinnes.ie Wipe out our /etc/hosts changes before reboot This ensures the systemd/wicked logic is unaffected by our /etc/hosts changes. ------------------------------------------------------------------- Wed Mar 14 08:56:12 UTC 2018 - containers-bugowner@suse.de - Commit 292b025 by Kiall Mac Innes kiall@macinnes.ie Rename salt/dex -> salt/addons/dex Fundamentally, there is no difference between how dex is deployed and managed vs how kube-dns or tiller is deployed and managed. Lets treat them the same. ------------------------------------------------------------------- Wed Mar 14 08:55:10 UTC 2018 - containers-bugowner@suse.de - Commit e77e865 by Alvaro Saurin alvaro.saurin@gmail.com Node removal constraint: we must have at least one k8s minion https://trello.com/c/O7daOErL ------------------------------------------------------------------- Tue Mar 13 10:38:44 UTC 2018 - containers-bugowner@suse.de - Commit 83ae5d3 by Kiall Mac Innes kiall@macinnes.ie Add liveness/readiness probes to Dex deployment This will ensure Kubernetes waits for the pods to become ready before starting to send them traffic, which should in turn prevent the orchestration proceeding and bootstrap completing until we have at least one working Dex pod Fixes bsc#1062542 ------------------------------------------------------------------- Mon Mar 12 09:55:53 UTC 2018 - containers-bugowner@suse.de - Commit 0ebaf16 by Maximilian Meister mmeister@suse.de cmd has moved to its own state for the proxy config require the pkg instead to make sure that the docker requisite is met Signed-off-by: Maximilian Meister ------------------------------------------------------------------- Fri Mar 9 11:15:27 UTC 2018 - containers-bugowner@suse.de - Commit 1427b2f by Rafael Fernández López ereslibre@ereslibre.es When populating the cache, don't fail if this fails for some reason. There's a race condition in which the cache directory does not exist, but when tried to be created it has already been created by something else, and an exception is raised, stopping the execution. When populating the cache, we don't really care if it was correctly populated or not in that *specific* call, so move on. Fixes: bsc#1084441 ------------------------------------------------------------------- Fri Mar 9 08:35:50 UTC 2018 - containers-bugowner@suse.de - Commit d0ce17c by Rafael Fernández López ereslibre@ereslibre.es Run the highstate on the admin after `sync_all` has been called. The admin node might use features not yet discovered, make sure we run `sync_all` before we enforce a `highstate` on the admin node too. ------------------------------------------------------------------- Tue Mar 6 16:13:45 UTC 2018 - containers-bugowner@suse.de - Commit d68ff78 by Rafael Fernández López ereslibre@ereslibre.es Remove the TODO message for using the standard `/opt/cni/bin`. Internal constraints won't allow us to use `/opt`, so we'll stick to `/var/lib/kubelet/cni/bin`. ------------------------------------------------------------------- Mon Mar 5 12:24:44 UTC 2018 - containers-bugowner@suse.de - Commit f129021 by Kiall Mac Innes kiall@macinnes.ie Ensure external_fqdn is not rendered to /etc/hosts if it's an IP ------------------------------------------------------------------- Mon Mar 5 10:50:19 UTC 2018 - containers-bugowner@suse.de - Commit 7464fda by Rafael Fernández López ereslibre@ereslibre.es Update `etcd` certificates before updating any machine We need to include the new SAN on all the certificates before restarting the first machine. Otherwise, this machine (a master) can find itself isolated without being able to contact any etcd member with the name it has (as the rest of the nodes haven't updated their certificates yet to also include the new name on the SAN). ------------------------------------------------------------------- Mon Mar 5 09:39:21 UTC 2018 - containers-bugowner@suse.de - Commit 453260e by Kiall Mac Innes kiall@macinnes.ie Add a suse:caasp:tiller-user ClusterRole This role represents the minimum RBAC requirements needed to make use of Helm's Tiller service. ------------------------------------------------------------------- Mon Mar 5 08:53:32 UTC 2018 - containers-bugowner@suse.de - Commit 4746436 by Rafael Fernández López ereslibre@ereslibre.es Make kubelet rename migration idempotent. If the new name already exists, also do nothing. A faulty update could make this script fail over and over again because of its `set -e` and the `kubectl create -f` command failing as the new node name already exists. ------------------------------------------------------------------- Fri Mar 2 17:13:56 UTC 2018 - containers-bugowner@suse.de - Commit d0dd517 by Michal Jura mjura@suse.com Add port number to flannel configuration template, bsc#1080608 ------------------------------------------------------------------- Fri Mar 2 13:04:01 UTC 2018 - containers-bugowner@suse.de - Commit f0923d0 by Michal Jura mjura@suse.com Cleaning nodes after removing them from CaaSP cluster (cherry picked from commit 3423788fdb4e14c98b46666cae5b01e9018f5692) ------------------------------------------------------------------- Thu Mar 1 16:31:16 UTC 2018 - containers-bugowner@suse.de - Commit 3f8a699 by Kiall Mac Innes kiall@macinnes.ie Add exit handler to kubelet/update-pre-orchestration.sh ------------------------------------------------------------------- Thu Mar 1 12:43:44 UTC 2018 - containers-bugowner@suse.de - Commit 94971ed by Rafael Fernández López ereslibre@ereslibre.es Do not produce empty `require` list. Make sure the require has at least the latest element that is always present. ------------------------------------------------------------------- Thu Mar 1 11:19:53 UTC 2018 - containers-bugowner@suse.de - Commit 5d32a43 by Michal Jura mjura@suse.com Add external API fqdn to /etc/hosts for Admin node, bsc#1080608 ------------------------------------------------------------------- Wed Feb 28 13:15:30 UTC 2018 - containers-bugowner@suse.de - Commit 07aada2 by Rafael Fernández López ereslibre@ereslibre.es Only remove the `kubelet:should_uncordon` grain when we actually uncordon the node. As part of the update process, we are cordoning the nodes, so they don't get new jobs when we are planning to reboot them. If an update fails for whatever reason, it might happen that we didn't uncordon the node, but removed the `kubelet:should_uncordon` grain. This would cause that subsequent retries will never uncordon the worker node again, because without this grain we'll think that this node was cordoned by the user and will not take any action. ------------------------------------------------------------------- Wed Feb 28 08:50:32 UTC 2018 - containers-bugowner@suse.de - Commit 49a98ec by Kiall Mac Innes kiall@macinnes.ie Ensure default labels and annotations are copied when renaming a node This copies the default labels and annotations from the "old" minion-id based node to the new hostname based node. Fixes bsc#1083113 ------------------------------------------------------------------- Tue Feb 27 14:42:41 UTC 2018 - containers-bugowner@suse.de - Commit cf52552 by Kiall Mac Innes kiall@macinnes.ie Update addon tolerations to allow execution on masters Update all addons, dex, kube-dns, etc to tolerate running on the tainted master nodes. Commit 3589595 by Kiall Mac Innes kiall@macinnes.ie Taint and Label Masters Masters should be tainted and labelled as masters, rather than setting these nodes as unschedulable. ------------------------------------------------------------------- Tue Feb 27 14:14:17 UTC 2018 - containers-bugowner@suse.de - Commit 1b37294 by Kiall Mac Innes kiall@macinnes.ie Don't allow docker restart/kill failures to fail the orch This avoids a race condition between docker ps and docker kill/restart. ------------------------------------------------------------------- Tue Feb 27 08:53:37 UTC 2018 - containers-bugowner@suse.de - Commit a2a9756 by Rafael Fernández López ereslibre@ereslibre.es Relax dex deployment anti-affinity. This can't be met on a cluster of n+2 size (n masters, 2 workers), as we are creating a deployment of 3. Let's relax the scheduling from required to preferred. ------------------------------------------------------------------- Mon Feb 26 13:22:00 UTC 2018 - containers-bugowner@suse.de - Commit 0ae2ecf by Kiall Mac Innes kiall@macinnes.ie Remove unnecessary check from rebootmgr state DevEnv no longer runs this way, so the check was doing nothing of value. ------------------------------------------------------------------- Mon Feb 26 10:51:49 UTC 2018 - containers-bugowner@suse.de - Commit 940766a by Kiall Mac Innes kiall@macinnes.ie Restart instead of reload container-feeder container-feeder is a oneshot service, where reload makes no sense and in unsupported. If this triggers, we ended up getting: salt-minion[2454]: [ERROR ] Command '['systemd-run', '--scope', 'systemctl', 'reload', 'container-feeder.service']' failed with return code: 3 salt-minion[2454]: Failed to reload container-feeder.service: Job type reload is not applicable for unit container-feeder.service. ------------------------------------------------------------------- Thu Feb 22 11:51:28 UTC 2018 - containers-bugowner@suse.de - Commit 4667ecd by Maximilian Meister mmeister@suse.de also add ldap to etc-hosts to make sure it's persisted Signed-off-by: Maximilian Meister Commit 6429d6f by Maximilian Meister mmeister@suse.de add ldap.infra.caasp.local to the certificate feature#net-ldap-cert Signed-off-by: Maximilian Meister ------------------------------------------------------------------- Wed Feb 21 15:27:46 UTC 2018 - containers-bugowner@suse.de - Commit 0bca62e by Alvaro Saurin alvaro.saurin@gmail.com A very basic README on the file naming conventions ------------------------------------------------------------------- Fri Feb 16 20:09:08 UTC 2018 - containers-bugowner@suse.de - Commit f29c60a by Kiall Mac Innes kiall@macinnes.ie Comment out worker_threads salt setting With the recent kernel update in our SLE SP3 snapshot, meltdown and spectre mitigations have been brought in. As it stands, salt with 20 workers performs very slowly under this configuration. Commenting out the workers config value is a temporary fix to allow CI to continue to pass. ------------------------------------------------------------------- Fri Feb 16 14:02:54 UTC 2018 - containers-bugowner@suse.de - Commit 7b4d85e by Kiall Mac Innes kiall@macinnes.ie Velum Dash and API both attempt to bind to the same port It's not possible to reliably bind to 0.0.0.0:443 for one service, and 127.0.0.1:443 for another service. As such, we'll move velum-api over to 127.0.0.1:444 ------------------------------------------------------------------- Fri Feb 16 13:34:36 UTC 2018 - containers-bugowner@suse.de - Commit 355546f by Kiall Mac Innes kiall@macinnes.ie Add some additional logging to velum pillar module Add some logging to the Velum pillar module so we can see when it's get loaded by salt, and when it gets called by salt. ------------------------------------------------------------------- Thu Feb 15 16:32:30 UTC 2018 - containers-bugowner@suse.de - Commit 12e977b by Kiall Mac Innes kiall@macinnes.ie Increase haproxy timeouts from 50sec, to 120sec Some components have a 60 second timeout for salt request timeouts, e.g the salt-api server which is called by Velum. Increase this timeout to double their timeouts to allow the real failures to be disclosed. We'll likely want to rework how timeouts are handled soon accross all our components. ------------------------------------------------------------------- Thu Feb 15 12:28:10 UTC 2018 - containers-bugowner@suse.de - Commit f55acf6 by Kiall Mac Innes kiall@macinnes.ie Salt-API should log requests and timestamps Currently, salt-api logs nothing post-startup expect for failures. This is far from ideal when debugging, so we increase the level from warning to info, and prefix log lines with timestamps. ------------------------------------------------------------------- Thu Feb 15 09:13:06 UTC 2018 - containers-bugowner@suse.de - Commit 1706196 by Michal Jura mjura@suse.com Add python-pyOpenSSL requires for salt x509.crl_managed module ------------------------------------------------------------------- Tue Feb 13 09:28:01 UTC 2018 - containers-bugowner@suse.de - Commit d8bc095 by Rafael Fernández López ereslibre@ereslibre.es When executing a highstate of `apiserver` make sure that we check the local `apiserver` instance When executing the highstate make sure the `apiserver` we are checking is the local one, not *any* master through haproxy. Make haproxy more reliable. - Let it redispatch requests. - Really restart the service when the config changes. - Apply configuration before highstates with a small batch, so we control the restarts. - When the admin node's haproxy is restarted, wait for it to be back before going on. Wait for the apiserver to be up and responding behind HAProxy Fixes: bsc#1079460 ------------------------------------------------------------------- Mon Feb 12 16:10:39 UTC 2018 - containers-bugowner@suse.de - Commit 3f6c945 by Alvaro Saurin alvaro.saurin@gmail.com Remove the etcd discovery mechanism Mark all the etcd members of the cluster with the 'etcd' role before doing the update ------------------------------------------------------------------- Mon Feb 12 15:21:42 UTC 2018 - containers-bugowner@suse.de - Commit cbc22fb by Alvaro Saurin alvaro.saurin@gmail.com Make sure we do not crash on pillars that are not properly formatted. ------------------------------------------------------------------- Mon Feb 12 13:38:51 UTC 2018 - containers-bugowner@suse.de - Commit c194707 by Alvaro Saurin alvaro.saurin@gmail.com Remove the etcd discovery mechanism Mark all the etcd members of the cluster with the 'etcd' role before doing the update ------------------------------------------------------------------- Mon Feb 12 11:25:24 UTC 2018 - containers-bugowner@suse.de - Commit d85fb55 by Kiall Mac Innes kiall@macinnes.ie Move haproxy config to /etc/caasp/haproxy This avoids a conflict between the caasp-container-manifests package, and the haproxy package. ------------------------------------------------------------------- Thu Feb 8 17:55:45 UTC 2018 - containers-bugowner@suse.de - Commit 37fccd3 by Flavio Castelli fcastelli@suse.com Dex pods: introduce anti-affinity rule Our dex deployment creates 3 pods running the dex service. There are really high chances (or even certainty in the case of clusters made by 1 or 2 worker nodes) that all the dex pods end up running on the same node. This is bad from a HA perspective, plus we end up taking away resources from small clusters. With the following change we enforce the kubernetes scheduler to always spread the dex pods over different nodes. On small clusters (1 or 2 nodes) the deployment will be running with a lower number of replicas until new nodes are added. This doesn't cause our orchestration to fail. Adding new nodes at a later stage will allow the deployment to reach the desired replica size without any intervention from us or the user. Signed-off-by: Flavio Castelli ------------------------------------------------------------------- Thu Feb 8 17:35:30 UTC 2018 - containers-bugowner@suse.de - Commit b578f87 by Kiall Mac Innes kiall@macinnes.ie Dex: Avoid using the external_fqdn to reach dex In some environments, the external_fqdn is unreachable from inside the cluster - avoid using it where possible. ------------------------------------------------------------------- Wed Feb 7 17:24:14 UTC 2018 - containers-bugowner@suse.de - Commit 6a11de3 by Kiall Mac Innes kiall@macinnes.ie Use separate Dex clients for each actual client Previously Velum, CaaSP CLI, and Kubernetes all shared a single Dex client. From a security perspective, this was far from ideal. Update Dex with 3 clients, one for each actual client. Both the Velum and CaaSP CLI clients are allowed to issue tokens for the Kubernetes client. ------------------------------------------------------------------- Wed Feb 7 10:12:48 UTC 2018 - containers-bugowner@suse.de - Commit 3d63b18 by Joachim Gleissner jgleissner@suse.com Add pillar root for public cloud specific config ------------------------------------------------------------------- Tue Feb 6 17:49:24 UTC 2018 - containers-bugowner@suse.de - Commit e23fb43 by Flavio Castelli fcastelli@suse.com Mark the haproxy as critical pod Flag the haproxy pods providing connectivity to the API server as critical ones. This should force kubelet and the scheduler to never ever get rid of them. If these pods are killed to make more space for other ones, the node would not be able to talk with the API server making it useless. More details inside upstream doc: https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/ Signed-off-by: Flavio Castelli ------------------------------------------------------------------- Mon Feb 5 16:52:13 UTC 2018 - containers-bugowner@suse.de - Commit 21d9ab7 by Jordi Massaguer Pla jmassaguerpla@suse.de [packaging] Replace | by # in sed expression as % is reserved for rpm macros Signed-off-by: Jordi Massaguer Pla ------------------------------------------------------------------- Mon Feb 5 15:53:16 UTC 2018 - containers-bugowner@suse.de - Commit 0126b32 by Kiall Mac Innes kiall@macinnes.ie Namespace the roles and cluster roles we create When we create a role, rolebinding etc, we should namespace the names in order to make it obvious these are deployed as part of CaaSP, as well as to help ensure these are obviously part of CaaSP, not a stock part of Kubernetes. I've gone with a "suse:caasp:" prefix, which matches the "system:" prefix for built in roles/rolebindings/etc. ------------------------------------------------------------------- Mon Feb 5 10:28:39 UTC 2018 - containers-bugowner@suse.de - Commit 40731ca by Flavio Castelli fcastelli@suse.com Update our manifests to reflect kubernetes 1.8 changes * rbac has been promoted to stable * deploymen is now v1beta2 * deamonset is now v1beta2 Signed-off-by: Flavio Castelli ------------------------------------------------------------------- Fri Feb 2 16:30:33 UTC 2018 - containers-bugowner@suse.de - Commit 9ecb201 by Kiall Mac Innes kiall@macinnes.ie Remove old mis-named tiller deployment Commit a66edac by Nikhil Manchanda SlickNik@gmail.com helm should detect salt-installed tiller service The helm client looks for a tiller deployment called 'tiller-deploy' to establish if tiller is already installed in the cluster, or not. Update our salt install of tiller to use a deployment with the same name so that it will be recognized by the helm client as already being installed. Fixes: bsc#1066201 ------------------------------------------------------------------- Fri Feb 2 11:55:31 UTC 2018 - containers-bugowner@suse.de - Commit 5b2893d by Alvaro Saurin alvaro.saurin@gmail.com Do not try to remove some flannel file that cannot be removed, and remove some other instead ------------------------------------------------------------------- Fri Feb 2 10:42:01 UTC 2018 - containers-bugowner@suse.de - Commit cb27ba1 by Kiall Mac Innes kiall@macinnes.ie Update flannel image tag to match flannel version ------------------------------------------------------------------- Fri Feb 2 09:41:56 UTC 2018 - containers-bugowner@suse.de - Commit 2eb40f1 by Jordi Massaguer Pla jmassaguerpla@suse.de replace sle12 for tumbleweed if the package is building in tumbleweed ------------------------------------------------------------------- Fri Feb 2 09:16:38 UTC 2018 - containers-bugowner@suse.de - Commit 37e99c4 by Alvaro Saurin alvaro.saurin@gmail.com Use the same code convention for ids in the orchestration as all the other ids. Cleanup some files when updating CNI. ------------------------------------------------------------------- Thu Feb 1 15:53:55 UTC 2018 - containers-bugowner@suse.de - Commit cf53150 by Kiall Mac Innes kiall@macinnes.ie No longer use machine-id's as node names With CaaSP 3.0, we're introducing a requirement for machines to have valid+unique hostnames in order to allow for the K8S CPIs to function correctly. This means our generated hostname is no longer needed, as our environment requirements force operators to provision servers with unique hostnames. ------------------------------------------------------------------- Thu Feb 1 13:06:16 UTC 2018 - containers-bugowner@suse.de - Commit 4ba7007 by Kiall Mac Innes kiall@macinnes.ie Update dex binary name to caasp-dex ------------------------------------------------------------------- Wed Jan 31 13:10:03 UTC 2018 - containers-bugowner@suse.de - Commit 18743e6 by Kiall Mac Innes kiall@macinnes.ie Fix breakage introduced by docker update * Docker will no longer accept a `docker cp` over /etc/hosts * Fix docker package name ------------------------------------------------------------------- Wed Jan 31 12:58:43 UTC 2018 - containers-bugowner@suse.de - Commit 8b84809 by Flavio Castelli fcastelli@suse.com Remove contrib directory We don't need these files. Signed-off-by: Flavio Castelli ------------------------------------------------------------------- Thu Jan 25 17:42:48 UTC 2018 - containers-bugowner@suse.de - Commit dfd3b8a by Alvaro Saurin alvaro.saurin@gmail.com Replace the _macros/net by a Python module, so we can get rid of the Jinja limitations (specially when returning lists). Add a logging module (until we use a Salt version that includes it). ------------------------------------------------------------------- Thu Jan 25 15:42:24 UTC 2018 - containers-bugowner@suse.de - Commit b6105b1 by Rafael Fernández López ereslibre@ereslibre.es Early mark nodes requiring update reboot as update in progress. This will allow us to reduce the timeframe in which the update-etc-hosts orchestration can pop up, eventually running states on minions effectively taking their lock and making this orchestration fail. We don't want the update-etc-hosts orchestration to interfere with the main update orchestration. We'll release minion per minion grain when they are done, but let's block all of them at the very beginning. Fixes: bsc#1077086 ------------------------------------------------------------------- Wed Jan 24 15:44:47 UTC 2018 - containers-bugowner@suse.de - Commit 6fdc440 by Rafael Fernández López ereslibre@ereslibre.es Retry certificate generation This will make the certificate request to the CA more resilient to transient errors, in case of overload or any other reasons that make the CA slow when creating new requested certificates. Fixes: bsc#1070989 ------------------------------------------------------------------- Wed Jan 24 15:30:25 UTC 2018 - containers-bugowner@suse.de - Commit f19fbd4 by Rafael Fernández López ereslibre@ereslibre.es Do not remove flannel interface when updating 3.x Between minor updates on 3.x we can get a bad timing when removing the flannel.1 interface as the DaemonSet will start right after the worker reboot, and we could remove the interface when flannel thinks it exists and it goes to add arp entries to it, leading to a failure and to an invalid kubernetes networking status. ------------------------------------------------------------------- Fri Jan 19 14:47:07 UTC 2018 - containers-bugowner@suse.de - Commit d3a3bed by Nikhil Manchanda SlickNik@gmail.com Update salt to use 2.7.2 version of tiller Update the salt template for the tiller deployment to install the sles12/tiller:2.7.2 container image which is the latest version for this image. ------------------------------------------------------------------- Wed Jan 17 13:19:43 UTC 2018 - containers-bugowner@suse.de - Commit 9e358bb by Federico Ceratto federico.ceratto@suse.de Add swap disabling ------------------------------------------------------------------- Tue Jan 16 09:49:51 UTC 2018 - containers-bugowner@suse.de - Commit 02fa131 by Maximilian Meister mmeister@suse.de Configure docker via config file, not args docker can be configured via /etc/docker/daemon.json registries can be configured there too, but need to be in their own dedicated pillar as we need to map certificates to the registry names Signed-off-by: Maximilian Meister ------------------------------------------------------------------- Mon Jan 15 13:39:24 UTC 2018 - containers-bugowner@suse.de - Commit 73189f3 by Rafael Fernández López ereslibre@ereslibre.es Fix version to 3.0.0+dev ------------------------------------------------------------------- Thu Jan 11 16:55:34 UTC 2018 - containers-bugowner@suse.de - Commit 1215ced by Rafael Fernández López ereslibre@ereslibre.es Migrate CNI metadata on workers before doing anything else This does not give any chance for kubelets to try to request a new `podCIDR`. Also, fix node patching of the CNI migration Before restarting the master with the new configuration we migrate the workers to their expected `podCIDR` values, then we start with the general update procedure: masters first, then workers. ------------------------------------------------------------------- Thu Jan 11 12:53:29 UTC 2018 - containers-bugowner@suse.de - Commit f5e1dd3 by Alvaro Saurin alvaro.saurin@gmail.com Use a bath size for etcd setup equal to the number of etcd masters (bsc#1066695) Minor cleanups and a fix for a case where caasp_etcd.py could return 0. ------------------------------------------------------------------- Thu Jan 11 12:09:34 UTC 2018 - containers-bugowner@suse.de - Commit b8bff11 by Kiall Mac Innes kiall@macinnes.ie Remove discovered IP addresses from certs As the discovered IP addresses are not static, that we don't maintain that the certs are updated+services are reloaded upon cert change, that we're including all IPs - even 127.0.0.1 - in this list, and that we don't make use of any of these SAN's, we should remove them. ------------------------------------------------------------------- Tue Jan 9 09:56:55 UTC 2018 - containers-bugowner@suse.de - Commit 94e697f by Rafael Fernández López ereslibre@ereslibre.es Only uncordon nodes that were cordoned because of our own processes Fix kubelet highstate to uncordon the node only if we did cordon it by one of our processes (like an update). Without this patch, adding new nodes or performing an update would uncordon all nodes unconditionally, without taking into account if a user had a node cordoned for some reason (e.g. hardware failures or other reasons). Do not uncordon those nodes, keep them cordoned. Fixes: bsc#1050017 ------------------------------------------------------------------- Mon Jan 8 09:04:17 UTC 2018 - containers-bugowner@suse.de - Commit 208a0da by Alvaro Saurin alvaro.saurin@gmail.com Let flannel calculate the Max and Min subnet from other parameters we are providing. More documentation on the flannel configuration. ------------------------------------------------------------------- Fri Dec 22 15:02:22 UTC 2017 - containers-bugowner@suse.de - Commit cc2aae4 by Rafael Fernández López ereslibre@ereslibre.es Do not check if we need to uncordon this node depending on its state. The `onlyif` section can fail its check (without retrial opportunity), making the whole uncordon process to abort, when we really want to uncordon a node. In the future, we need to keep track of cordoned nodes by the update so we only uncordon those, leaving cordoned the nodes that were cordoned by the user. In any case, for this issue, `kubectl` will be smart enough: - For a cordoned node, uncordoning: ``` ~ KUBECONFIG=~/Downloads/kubeconfig kubectl uncordon 7a4f4985eaed4f519e27900ece559b8e.infra.caasp.local node "7a4f4985eaed4f519e27900ece559b8e.infra.caasp.local" uncordoned ~ echo $? 0 ``` - For an uncordoned node, uncordoning again: ``` ~ KUBECONFIG=~/Downloads/kubeconfig kubectl uncordon 7a4f4985eaed4f519e27900ece559b8e.infra.caasp.local node "7a4f4985eaed4f519e27900ece559b8e.infra.caasp.local" already uncordoned ~ echo $? 0 ``` We know we want to uncordon the node, let's do that directly, and it will just succeed in any case (unless the process of uncordoning fails for some reason, and in that case we have the `retries` in place). Fixes: bsc#1073919 Fixes: #336 ------------------------------------------------------------------- Fri Dec 22 13:01:33 UTC 2017 - containers-bugowner@suse.de - Commit 628ba55 by Alvaro Saurin alvaro.saurin@gmail.com Explicitly pass the kubeconfig file to kubectl ------------------------------------------------------------------- Thu Dec 21 12:53:20 UTC 2017 - containers-bugowner@suse.de - Commit 3c64b88 by Rafael Fernández López ereslibre@ereslibre.es Add beacon to notify network changes only on the default network interface Fixes: bsc#1063709 ------------------------------------------------------------------- Mon Dec 18 18:37:20 UTC 2017 - containers-bugowner@suse.de - Commit 1863c06 by Rafael Fernández López ereslibre@ereslibre.es Bump dex version ------------------------------------------------------------------- Tue Dec 12 15:13:24 UTC 2017 - containers-bugowner@suse.de - Commit 8fb3e79 by Alvaro Saurin alvaro.saurin@gmail.com Use a sanitized version of pillar.get ------------------------------------------------------------------- Wed Nov 29 11:51:37 UTC 2017 - containers-bugowner@suse.de - Commit c91add1 by Kiall Mac Innes kiall@macinnes.ie Remove empty state from etc-hosts orch The final state in the etc-hosts orch was not actually calling anything, and hasn't been for quite a while. Lets remove it, so that the error it logs can be finally be gone! ------------------------------------------------------------------- Wed Nov 29 11:15:18 UTC 2017 - containers-bugowner@suse.de - Commit fd431b6 by Alvaro Saurin alvaro.saurin@gmail.com Run some things in only one master instead of in all the masters in the cluster. ------------------------------------------------------------------- Wed Nov 29 09:29:04 UTC 2017 - containers-bugowner@suse.de - Commit 20070dc by Alvaro Saurin alvaro.saurin@gmail.com In the certs macros, do not assume "names" are always names and "ips" are always IPs: just filter with the "is_ip" filter. Minor shortcuts in the arguments. Fixes: bsc#1069205 ------------------------------------------------------------------- Tue Nov 28 17:50:25 UTC 2017 - containers-bugowner@suse.de - Commit af1428a by Rafael Fernández López ereslibre@ereslibre.es Never write `None` if we get `null` on the pillar override Instead, we write an empty string, because we don't intend to write `None` on the configuration file. ------------------------------------------------------------------- Tue Nov 28 16:08:14 UTC 2017 - containers-bugowner@suse.de - Commit 4ed69ee by Kiall Mac Innes kiall@macinnes.ie Support IPs as Kube external FQDN in /etc/hosts Currently, we assumed external names were FQDNs. When an IP was used instead, we would generate an incorrect /etc/hosts. bsc#1070154 ------------------------------------------------------------------- Mon Nov 27 14:17:43 UTC 2017 - containers-bugowner@suse.de - Commit 73a9fd3 by Rafael Fernández López ereslibre@ereslibre.es Preserve haproxy configurations for Velum * Handle `haproxy` configuration. * Generate `pem` certificates, that include the certificate and private key. * Remove `velum` container restart. ------------------------------------------------------------------- Mon Nov 27 13:03:57 UTC 2017 - containers-bugowner@suse.de - Commit 182c840 by Alvaro Saurin alvaro.saurin@gmail.com Use some Jinja macros for getting the default interface's IP. (bsc#1058079) Get rid of our custom grain. ------------------------------------------------------------------- Mon Nov 27 10:46:11 UTC 2017 - containers-bugowner@suse.de - Commit f215a10 by Rafael Fernández López ereslibre@ereslibre.es Include `Internal Dashboard FQDN/IP` value in the LDAP certificate Since Dex will connect to LDAP using this FQDN/IP, make sure that the TLS handshake will succeed by regenerating the certificate early in the orchestration, so it includes this FQDN/IP in the SAN extensions of the LDAP certificate. Fixes: bsc#1069175 ------------------------------------------------------------------- Thu Nov 23 14:09:53 UTC 2017 - containers-bugowner@suse.de - Commit ef4bd9b by Rafael Fernández López ereslibre@ereslibre.es Sync _pillar modules only. We want to sync the pillars on the master first. ------------------------------------------------------------------- Tue Nov 21 09:26:03 UTC 2017 - containers-bugowner@suse.de - Commit 072a014 by Rafael Fernández López ereslibre@ereslibre.es Introduce Velum pillar * Use Velum pillar that serves json content * Cache the result if it differs from what we got * Serve the cached result if a connection problem happens Fixes: bsc#1069145 ------------------------------------------------------------------- Mon Nov 20 16:33:52 UTC 2017 - containers-bugowner@suse.de - Commit 3af7f41 by Maximilian Meister mmeister@suse.de only set service entries for localhost on kube-master also explain in a comment why we need to set the apiserver for 127.0.0.1 on all hosts (bsc#1067219) Signed-off-by: Maximilian Meister ------------------------------------------------------------------- Fri Nov 10 13:43:32 UTC 2017 - containers-bugowner@suse.de - Commit f74c756 by Rafael Fernández López ereslibre@ereslibre.es Disable container-feeder before rebooting. This will allow us to control when container-feeder starts to load new images from the filesystem. Due to some possible docker configuration changes it might be restarted while container-feeder is working (if we keep it enabled). Force to disable the service before rebooting. Fixes: bsc#1066653 ------------------------------------------------------------------- Fri Nov 10 13:42:38 UTC 2017 - containers-bugowner@suse.de - Commit ebd1907 by Rafael Fernández López ereslibre@ereslibre.es Generate sa key in the update orchestration This is the safest path, but a refactor should come to make this part of the ca highstate so the update and the kubernetes orchestrations just force the ca highstate on both cases. Related: bsc#1066653 ------------------------------------------------------------------- Thu Nov 9 08:56:18 UTC 2017 - containers-bugowner@suse.de - Commit bc29cc9 by Kiall Mac Innes kiall@macinnes.ie Removed unused flannel iface grain This is a followup to 129e927 ------------------------------------------------------------------- Fri Nov 3 16:44:45 UTC 2017 - containers-bugowner@suse.de - Commit ce396af by Alvaro Saurin alvaro.saurin@gmail.com Replace some other certificates by Jinja templates ------------------------------------------------------------------- Fri Nov 3 15:28:03 UTC 2017 - containers-bugowner@suse.de - Commit 771634b by Alvaro Saurin alvaro.saurin@gmail.com Reorganize the addons in a subdirectory per addon Use some Jinja macros for running kubectl with retries, the kubectl path and the right dependencies ------------------------------------------------------------------- Mon Oct 30 09:43:28 UTC 2017 - containers-bugowner@suse.de - Commit a5fef22 by Flavio Castelli fcastelli@suse.com Retry all iptables states Retry all iptables states to prevent failures like seen with bsc#1064186. Signed-off-by: Flavio Castelli Commit 2646dc4 by Flavio Castelli fcastelli@suse.com Introduce caasp_retriable Provide a generic way to retry any kind of salt state. Signed-off-by: Flavio Castelli ------------------------------------------------------------------- Mon Oct 30 09:42:20 UTC 2017 - containers-bugowner@suse.de - Commit 2974490 by Alvaro Saurin alvaro.saurin@gmail.com Increase worker threads and backlog length (bsc#1065018) ------------------------------------------------------------------- Fri Oct 27 09:39:36 UTC 2017 - containers-bugowner@suse.de - Commit d78fe5d by Alvaro Saurin alvaro.saurin@gmail.com New 'retry[until]' argument for caasp_cmd.run Use a unless/onlyif and retry[until] for skipping some executions and not using some nasty loops ------------------------------------------------------------------- Thu Oct 26 16:41:06 UTC 2017 - containers-bugowner@suse.de - Commit e869357 by Alvaro Saurin alvaro.saurin@gmail.com Wait for etcd before trying to set anything, or just retry of etcd is not responding ------------------------------------------------------------------- Wed Oct 25 11:03:50 UTC 2017 - containers-bugowner@suse.de - Commit e8d8612 by Alvaro Saurin alvaro.saurin@gmail.com Use http.wait_for_successful_query instead of looping with curl ------------------------------------------------------------------- Wed Oct 25 10:46:40 UTC 2017 - containers-bugowner@suse.de - Commit 98c214f by Alvaro Saurin alvaro.saurin@gmail.com Minor: rename k8s_etcd to caasp_etcd (following the implicit code conventions) ------------------------------------------------------------------- Tue Oct 24 10:52:02 UTC 2017 - containers-bugowner@suse.de - Commit 7e88148 by Alvaro Saurin alvaro.saurin@gmail.com Use some Jinja macros for generating certificates ------------------------------------------------------------------- Tue Oct 24 10:37:48 UTC 2017 - containers-bugowner@suse.de - Commit 9dedba0 by Michal Jura mjura@suse.com Fix whitespaces striping in Kubernetes api jinja template ------------------------------------------------------------------- Tue Oct 24 10:35:02 UTC 2017 - containers-bugowner@suse.de - Commit 129e927 by Alvaro Saurin alvaro.saurin@gmail.com Use the default network interface instead of the hardcoded 'eth0' (bsc#1058079) ------------------------------------------------------------------- Tue Oct 24 10:33:46 UTC 2017 - containers-bugowner@suse.de - Commit a2f0485 by Rafael Fernández López ereslibre@ereslibre.es Add `caasp_cmd` state module featuring `run` with retry feature This state module will provide `run` state with `retry` option that accepts `attempts` and `interval` arguments. This allow us to retry a command if it failed, and retry to this maximum number of retries, sleeping between retries. ------------------------------------------------------------------- Fri Oct 20 10:34:17 UTC 2017 - containers-bugowner@suse.de - Commit ef91829 by Michal Jura mjura@suse.com Add comment message about keeping update /etc/hosts in velum container See https://github.com/kubic-project/salt/pull/265#issuecomment-337256898 ------------------------------------------------------------------- Fri Oct 20 10:27:28 UTC 2017 - containers-bugowner@suse.de - Commit 51f2da2 by Kiall Mac Innes kiall@macinnes.ie Correctly handle FQDN `dashboard` values in Velum cert Ensure we correctly handle FQDN values for the `dashboard` pillar when generating the Velum TLS certificate. Fixes bsc#1064284 ------------------------------------------------------------------- Fri Oct 20 08:58:53 UTC 2017 - containers-bugowner@suse.de - Commit 21ec9f3 by Rafael Fernández López ereslibre@ereslibre.es Remove outdated comment and improve it. ------------------------------------------------------------------- Thu Oct 19 15:58:34 UTC 2017 - containers-bugowner@suse.de - Commit 0d3cdfe by Flavio Castelli fcastelli@suse.com Add help message to etc/sysconfig/etcdctl Quick tip about how to source the variables defined inside of the file to quickly have etcdctl work. Signed-off-by: Flavio Castelli ------------------------------------------------------------------- Wed Oct 18 12:03:03 UTC 2017 - containers-bugowner@suse.de - Commit 863cc73 by Kiall Mac Innes kiall@macinnes.ie Manage the Velum TLS cert This ensures that the dashboard_external_fqdn is registered within the velum TLS certificate. bsc#1063998 ------------------------------------------------------------------- Tue Oct 17 18:15:24 UTC 2017 - containers-bugowner@suse.de - Commit 061c968 by Michal Jura mjura@suse.com Keep updated /etc/hosts on velum-dashboard container, bsc#1062728 We would like to keep /etc/hosts file updated for velum-dashboard with Admin host. Velum needs to know external name of Kube API which will be used to register in Dex service. Problem was discovered and discribed in bug 1062728 ------------------------------------------------------------------- Tue Oct 17 17:50:49 UTC 2017 - containers-bugowner@suse.de - Commit c9d4710 by Kiall Mac Innes kiall@macinnes.ie Docker package was renamed to docker_1_12_6 Update salt to reference the new docker package name, as this was renamed from "docker" to "docker_1_12_6" ------------------------------------------------------------------- Tue Oct 17 13:33:27 UTC 2017 - containers-bugowner@suse.de - Commit 146e288 by Kiall Mac Innes kiall@macinnes.ie Revert K8S to use etcd2 storage format With etcd3, the kubernetes api server will sit in a (slow) restart loop when multimaster is enabled, logging a stacktrace and then restarting. This will manifest as, most commonly, "Unable to connect to the server: unexpected EOF" from kubectl. This will break bootstrap as we need to talk to K8S API to deploy dex, kube-dns, and tiller. bsc#1063235 bsc#1063285 bsc#1063543 ------------------------------------------------------------------- Tue Oct 17 06:59:49 UTC 2017 - containers-bugowner@suse.de - Commit 75145fe by Kiall Mac Innes kiall@macinnes.ie Revert "Revert K8S to use etcd2 storage format" This reverts commit 5e95b0b0fb90d3d8ebd37df0e640303579c9e2c4. This was pushed to master, rather than a branch, by accident. ------------------------------------------------------------------- Wed Oct 11 16:12:37 UTC 2017 - containers-bugowner@suse.de - Commit e3b0d3b by Rafael Fernández López ereslibre@ereslibre.es Fix missing requirement during the upgrade process. Fixes: bsc#1062824 ------------------------------------------------------------------- Wed Oct 11 12:01:31 UTC 2017 - containers-bugowner@suse.de - Commit 1e04919 by Kiall Mac Innes kiall@macinnes.ie Allow Dex to redirect to the Dashboard's external FQDN Some scenarios where the admin node's private IP is not accessible to the outside world require that we use a end user provided FQDN - e.g. as is the case on OpenStack and possibly other cloud environments. Allow redirections to this FQDN. Part of bsc#1062291 ------------------------------------------------------------------- Tue Oct 10 07:11:36 UTC 2017 - containers-bugowner@suse.de - Commit 75e85a0 by Nikhil Manchanda SlickNik@gmail.com Update tiller deployment to use sles-based docker image Currently the tiller image being used for the tiller deployment is from the upstream registry at gcr.io. We should be using the SLES based docker image instead of the upstream one. Fixes: bsc#1062380 ------------------------------------------------------------------- Sat Oct 7 08:47:44 UTC 2017 - containers-bugowner@suse.de - Commit 1df2665 by Kiall Mac Innes kiall@macinnes.ie Update VERSION file to 2.0.0+dev ------------------------------------------------------------------- Fri Oct 6 15:18:46 UTC 2017 - containers-bugowner@suse.de - Commit 497891d by Michal Jura mjura@suse.com Add floating network to cloud-provider integration with OpenStack We would like add new pillar value floating, which will be used to configure floating network for cloud provider intergration with OpenStack. If this option is specified, it will create floating ip for loadbalancer automatically. ------------------------------------------------------------------- Fri Oct 6 14:42:17 UTC 2017 - containers-bugowner@suse.de - Commit ba9c3f8 by Rafael Fernández López ereslibre@ereslibre.es Set frontend settings: `dir` and `theme`. ------------------------------------------------------------------- Fri Oct 6 14:11:56 UTC 2017 - containers-bugowner@suse.de - Commit 1ecef44 by Kiall Mac Innes kiall@macinnes.ie Dex: Wait for Dex to be fully up and running We shouldn't allow a bootstrap to complete without Dex being up and running, so lets wait for the Dex API to start responding. ------------------------------------------------------------------- Fri Oct 6 11:46:08 UTC 2017 - containers-bugowner@suse.de - Commit c4b42e6 by Michal Jura mjura@suse.com Remove duplicated storage-backend option for Kubernetes API, bsc#1061810 Option storage-backend is provided two times for Kubernetes API configuration. We have to keep only one option with value provided from pillar. ------------------------------------------------------------------- Fri Oct 6 09:47:20 UTC 2017 - containers-bugowner@suse.de - Commit 3e654d9 by Robert Roland robert.roland@suse.com Add a URL off Velum as a valid OIDC redirect URI This will make it so that Dex will be happy to redirect you to velum ------------------------------------------------------------------- Thu Sep 21 13:56:59 UTC 2017 - containers-bugowner@suse.de - Commit 50f84f4 by Rafael Fernández López ereslibre@ereslibre.es Add `caasp_service.running_stable` This new state will allow us to make sure that a service is running in a stable manner. Also, will do some waits in case systemd will do retries on the background, what avoids instant failure from salt being reported with a regular `service.running`. Fixes: bsc#1059105 ------------------------------------------------------------------- Thu Sep 21 13:13:22 UTC 2017 - containers-bugowner@suse.de - Commit 408ab7a by Kiall Mac Innes kiall@macinnes.ie Allow custom options to be passed to the Salt Master Rename the salt master configurations, so that custom options can be loaded after the stock options, allowing an override. bsc#1059724 ------------------------------------------------------------------- Thu Sep 21 10:14:09 UTC 2017 - containers-bugowner@suse.de - Commit 60e6a69 by Alvaro Saurin alvaro.saurin@gmail.com Do not access infra machines through the proxy (bsc#1053739) ------------------------------------------------------------------- Thu Sep 21 09:57:03 UTC 2017 - containers-bugowner@suse.de - Commit f730743 by Kiall Mac Innes kiall@macinnes.ie Ensure cluster-service labels are consistent These were inconsistent, with some services using the labels, and others not. Within services, some of the resoures the label should be applied to were not, even though other parts of the same service did have the label applied. Commit 6520870 by Kiall Mac Innes kiall@macinnes.ie Add CriticalAddonsOnly tolerations Add CriticalAddonsOnly toleration to dex/kube-dns/timmer, this syncs them with upstream, and allows for masters to be flagged as suitable for running these critical contains if desired. Commit 6cde454 by Kiall Mac Innes kiall@macinnes.ie Remove Kube addonmanager references As Kubernetes addonmanager is not used to deploy these, we should not apply the addonmanager labels. Should a end user deploy kube addonmanager, it will believe these pods are under it's control and potentially remove or change them. bsc#1059516 ------------------------------------------------------------------- Thu Sep 21 09:12:20 UTC 2017 - containers-bugowner@suse.de - Commit 7184f5e by Kiall Mac Innes kiall@macinnes.ie Prevent update-etc-hosts conflicting with bootstrap Fix another case where the etc hosts update orchestration would otherwise conflict with the bootstrap / add node orchestration. bsc#1059577 ------------------------------------------------------------------- Wed Sep 20 09:51:41 UTC 2017 - containers-bugowner@suse.de - Commit 8865d73 by Robert Roland rob.roland@gmail.com Making the service account key the same on all nodes (#230) The kube-apiserver and kube-controller-manager must agree on what the private key is for service account generation. In a multi-master scenario, where an api server starts on one machine, and the controller-manager on another machine becomes primary, pods cannot be created because kube-controller-manager cannot communicate with the apiserver. So, now, we generate the service account key on the ca minion and store it in the mine, so that it's generated once. Fixes bsc#1059398 ------------------------------------------------------------------- Tue Sep 19 22:27:46 UTC 2017 - containers-bugowner@suse.de - Commit 6868ea5 by Alvaro Saurin alvaro.saurin@gmail.com Set a default external fqdn ------------------------------------------------------------------- Tue Sep 19 22:26:44 UTC 2017 - containers-bugowner@suse.de - Commit 2df25a0 by Aishwarya Thangappa aishwarya.thangappa@gmail.com Fix the race condition that occurs when starting Kube-DNS KubeDNS may fail to apply due to a race condition within `kubectl apply`, this mitigates that issue. ------------------------------------------------------------------- Fri Sep 15 10:36:14 UTC 2017 - containers-bugowner@suse.de - Commit 5d0e520 by Kiall Mac Innes kiall@macinnes.ie Update paths to match SLES based Dex container The SLES based dex container does not put dex in /usr/local/bin, additionally, we install the web content in /usr/share/caasp-dex/web. Part of bsc#1058833 ------------------------------------------------------------------- Wed Sep 13 12:59:55 UTC 2017 - containers-bugowner@suse.de - Commit e966106 by Michal Jura mjura@suse.com Add OpenStack block storage version as a option ------------------------------------------------------------------- Wed Sep 13 12:58:52 UTC 2017 - containers-bugowner@suse.de - Commit 8e90c5c by Kiall Mac Innes kiall@macinnes.ie Include kube-apiserver in the dex role Without this, We're seeing an error post-bootstrap, so deployments look green, but fail with: The following requisites were not found: require: id: kube-apiserver ------------------------------------------------------------------- Wed Sep 13 10:03:30 UTC 2017 - containers-bugowner@suse.de - Commit cc32e39 by Robert Roland robert.roland@suse.com Switch to the sles12/caasp-dex image ------------------------------------------------------------------- Wed Sep 13 08:54:40 UTC 2017 - containers-bugowner@suse.de - Commit 6c2b47a by Michal Jura mjura@suse.com Add orchestration for etcd storage 'etcd2' to 'etcd3' In Kubernetes v1.7 default storage backend for apiserver is 'etcd3'. We need orchestrate migration between version 'etcd2' and 'etcd3'. ------------------------------------------------------------------- Wed Sep 13 08:52:38 UTC 2017 - containers-bugowner@suse.de - Commit c26d987 by Robert Roland rob.roland@gmail.com Role-based access control (#192) Adding role-based access control based on CoreOS Dex and OpenLDAP ------------------------------------------------------------------- Tue Sep 12 14:27:59 UTC 2017 - containers-bugowner@suse.de - Commit 2b5dd9b by Nikhil Manchanda SlickNik@gmail.com Add cluster role binding for tiller Tiller requires a cluster role binding to work correctly with the new RBAC changes. Add this cluster role binding so that helm commands work correctly. ------------------------------------------------------------------- Tue Sep 12 09:03:03 UTC 2017 - containers-bugowner@suse.de - Commit efd8877 by Rafael Fernández López ereslibre@ereslibre.es Set etcd3 as default backend storage ------------------------------------------------------------------- Sat Sep 9 09:01:51 UTC 2017 - containers-bugowner@suse.de - Commit 3e9bcd6 by Kiall Mac Innes kiall@macinnes.ie Move External FQDN to 127.0.0.1 address s was added to ensure Dex was always reachable, however, with multi masters, this name was assigned to 3 different lines in /etc/hosts. Most consumers of /etc/hosts do not deal with this as they would a round-robin DNS entry which returns multiple IPs. When the "selected" master is powered off, this name continues to resolve the same dead IP address. As Dex uses a NodePort service, putting this to 127.0.0.1 works as we expect it to. ------------------------------------------------------------------- Fri Sep 8 12:46:25 UTC 2017 - containers-bugowner@suse.de - Commit 5e89d99 by Alvaro Saurin alvaro.saurin@gmail.com Refactor the wait-for-apiserver so it can be used in some other parts of the code ------------------------------------------------------------------- Fri Sep 8 12:45:44 UTC 2017 - containers-bugowner@suse.de - Commit 5a13bbc by Kiall Mac Innes kiall@macinnes.ie Ensure systemd is reloaded after units are changed Ensure systemd is reloaded as soon as a unit is changed, rather than relying on a task later within the orchestration to execute. Fixes bsc#1057641 ------------------------------------------------------------------- Fri Sep 8 11:37:54 UTC 2017 - containers-bugowner@suse.de - Commit a601b38 by Kiall Mac Innes kiall@macinnes.ie Include short hostname for masters The short hostname for masters was not being set, as it was for both the admin node, and worker nodes Fixes bsc#1057794 ------------------------------------------------------------------- Fri Sep 8 11:09:21 UTC 2017 - containers-bugowner@suse.de - Commit 755ad7c by Sam Leavens rbwsam@gmail.com Adding optional addon for Helm's tiller ------------------------------------------------------------------- Fri Sep 8 10:23:47 UTC 2017 - containers-bugowner@suse.de - Commit e0727d2 by Kiall Mac Innes kiall@macinnes.ie Combine etcd and etcd-proxy formulas The base etcd formula is never used on it's own, lets remove this unnecessary complexity. ------------------------------------------------------------------- Thu Sep 7 13:23:50 UTC 2017 - containers-bugowner@suse.de - Commit c0bbaba by Kiall Mac Innes kiall@macinnes.ie Include both v2 and v3 flags in etcdctl vars ------------------------------------------------------------------- Tue Sep 5 17:13:10 UTC 2017 - containers-bugowner@suse.de - Commit c1c851c by Robert Roland rob.roland@gmail.com Role-based access control (#192) Adding role-based access control based on CoreOS Dex and OpenLDAP ------------------------------------------------------------------- Wed Aug 30 09:29:40 UTC 2017 - containers-bugowner@suse.de - Commit 66b0de2 by Aishwarya Thangappa aishwarya.thangappa@gmail.com Update docker images for KubeDNS to ones based on SLES from the rpms in MicroOS ------------------------------------------------------------------- Tue Aug 29 15:55:29 UTC 2017 - containers-bugowner@suse.de - Commit 67846f6 by Kiall Mac Innes kiall@macinnes.ie Fix flannel config for 0.8.0 Flannel in 0.8.0 rejects the "-logtostderr" flag we were providing, this doesn't seem to have ever been an option, however it was silently ignored in the past. ------------------------------------------------------------------- Tue Aug 29 14:48:45 UTC 2017 - containers-bugowner@suse.de - Commit 5c4bf44 by Michal Jura mjura@suse.com Set kube-apiserver storage backend as option Parametrize Kubernetes apiserver storage backend. This will be used in future for migration process from storage etcd2 to etcd3. ------------------------------------------------------------------- Fri Aug 25 17:50:59 UTC 2017 - containers-bugowner@suse.de - Commit 0a8f3e2 by Michal Jura mjura@suse.com Add cloud provider integration for OpenStack Storage Commit 885cc4d by Michal Jura mjura@suse.com Add cloud provider integration for OpenStack LoadBalancer ------------------------------------------------------------------- Tue Aug 22 10:42:22 UTC 2017 - containers-bugowner@suse.de - Commit 6ac7ffb by Kiall Mac Innes kiall@macinnes.ie Use haproxy to load balance Kube API requests Now that we can have multiple masters, we need a way for the various services and end-users to be load balanced over the set of kube-api servers. We install haproxy on each node, inside a docker container, configured to load balance requests over all the cluster masters. This haproxy is configured to listen on 0.0.0.0 on the masters, and 127.0.0.1 on the workers. This is to allow the minions to simply "talk" to 127.0.0.0, and be routed to an active kube-api server. ------------------------------------------------------------------- Mon Aug 21 14:22:13 UTC 2017 - containers-bugowner@suse.de - Commit 2269176 by Kiall Mac Innes kiall@macinnes.ie Use apply instead of create for addons kubectl apply is generally idempotent, while kubectl create is not. With multi-master now enabled, if two masters execute this script at once, one of them is likely to fail given the check+set race within this script - Switching to apply removes part of this this C+S race. The second part of this race, is it client-side decision by apply to create or update, by retrying the command once if it fails, we can ensure when two masters run this script at the same time, for the first time, the C+S race will be avoided here too. ------------------------------------------------------------------- Mon Aug 21 08:43:16 UTC 2017 - containers-bugowner@suse.de - Commit b470a20 by Kiall Mac Innes kiall@macinnes.ie Ensure k8s_etcd.get_cluster_size works for multi-master If we had enough masters to form a etcd cluster, we would end up returning "None" from this method, preventing the cluster formation. ------------------------------------------------------------------- Mon Aug 21 08:34:11 UTC 2017 - containers-bugowner@suse.de - Commit 06033b3 by Alvaro Saurin alvaro.saurin@gmail.com Wait for the API server after starting the service. ------------------------------------------------------------------- Mon Aug 21 08:01:52 UTC 2017 - containers-bugowner@suse.de - Commit af41306 by Alvaro Saurin alvaro.saurin@gmail.com Do not generate an empty --proxy line in curlrc ------------------------------------------------------------------- Fri Aug 18 14:56:14 UTC 2017 - containers-bugowner@suse.de - Commit bdd9b9c by Kiall Mac Innes kiall@macinnes.ie Grow flannel CIDR to accommodate 1024 workers Flannel was setup such that 150 workers could obtain a subnet before there were not none left. By growing this range, and the size of the individual allocations, we allow for up to 1024 workers with 510 pods on each. bsc#1047847 ------------------------------------------------------------------- Thu Aug 17 16:43:08 UTC 2017 - containers-bugowner@suse.de - Commit 4b40d4c by Aishwarya Thangappa aishwarya.thangappa@gmail.com Add kube-dns service account ------------------------------------------------------------------- Thu Aug 17 14:38:54 UTC 2017 - containers-bugowner@suse.de - Commit e1d5650 by Kiall Mac Innes kiall@macinnes.ie Disable Salt's Job Cache Salt's job cache is buggy, causing random failures to lookup mine data, which in turn causes our deployments to fail. Fixes bsc#1054256 ------------------------------------------------------------------- Thu Aug 17 13:53:02 UTC 2017 - containers-bugowner@suse.de - Commit 7c47d63 by Alvaro Saurin alvaro.saurin@gmail.com Properly wait for a HTTP endpoint ------------------------------------------------------------------- Wed Aug 16 18:14:24 UTC 2017 - containers-bugowner@suse.de - Commit a4a049e by Kiall Mac Innes kiall@macinnes.ie Kube-API: Set storage-backend to etcd2 In our current configuration, kube-api logs a series of errors unless this is set. ------------------------------------------------------------------- Wed Aug 9 12:03:51 UTC 2017 - containers-bugowner@suse.de - Commit 6caa9fa by Robert Roland robert.roland@suse.com Dedicated certificate for kube-controller-manager Commit 5e5dfb5 by Robert Roland robert.roland@suse.com Dedicated certificate for kube-proxy Commit afe4f63 by Robert Roland robert.roland@suse.com Dedicated certificate for kubelet Commit 8acea7c by Robert Roland robert.roland@suse.com Dedicated certificate for kube-scheduler Commit e59670e by Robert Roland robert.roland@suse.com Adapting kube-apiserver wait fix into this branch Commit c4eef4d by Robert Roland robert.roland@suse.com eliminated the kubernetes-master formula the daemons are all separate now, so it's controlled by role membership in the top.sls file moved addons to a separate salt formula Commit 9232705 by Robert Roland robert.roland@suse.com kube-proxy as a separate salt formula Commit 15ff190 by Robert Roland robert.roland@suse.com kubelet as a separate salt formula Commit 4412b9d by Robert Roland robert.roland@suse.com kube-scheduler as its own formula fixing a bug where we uncordon master nodes. but we should never do that. Commit 4662dd1 by Robert Roland robert.roland@suse.com kube-controller-manager as a separate formula Commit ee9fb0b by Robert Roland robert.roland@suse.com kube-apiserver as a separate formula Makes a dedicated formula for the kube-apiserver Generates a cert specifically for the kube-apiserver ------------------------------------------------------------------- Mon Aug 7 20:34:39 UTC 2017 - containers-bugowner@suse.de - Commit 65b9e9c by Robert Roland robert.roland@suse.com can't talk to 6443 without a client cert talk to the insecure-bind-address instead. Commit 5c6d2e1 by Kiall Mac Innes kiall@macinnes.ie Wait for Kube-API before installing Kube-DNS ------------------------------------------------------------------- Thu Aug 3 16:51:38 UTC 2017 - containers-bugowner@suse.de - Commit 3a6869d by Aishwarya Thangappa aishwarya.thangappa@gmail.com Install Kube-DNS by default 1. Removed the skydns template files and added kubedns template files. We will be using deployments instead of replication controllers. 2. Modified the deploy script to check for the existence of kube-dns deployment, kube-dns service and config map before creating one. 3. Turned on the addon:dns flag so as to install KubeDNS by default. ------------------------------------------------------------------- Wed Aug 2 22:34:57 UTC 2017 - containers-bugowner@suse.de - Commit d1abfaa by Thomas Hipp thipp@suse.de update k8s version Signed-off-by: Thomas Hipp ------------------------------------------------------------------- Tue Aug 1 14:27:32 UTC 2017 - containers-bugowner@suse.de - Commit bc3adf7 by Robert Roland robert.roland@suse.com Explicit dependency ordering Commit 1086ebf by Robert Roland robert.roland@suse.com Run kubelet and kube-proxy on the master node A standard Kubernetes installation runs a kubelet and kube-proxy on every node, and then you decide where to run apiserver, controller-manager and scheduler. This change is required to support RBAC, DaemonSets and many other changes. Requires an updated kubernetes-client package that contains: https://build.opensuse.org/request/show/494998 ------------------------------------------------------------------- Thu Jul 20 15:15:54 UTC 2017 - containers-bugowner@suse.de - Commit 5df94da by Kiall Mac Innes kiall@macinnes.ie Delay reboots during upgrade by 15 seconds Even with backgrounding the call, salt-minion sometimes still does not have enough time to respond before systemd shuts down salt-minion on some environments. By adding a 15 second delay, we give salt-minion much more time than it should need in a healthy cluster to respond. Additionally, switch from the deprecated syntax for supplying bg=True, to the newer syntax which no longer logs a warning. Followup up fix for bsc#1049200 ------------------------------------------------------------------- Thu Jul 20 12:34:09 UTC 2017 - containers-bugowner@suse.de - Commit 4920c7a by Rafael Fernández López ereslibre@ereslibre.es Do not publish the `ca.crt` from the `ca` SLS, use `mine_functions` We will be publishing this contents when the `ca` minion starts, so there's no need to do this during the orchestration. `mine.send` is not reliable enough since we cannot confirm that the contents are there yet, and waiting a random amount of time is not appropriate as we are just hiding the real problem. In the near future we can do an active wait for the content to be there using `retry`, but for now we just publish the contents of the `ca.crt` using `mine_functions`, so it is sent when the `ca` minion starts. There's no need to refresh the mine, as this was just hiding the real problem when we were publishing this contents during the orchestration phase. Fixes: bsc#1049137 Fixes: bsc#1048548 ------------------------------------------------------------------- Wed Jul 19 14:55:03 UTC 2017 - containers-bugowner@suse.de - Commit 3e5cf9f by Kiall Mac Innes kiall@macinnes.ie Add extra requisites to the update orchestration These additional requisites enforce a stricter ordering of tasks during the upgrade. In some case, "-set-update-grain" would not execute in the right place, potentially leading to a failed upgrade. bsc#1045381 ------------------------------------------------------------------- Wed Jul 19 11:40:34 UTC 2017 - containers-bugowner@suse.de - Commit d97a24e by Kiall Mac Innes kiall@macinnes.ie Don't wait for minion responses when rebooting When we instruct a minion to reboot, we can't reliably wait for the response from salt-minion letting us know that the "systemctl reboot" command succeeded, as systemd may choose to shutdown the salt-minion service before it can sent out the "Yes, that worked" response. Salt does not make any attempt to finish in progress tasks when it receives a SIGTERM, leaving us with few other viable choices for this. Fixes bsc#1049200 ------------------------------------------------------------------- Tue Jul 18 10:11:10 UTC 2017 - containers-bugowner@suse.de - Commit 0692dbf by Rafael Fernández López ereslibre@ereslibre.es Explicitly refresh the mine on all minions after the `ca` has published the `ca.crt` We will explicitly force all minions to refresh the mine after the `ca` minion has published the `ca.crt` certificate on the mine, to avoid rendering problems with later SLS being executed. It might happen that a minion was missing this information on its mine, so the rendering of the SLS failed, effectively stopping the whole orchestration process. Fixes: bsc#1048548 ------------------------------------------------------------------- Mon Jul 17 12:55:09 UTC 2017 - containers-bugowner@suse.de - Commit 219b7d5 by Kiall Mac Innes kiall@macinnes.ie Upgrade: Wait longer for minions to reboot Wait 1200 seconds (20 minutes) for minions to reboot, instead of the default 300 seconds (5 minutes). We increase this to cover off cases where slower to boot physical hardware is used. 20 minutes was chosen as, I've seen physical hardware take 10-12 minutes in the past, and someone likely has something that is slower to reboot. bsc#1048683 ------------------------------------------------------------------- Fri Jul 14 15:59:05 UTC 2017 - containers-bugowner@suse.de - Commit 1e41512 by Alvaro Saurin alvaro.saurin@gmail.com Add some extra naames to the AIP server certificate (bsc#1033671) ------------------------------------------------------------------- Fri Jul 14 14:46:02 UTC 2017 - containers-bugowner@suse.de - Commit 6b146d5 by Maximilian Meister mmeister@suse.de make branch safe by transforming slashes to dashes Signed-off-by: Maximilian Meister Commit 588b834 by Maximilian Meister mmeister@suse.de packaging: make branch configurable Signed-off-by: Maximilian Meister ------------------------------------------------------------------- Fri Jul 14 13:45:02 UTC 2017 - containers-bugowner@suse.de - Commit 6b146d5 by Maximilian Meister mmeister@suse.de make branch safe by transforming slashes to dashes Signed-off-by: Maximilian Meister Commit 588b834 by Maximilian Meister mmeister@suse.de packaging: make branch configurable Signed-off-by: Maximilian Meister ------------------------------------------------------------------- Fri Jul 14 08:26:11 UTC 2017 - containers-bugowner@suse.de - Commit c59070d by Rafael Fernández López ereslibre@ereslibre.es Fix `ca` key path This was a leftover from the previous implementation. Now the ca key is present under `/etc/pki/private` in the ca container too (as it mounts `/etc/pki`) ------------------------------------------------------------------- Thu Jul 13 19:29:28 UTC 2017 - containers-bugowner@suse.de - Commit b6281ae by Kiall Mac Innes kiall@macinnes.ie Ensure grains are always refreshed periodically Salt's grains_refresh_every configuration param does not quite do what we need it to, it's failing to refresh grains from the `grains` file - leading to updates going undetected. This change adds a slightly modified version of what this config param internally does, adding the force_refresh: True argument, ensuring we correctly refresh. bsc#1048583 ------------------------------------------------------------------- Tue Jul 11 14:57:50 UTC 2017 - containers-bugowner@suse.de - Commit 88e9ff9 by Rafael Fernández López ereslibre@ereslibre.es Keep `job_cache: True` as it's discouraged to disable it Our deployment is also failing probably due to the fact that we were disabling the salt `job_cache`. Commit b0547af by Miquel Sabaté Solà msabate@suse.com Set MySQL as the job cache for the Salt master First of all, we can specify an external job cache. If we don't do that, then the `keep_jobs` option only applies to the local cache. This means that Salt will not clean up jobs, events and returns older than the specified `keep_jobs` value (default: 24h) for the MySQL returner that we have already configured. Moreover, since we'd already be using MySQL as a job cache, we don't have to use the local system (/var/cache/salt/master/jobs/) as a cache (note that Salt would still be using this directory to avoid JID collisions). The documentation also says that the local cache can be a burden for large deployments. See bsc#1044133 Signed-off-by: Miquel Sabaté Solà ------------------------------------------------------------------- Tue Jul 11 14:06:57 UTC 2017 - containers-bugowner@suse.de - Commit 31ad98d by Michal Jura mjura@suse.com Don't duplicate log level argument for k8s services, bsc#1046407 ------------------------------------------------------------------- Tue Jul 11 12:54:53 UTC 2017 - containers-bugowner@suse.de - Commit fcbfd6b by Michal Jura mjura@suse.com Make log level configurable for dockerd service, bsc#1046407 Set the logging level for dockerd, possible values are: [ debug, info, warn, error, fatal ] ------------------------------------------------------------------- Tue Jul 11 10:18:31 UTC 2017 - containers-bugowner@suse.de - Commit e3c9c21 by Kiall Mac Innes kiall@macinnes.ie Add Jenkinsfile The Jenkinsfile in each repo, if we adopt Jenkins in the end, will be very thin, including just a single library load, and a single method call. This prevents us from needing to keep each projects Jenkinsfile in sync as CI changes are made. ------------------------------------------------------------------- Mon Jul 10 20:59:33 UTC 2017 - containers-bugowner@suse.de - Commit 08a0960 by Kiall Mac Innes kiall@macinnes.ie Revert "Set MySQL as the job cache for the Salt master" This reverts commit de22c660a99bc1425295c86be7d7dc3e79089845. ------------------------------------------------------------------- Mon Jul 10 12:57:44 UTC 2017 - containers-bugowner@suse.de - Commit de22c66 by Miquel Sabaté Solà msabate@suse.com Set MySQL as the job cache for the Salt master First of all, we can specify an external job cache. If we don't do that, then the `keep_jobs` option only applies to the local cache. This means that Salt will not clean up jobs, events and returns older than the specified `keep_jobs` value (default: 24h) for the MySQL returner that we have already configured. Moreover, since we'd already be using MySQL as a job cache, we don't have to use the local system (/var/cache/salt/master/jobs/) as a cache (note that Salt would still be using this directory to avoid JID collisions). The documentation also says that the local cache can be a burden for large deployments. See bsc#1044133 Signed-off-by: Miquel Sabaté Solà ------------------------------------------------------------------- Fri Jul 7 09:34:03 UTC 2017 - containers-bugowner@suse.de - Commit d2df0ed by Rafael Fernández López ereslibre@ereslibre.es When generating the certificate use the pillar path Since we added the minion certificate location to the pillar, also take the public key location from the pillar, or the certificate generation will fail if the pillar value changes. ------------------------------------------------------------------- Fri Jul 7 09:31:58 UTC 2017 - containers-bugowner@suse.de - Commit ce45c56 by Rafael Fernández López ereslibre@ereslibre.es Remove unneeded signing policies These signing policies were used when the CA wasn't containerized, when we containerized it, they were moved to `caasp-container-manifests`, and the CA container is mounting it from there. If we uncontainerize the CA in the future we can move it back if needed, but let's keep this clean so it's not misleading. ------------------------------------------------------------------- Fri Jul 7 08:11:40 UTC 2017 - containers-bugowner@suse.de - Commit 871a9dc by Michal Jura mjura@suse.com Fix JINJA escaping for docker_opts in docker state module ------------------------------------------------------------------- Thu Jul 6 12:59:46 UTC 2017 - containers-bugowner@suse.de - Commit 2bd42f5 by Rafael Fernández López ereslibre@ereslibre.es Add prerequisite for key to be present on `cert` sls Add a specific dependency for the key to be present when generating the certificate for the minion. ------------------------------------------------------------------- Thu Jul 6 12:57:41 UTC 2017 - containers-bugowner@suse.de - Commit eb852df by Rafael Fernández López ereslibre@ereslibre.es Add kubectl client certificate This certificate will be served by Velum when downloading the `kubeconfig` file, and is specific for that usage. Fixes: bsc#1046963 ------------------------------------------------------------------- Fri Jun 30 10:24:19 UTC 2017 - containers-bugowner@suse.de - Commit 9950702 by Kiall Mac Innes kiall@macinnes.ie Ensure bootstrap_complete grain is set At the time this if block is called, the mine / grains sync hasn't happened yet. This reverts a change from commit fc8347c (bsc#1043589) ------------------------------------------------------------------- Fri Jun 30 10:16:13 UTC 2017 - containers-bugowner@suse.de - Commit 5e7c46f by Michal Jura mjura@suse.com Define etcdctl config file with SSL variables Let's add /etc/sysconfig/etcdctl with paths to the client server TLS files and endpoint. This will make possible to run etcdctl command in easy way, e.g. source /etc/sysconfig/etcdctl etcdctl cluster-health fixes bsc#1046818 ------------------------------------------------------------------- Fri Jun 30 09:34:50 UTC 2017 - containers-bugowner@suse.de - Commit 15748cd by Flavio Castelli fcastelli@suse.com Handle curl proxy settings YaST is also configuring proxy settings inside of `/root/.curlrc`, this is needed because zypper is using libcurl. So if you run zypper from a cronjob or `su`, the `/etc/sysconfig/proxy` variables are not parsed and set in the environment. Which means, zypper will not use the proxy and fail. With `/root/.curlrc`, libcurl will use the proxies configured there. Signed-off-by: Flavio Castelli ------------------------------------------------------------------- Thu Jun 29 17:11:15 UTC 2017 - containers-bugowner@suse.de - Commit fc8347c by Rafael Fernández López ereslibre@ereslibre.es Enable TLS on the salt-api service Fixes: bsc#1043589 ------------------------------------------------------------------- Thu Jun 29 16:26:45 UTC 2017 - containers-bugowner@suse.de - Commit 465a4d6 by Kiall Mac Innes kiall@macinnes.ie Add proxy state to admin node Installs proxies onto the admin node - bsc#1043538 Commit a16c19e by Kiall Mac Innes kiall@macinnes.ie Disable rebootmgr on admin node Once the system bootstraps, we now disable rebootmgr on the admin node. This allows the velum initiated updates to takeover and prevent any unexpected surprises. bsc#1046602 Commit ef8ba5b by Kiall Mac Innes kiall@macinnes.ie Render /etc/hosts on admin node Render the /etc/hosts file on the admin node, so nodes are reacable via their internal FQDNs everywhere. Additionally, include the admin node in the /etc/hosts files. bsc#1045186 ------------------------------------------------------------------- Thu Jun 29 13:04:10 UTC 2017 - containers-bugowner@suse.de - Commit eadd8e1 by Kiall Mac Innes kiall@macinnes.ie Increase salt-master timeout When dealing with a large number of minions, timeouts are visible when using the default value of 5 seconds. Increasing the CPU/RAM resources allocated to the master helps, but given it it's short bursts of heavy usage (bootstrap and upgrade), this shouldn't be necessary. We increase the timeout from 5 to 20 seconds, allowing tasks to take longer yet still succeed. ------------------------------------------------------------------- Wed Jun 28 15:36:29 UTC 2017 - containers-bugowner@suse.de - Commit 3f2c44b by Graham Hayes graham.hayes@suse.com bsc#1045381 Ensure updates do not conflict with etc-hosts This ensure that the etc-hosts orchestration does not run during an upgrade, as this can cause conflicts on the nodes, which cause salt to fail to complete an `orch.update` run. ------------------------------------------------------------------- Tue Jun 27 10:45:02 UTC 2017 - containers-bugowner@suse.de - Commit 5f492f9 by Graham Hayes graham.hayes@suse.com Turn off `auto_accept` ------------------------------------------------------------------- Mon Jun 26 18:56:16 UTC 2017 - containers-bugowner@suse.de - Commit 197d164 by Michal Jura mjura@suse.com Enable etcd authentication based on client certificates Enable ETCD_CLIENT_CERT_AUTH and ETCD_PEER_CLIENT_CERT_AUTH in etcd-proxy state module. - Enable client cert authentication ETCD_CLIENT_CERT_AUTH="true" - Enable peer client cert authentication. ETCD_PEER_CLIENT_CERT_AUTH="true" Commit 970a590 by Michal Jura mjura@suse.com Use Kubernetes API server etcd ssl Commit 776bf33 by Michal Jura mjura@suse.com Enable https for flanneld service Commit b762959 by Michal Jura mjura@suse.com Add ssl pillar profile Commit 07a5652 by Michal Jura mjura@suse.com Enable https for etcd-proxy services All these fixes bsc#1043595 ------------------------------------------------------------------- Fri Jun 23 13:42:40 UTC 2017 - containers-bugowner@suse.de - Commit a567814 by Kiall Mac Innes kiall@macinnes.ie Ensure CA fields are static (bsc#1045766) As the DHCP domain name can change, we should avoid using it in our CA cert in order to prevent it being unnecessarily regenerated. Fixes bsc#1045766 ------------------------------------------------------------------- Thu Jun 22 16:40:48 UTC 2017 - containers-bugowner@suse.de - Commit 9e20d89 by Alvaro Saurin alvaro.saurin@gmail.com Option for using the proxy settings system-wide (bsc#1036627) ------------------------------------------------------------------- Wed Jun 21 14:47:21 UTC 2017 - containers-bugowner@suse.de - Commit 5042479 by Rafael Fernández López ereslibre@ereslibre.es Do not run etcd discovery on every orchestration run, only the first time When adding new nodes, the `orch.kubernetes` orchestration was failing because etcd is refusing to start since the etcd discovery mechanism was already used when bootstrapping the cluster. With this change we ensure that we use the discovery mechanism only when we are boostrapping the cluster. ------------------------------------------------------------------- Tue Jun 20 16:21:12 UTC 2017 - containers-bugowner@suse.de - Commit e51791e by Kiall Mac Innes kiall@macinnes.ie Set etcd batch size to 3 nodes Currently, we never ask for more than 3 members. Setting this to 3 ensures we don't let more than 3 members attempt etcd discovery before a cluster has been fully formed. If we have less this 3, this will still succeed, as the exact number of members we expect will also end up attempting discovery at the same time. ------------------------------------------------------------------- Tue Jun 20 13:35:24 UTC 2017 - containers-bugowner@suse.de - Commit a13010e by Rafael Fernández López ereslibre@ereslibre.es Do not fail if `salt.function` has no minions to target Currently, `update-etc-hosts` orchestration fails because `update_mine` `salt.function` cannot target any minions at the beginning, and since this is a prerequisite for other states, the Reactor orchestration fails. Only call to these `salt.function` if there are any minions to target. ------------------------------------------------------------------- Fri Jun 16 11:50:44 UTC 2017 - containers-bugowner@suse.de - Commit d2f8840 by Rafael Fernández López ereslibre@ereslibre.es Add missing `tgt_type` so we target the minions we intend to This last step on the orchestration was returning a `False` result because no targets were found to execute the grain set. ------------------------------------------------------------------- Fri Jun 16 08:52:34 UTC 2017 - containers-bugowner@suse.de - Commit 9ddaa5a by Flavio Castelli fcastelli@suse.com salt-api: listen to localhost [bsc#1043589] Do not expose the salt-api to the entire world. This is needed only by Velum to trigger salt actions. Given both the containers use the same network namespace we can just bind this service to localhost. By doing that we are going to reduce the attack surface. This fixes one of the two issues reported by bsc#1043589 Signed-off-by: Flavio Castelli ------------------------------------------------------------------- Thu Jun 15 13:56:56 UTC 2017 - containers-bugowner@suse.de - Commit a99d516 by Aishwarya Thangappa aishwarya.thangappa@gmail.com Making the cluster-dns and cluster-domain arguments default Right now, caasp doesn't support kube-dns out of the box. If customers wanted to have dns support, they have to bring it up on their own by using `kubectl create -f kubedns.yaml`. But this will not work until you add the cluster-dns and cluster-domain arguments to kubelet args and restart the kubelet. While doing this manually in every node is one pain point, salt will try to bring it back to its original state. Meaning that the changes you made to the kubelet args will no longer be there. So, unless you bring up the caasp cluster with the addon set to true, you cannot have kube-dns working reliably on the cluster. This change will make it a little easier, by having these arguements by default in every node. ------------------------------------------------------------------- Wed Jun 14 18:46:11 UTC 2017 - containers-bugowner@suse.de - Commit 706837b by Graham Hayes graham.hayes@suse.com Ensure that reactor states only run on completed nodes This ensures that we do not run reactor orchestrations on nodes that have not completed bootstrapping. This ensures that a node cannot have 2 states applied to it at the same time. ------------------------------------------------------------------- Wed Jun 14 17:10:03 UTC 2017 - containers-bugowner@suse.de - Commit e44cf82 by Kiall Mac Innes kiall@macinnes.ie Remove concurrent=True from orchestrations Salt's documentation calls this option out as dangerous, staging that the state must be able to be ran concurrently. This is not something we can reasonably ensure works, so lets not use it. From Salt's documentation: This flag is potentially dangerous. It is designed for use when multiple state runs can safely be run at the same time. Do not use this flag for performance optimization. ------------------------------------------------------------------- Wed Jun 14 17:09:04 UTC 2017 - containers-bugowner@suse.de - Commit 3fd0d08 by Kiall Mac Innes kiall@macinnes.ie Refresh grains at the start of orchestration Additionally, refresh pillars at the start of update-etc-hosts.sls for consistency. ------------------------------------------------------------------- Wed Jun 14 10:41:27 UTC 2017 - containers-bugowner@suse.de - Commit 7d0a037 by Graham Hayes graham.hayes@suse.com Update transactional-update to use "salt" option This will ensure that the transactional-update code will write a grain (`tx_update_reboot_needed:true`) on the node instead of rebooting the node. This also allows for increasing the frequency of the snapshots being built ------------------------------------------------------------------- Tue Jun 13 15:42:26 UTC 2017 - containers-bugowner@suse.de - Commit 91d649f by Alvaro Saurin alvaro.saurin@gmail.com React to IP changes by using beacons ------------------------------------------------------------------- Mon Jun 12 14:00:19 UTC 2017 - containers-bugowner@suse.de - Commit 53e389f by Rafael Fernández López ereslibre@ereslibre.es Only run `service.dead` on salt minions that we know support it. The `ca` container was reporting this error during the orchestration: ``` service.dead { "__run_num__": 0, "_stamp": "2017-06-12T10:33:29.009340", "changes": {}, "comment": "State 'service.dead' was not found in SLS 'rebootmgr' Reason: 'service' __virtual__ returned False: No service execution module loaded: check support for service management on SLES-12 ", "name": "rebootmgr", "result": false, "retcode": 2 } ``` Also, the overall result of the orchestration was not successfully (despite individual highstates reported success) because of this. Containers don't have `systemctl` available, so `salt` doesn't know how to handle this. Right now, rely on our roles for doing this (despite we could have used `virtual` grain -- but for some reason a container reports `physical`, which doesn't help) -- at least with the `salt` version we are currently using. The orchestration result overall looks like this with this change: ``` "outputter": "highstate", "retcode": 0 }, "success": true, "user": "saltapi" } ``` ------------------------------------------------------------------- Mon Jun 12 10:43:45 UTC 2017 - containers-bugowner@suse.de - Commit 0cd2559 by Graham Hayes graham.hayes@suse.com Batch runs of the `cert` state This allows more nodes to be deployed without causing timeouts and failed runs on the `cert` state. Also, remove concurrecny from the etcd member and proxy to ensure members are created before proxies bsc#1038814 ------------------------------------------------------------------- Fri Jun 9 16:50:30 UTC 2017 - containers-bugowner@suse.de - Commit 9b3652a by Kiall Mac Innes kiall@macinnes.ie Revert "Add module for removing etcd cluster members" - bsc#1043676 This reverts commit 27a4e81c331dc345e56266a57c5dcd86d1c1a177 Commit befe0b5 by Kiall Mac Innes kiall@macinnes.ie Revert "Add etcd_info salt grain module" - bsc#1043676 This reverts commit da17af3f0f9cb89a9057618b7561074a4e35818e. ------------------------------------------------------------------- Wed Jun 7 14:15:15 UTC 2017 - containers-bugowner@suse.de - Commit 4132fa9 by Rafael Fernández López ereslibre@ereslibre.es Remove hardcoded secrets ------------------------------------------------------------------- Wed Jun 7 08:31:10 UTC 2017 - containers-bugowner@suse.de - Commit 27a4e81 by Michal Jura mjura@suse.com Add module for removing etcd cluster members ------------------------------------------------------------------- Tue Jun 6 21:17:34 UTC 2017 - containers-bugowner@suse.de - Commit 40d8e9b by Robert Roland robert.roland@suse.com Fixing broken build Need to remove a reference to /var/lib/etcd if salt isn't managing it anymore ------------------------------------------------------------------- Tue Jun 6 15:38:43 UTC 2017 - containers-bugowner@suse.de - Commit 1100cfe by Graham Hayes graham.hayes@suse.com Stop managing /var/lib/etcd in salt This dir is created by the etcd rpm, and permissions are maintained by etcd when it is running The salt and etcd disagree an what these permissions are causing extra "changed" entries. As etcd is changing them to what it needs, and the directory is created by etcd (and its RPM) we should not try and manage it. ------------------------------------------------------------------- Tue Jun 6 11:40:55 UTC 2017 - containers-bugowner@suse.de - Commit 26fa83b by Jordi Massaguer Pla jmassaguerpla@suse.de use git revision in package version this way zypper sees each new commit as an update Otherwise, using the date, will create a conflict if 2 commits are from the same day Signed-off-by: Jordi Massaguer Pla ------------------------------------------------------------------- Fri Jun 2 19:43:18 UTC 2017 - containers-bugowner@suse.de - Commit e706873 by Michal Jura mjura@users.noreply.github.com Enable https for all services and create dedicated ssl pillar profile (#86) * Enable https for etcd-proxy services * Enable https for flanneld service * Add ssl pillar profile * Use Kubernetes API server etcd ssl ------------------------------------------------------------------- Fri Jun 2 18:47:43 UTC 2017 - containers-bugowner@suse.de - Commit da17af3 by Michal Jura mjura@suse.com Add etcd_info salt grain module To maintaine etcd cluster configuration by salt, it is needed to get etcd status about members and their roles in etcd cluster. This etcd_info grain module provides followind information: - 'etcd_module' - return "available" if python-etcd module is installed - 'members_all' - return list of all members in etcd cluster - 'member_type' - return role of local etcd service, possible values "proxy", "member", "leader" - 'member_id' - return unique id of local etcd service in the cluster This grain module will be used by salt_delete state module for removing etcd nodes from the cluster. To run this module is required to install following packages: - python-etcd - python-urllib3 - python-dnspython ------------------------------------------------------------------- Fri Jun 2 15:34:22 UTC 2017 - containers-bugowner@suse.de - Commit 7031d71 by Victor Palade vpalade@suse.com disable reboot manager when orchestration happens ------------------------------------------------------------------- Fri Jun 2 09:27:08 UTC 2017 - containers-bugowner@suse.de - Commit 9815b3b by Rafael Fernández López ereslibre@ereslibre.es Ensure our states are idempotent - Adapt some `cmd.run` to use `onchanges`, so they only execute when their `watched` states change. - Add `stateful: True` to some `cmd.run`s, so following the salt protocol for this we ensure that the command didn't change anything in the system state. - Move `ca-cert` to its own SLS, so `cert` will only now generate the `/etc/pki/minion.{key,crt}` files. - The `cert` SLS will now be the only responsible for generating certificates depending on the role of the machine. This way we ensure that without mattering how this SLS is included it behaves in the same way under all conditions. We might want to use a certificate for different services, but that will need some extra changes. - Change some `module.run` to `module.wait` so they only execute when the `watched` states change. - Remove cleanups that make it impossible to have idempotent states. ------------------------------------------------------------------- Fri Jun 2 07:40:33 UTC 2017 - containers-bugowner@suse.de - Commit c0667e3 by Kiall Mac Innes kiall@macinnes.ie Don't change the system hostname Operators don't want us to change the system hostname, which we previously did to account for environments which don't provide unique DHCP hostnames. We'll undo this change, as we have now removed our reliance on the system default hostname. Fixes bsc#1041789 ------------------------------------------------------------------- Thu Jun 1 11:23:18 UTC 2017 - containers-bugowner@suse.de - Commit 86ae430 by Alvaro Saurin alvaro.saurin@gmail.com Update the /etc/hosts by using a loop, so the file doesn not grow indefinetively. Do not set the IP address for API server in the API servers to 127.0.0.1 Commit acb76f3 by Alvaro Saurin alvaro.saurin@gmail.com Add the kubelet port configurable with a Pillar variable Open the kubelet port in the firewall ------------------------------------------------------------------- Thu Jun 1 11:14:15 UTC 2017 - containers-bugowner@suse.de - Commit 8bc25b2 by Kiall Mac Innes kiall@macinnes.ie Add a caasp_fqdn grain and migrate to it This adds a caasp_fqdn grain and migrates usage of fqdn to it. This is needed because the fqdn grain has proved unrelable, where we know *exactly* what we want, and salt's detection will be broken by a upcoming change. Partial fix for bsc#1041789 ------------------------------------------------------------------- Thu Jun 1 09:29:01 UTC 2017 - containers-bugowner@suse.de - Commit 7f7d9aa by Graham Hayes graham.hayes@suse.com Initial framework of update orchestration ------------------------------------------------------------------- Thu Jun 1 09:28:05 UTC 2017 - containers-bugowner@suse.de - Commit 631ea1d by Kiall Mac Innes kiall@macinnes.ie Allow for clean shutdown of nodes Add a stop SLS for each service we wish to shutdown clearly, doing any necessary pre-stop actions such as draining kubelet. ------------------------------------------------------------------- Tue May 30 15:51:55 UTC 2017 - containers-bugowner@suse.de - Commit d8ce355 by Rafael Fernández López ereslibre@ereslibre.es Do not include etcd-proxy on this last action This triggers a chain reaction when the reboot sls is called directly (salt-call state.apply reboot) on the last step of the orchestration, since etcd-proxy includes etcd, and etcd includes cert. Cert sls will generate a new certificate overriding the current one with all the correct DNS names and IP addresses, by one that only contains `fqdn` as the only dns name. Fixes: bsc#1040858 ------------------------------------------------------------------- Mon May 29 15:25:59 UTC 2017 - containers-bugowner@suse.de - Commit daadead by Rafael Fernández López ereslibre@ereslibre.es - Make cert always include `fqdn` - - The only component that was adding `fqdn` to the list of dns names of SAN - certificates is the `kube-master` role. - - However, depending on the size of the cluster and other possible reasons it - might happen that a etcd member falls in a `kube-minion` instance, where the - certificate is missing local ip addresses, as well as the `fqdn` of the - machine. With this change, we are enforcing `cert` to always generate this - information automatically, while we still allow to extend it, in case that's - still necessary (for example, as kubernetes-master still requires). - - Check https://bugzilla.novell.com/show_bug.cgi?id=1039269#c9 for further - information. - - Fixes: bsc#1039269 ------------------------------------------------------------------- Fri May 26 14:52:45 UTC 2017 - containers-bugowner@suse.de - Commit ce5954e by Alvaro Saurin alvaro.saurin@gmail.com - Minor changes in etcd: do not remoove /var/lib/etcd and close some ports we - don't really need ------------------------------------------------------------------- Thu May 25 11:12:23 UTC 2017 - containers-bugowner@suse.de - Commit 7317ca8 by Miquel Sabaté Solà msabate@suse.com - docker: reload container-feeder after starting docker - - See bsc#1040579 - - Signed-off-by: Miquel Sabaté Solà ------------------------------------------------------------------- Tue May 23 06:57:51 UTC 2017 - containers-bugowner@suse.de - Commit 6013d74 by Robert Roland rob.roland@gmail.com - Update etcd.conf - - Stray + character was causing this line to not execute, and I ended up with a - cluster with both folders present, preventing etcd from starting. ------------------------------------------------------------------- Mon May 22 16:34:25 UTC 2017 - containers-bugowner@suse.de - Commit 824101b by Alvaro Saurin alvaro.saurin@gmail.com - Fix some problems with Docker when HTTP proxy vars are empty ------------------------------------------------------------------- Thu May 18 20:18:33 UTC 2017 - containers-bugowner@suse.de - Commit 4f664e1 by PI-Victor palade.ionut@gmail.com - revert changes to etcd systemd drop-in unit ------------------------------------------------------------------- Thu May 18 15:45:05 UTC 2017 - containers-bugowner@suse.de - Commit bace710 by Rafael Fernández López ereslibre@ereslibre.es - Add apiserver main hostname - - Fixes: bsc#1039437 ------------------------------------------------------------------- Thu May 18 14:58:30 UTC 2017 - containers-bugowner@suse.de - Commit 88c1434 by Michal Jura mjura@suse.com - Configure ETCD_INITIAL_ADVERTISE_PEER_URLS only with FQDN - - We have to remove IP based ETCD_INITIAL_ADVERTISE_PEER_URLS, because they use - HTTPS, which is failing for IP URLS with following error - - health check for peer 100fbbb05571e58f could not connect: x509: - cannot validate certificate for 10.17.3.176 because it doesn't contain any - IP SANs ------------------------------------------------------------------- Thu May 18 10:49:25 UTC 2017 - containers-bugowner@suse.de - Commit fcc6f23 by Alvaro Saurin alvaro.saurin@gmail.com - Handle proxies in the docker daemon ------------------------------------------------------------------- Tue May 16 11:54:01 UTC 2017 - containers-bugowner@suse.de - Use colons as nesting instead of dots ------------------------------------------------------------------- Tue May 16 10:16:21 UTC 2017 - containers-bugowner@suse.de - Do a deeper cleanup before restarting etcd Some etcd deps Take flannel setup out of the master Perform flannel setup before k8s master setup ------------------------------------------------------------------- Thu May 11 16:21:50 UTC 2017 - containers-bugowner@suse.de - bump number of worker threads * to avoid minion calls to master timing out * fixes https://github.com/kubic-project/salt/issues/62 ------------------------------------------------------------------- Mon May 8 12:01:03 UTC 2017 - containers-bugowner@suse.de - Initial config files for the reactor, with an example sls for presence ------------------------------------------------------------------- Tue May 2 16:27:17 UTC 2017 - containers-bugowner@suse.de - Renamed docker registry variable ------------------------------------------------------------------- Tue May 2 13:54:42 UTC 2017 - containers-bugowner@suse.de - Update etcd member count logic ------------------------------------------------------------------- Tue May 2 11:17:43 UTC 2017 - containers-bugowner@suse.de - Cleanup the docker options ------------------------------------------------------------------- Thu Apr 27 16:06:38 UTC 2017 - containers-bugowner@suse.de - Set Hostname to match machine-id ------------------------------------------------------------------- Thu Apr 27 15:30:49 UTC 2017 - containers-bugowner@suse.de - Fix Jinja2 syntax error in kubelet.jinja ------------------------------------------------------------------- Thu Apr 27 15:22:23 UTC 2017 - containers-bugowner@suse.de - Fix Jinja2 syntax error in kubeconfig.jinja ------------------------------------------------------------------- Thu Apr 27 14:26:13 UTC 2017 - containers-bugowner@suse.de - Use some constant names for the API server ------------------------------------------------------------------- Thu Apr 27 14:12:12 UTC 2017 - containers-bugowner@suse.de - Use machine ID and domain as kubelet hostname ------------------------------------------------------------------- Thu Apr 27 14:09:10 UTC 2017 - containers-bugowner@suse.de - Update default etcd cluster size to match number of masters ------------------------------------------------------------------- Thu Apr 27 08:50:26 UTC 2017 - containers-bugowner@suse.de - Configure kube-{scheduler/controller-manager} leader elections ------------------------------------------------------------------- Tue Apr 25 12:20:40 UTC 2017 - containers-bugowner@suse.de - [WIP] Use machine ID as kubelet hostname ------------------------------------------------------------------- Mon Apr 24 16:00:34 UTC 2017 - containers-bugowner@suse.de - Replace the SVGs by PNGs ------------------------------------------------------------------- Mon Apr 24 15:55:29 UTC 2017 - containers-bugowner@suse.de - Some docs ------------------------------------------------------------------- Wed Apr 19 15:17:16 UTC 2017 - containers-bugowner@suse.de - Cleanup ------------------------------------------------------------------- Wed Apr 19 10:55:32 UTC 2017 - containers-bugowner@suse.de - Do not assume minion_id is hostname/fqdn ------------------------------------------------------------------- Tue Apr 18 09:43:21 UTC 2017 - containers-bugowner@suse.de - Allow the kubelet to run on Kubernetes 1.6 ------------------------------------------------------------------- Mon Apr 10 08:47:59 UTC 2017 - containers-bugowner@suse.de - Bug 1032379 - Must install flanneld on the kubernetes master node ------------------------------------------------------------------- Wed Mar 29 08:24:13 UTC 2017 - containers-bugowner@suse.de - Actually use `grains.get` default value ------------------------------------------------------------------- Tue Mar 28 18:17:25 UTC 2017 - containers-bugowner@suse.de - Always set `CN`. Even if no grains are set (because the domain could not be inferred), set the default dns domain from the pillar. ------------------------------------------------------------------- Tue Mar 28 16:13:12 UTC 2017 - containers-bugowner@suse.de - Fix etcd deps ------------------------------------------------------------------- Tue Mar 28 13:41:42 UTC 2017 - containers-bugowner@suse.de - Make etcd state a requirement for states that need etcd running on localhost ------------------------------------------------------------------- Mon Mar 27 15:53:38 UTC 2017 - containers-bugowner@suse.de - Do not indent (it's not a mine_function) ------------------------------------------------------------------- Mon Mar 27 14:10:38 UTC 2017 - containers-bugowner@suse.de - Fixed the infra container path for CaaSP ------------------------------------------------------------------- Mon Mar 27 13:00:42 UTC 2017 - containers-bugowner@suse.de - Do not set certificate `CN` if domain was not specified by a grain ------------------------------------------------------------------- Thu Mar 23 09:36:30 UTC 2017 - containers-bugowner@suse.de - Added parameters for passing extra arguments ------------------------------------------------------------------- Tue Mar 21 13:39:37 UTC 2017 - containers-bugowner@suse.de - Renamed API server vars ------------------------------------------------------------------- Mon Mar 20 15:48:33 UTC 2017 - containers-bugowner@suse.de - fix infra container image (=pause image) for opensuse ------------------------------------------------------------------- Mon Mar 20 12:32:44 UTC 2017 - containers-bugowner@suse.de - pod_infra_container_image is not optional anymore ------------------------------------------------------------------- Mon Mar 20 12:06:18 UTC 2017 - containers-bugowner@suse.de - Revert 6bae304 and fe1677c ------------------------------------------------------------------- Mon Mar 20 12:01:16 UTC 2017 - containers-bugowner@suse.de - fix etcd proxy instance failure on restart ------------------------------------------------------------------- Mon Mar 20 09:46:20 UTC 2017 - containers-bugowner@suse.de - Renamed API server vars ------------------------------------------------------------------- Fri Mar 17 10:15:57 UTC 2017 - containers-bugowner@suse.de - packaging: fix name of tarball directory ------------------------------------------------------------------- Fri Mar 17 09:56:00 UTC 2017 - containers-bugowner@suse.de - packaging: fix name of tarball directory ------------------------------------------------------------------- Fri Mar 17 09:02:45 UTC 2017 - containers-bugowner@suse.de - packaging: fix name of tarball directory ------------------------------------------------------------------- Thu Mar 9 12:33:22 UTC 2017 - jmassaguerpla@suse.com - Disable service as it needs to be this way in the final repo ------------------------------------------------------------------- Fri Mar 3 15:49:42 UTC 2017 - alvaro.saurin@suse.com - Updated for CaaSP ------------------------------------------------------------------- Thu Feb 23 11:47:37 UTC 2017 - alvaro.saurin@suse.com - Updated for k8s 1.5.3 ------------------------------------------------------------------- Thu Feb 23 10:09:27 UTC 2017 - alvaro.saurin@suse.com - Initial version