From: jbeulich@suse.com
Subject: x86/Xen: disable IBRS around CPU stopper function invocation
References: none so far
Patch-mainline: Never, SUSE-Xen specific

Xen's original migration logic can't deal with arbitrary MSRs; only
"Migration v2" is capable of that. A check has been put into the tool
stack however to make sure a guest won't silently lose any MSR value.
Therefore, for migration on Xen 4.5 and earlier to work, we need to
clear SPEC_CTRL on all CPUs on the suspend paths and restore the
intended setting on the resume paths.

The "fast suspend" mode of PV guest migration (which is the default) is
solely based on stop_machine(), which doesn't allow the remote CPUs to
do anything once the system has been quiesced. Place the IBRS clearing/
restoring right in cpu_stopper_thread() for the lack of a better
alternative. I think it is acceptable to do this independent of what
stop_machine() (or one of its siblings) was actually invoked for;
otherwise some flag would need to be introduced.

The old, "slow" suspend mode can be dealt with in Xen code. All
secondary CPUs are brought offline there first, so only CPU0 needs to
actually fiddle with SPEC_CTRL in that case.

--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -20,6 +20,9 @@
 #include <linux/kallsyms.h>
 
 #include <asm/atomic.h>
+#if defined(CONFIG_X86) && defined(CONFIG_XEN)
+#include <asm/spec_ctrl.h>
+#endif
 
 /*
  * Structure to determine completion condition and record errors.  May
@@ -277,10 +280,18 @@ repeat:
 		/* cpu stop callbacks are not allowed to sleep */
 		preempt_disable();
 
+#if defined(CONFIG_X86) && defined(CONFIG_XEN)
+		x86_disable_ibrs();
+#endif
+
 		ret = fn(arg);
 		if (ret)
 			done->ret = ret;
 
+#if defined(CONFIG_X86) && defined(CONFIG_XEN)
+		x86_enable_ibrs();
+#endif
+
 		/* restore preemption and check it's still balanced */
 		preempt_enable();
 		WARN_ONCE(preempt_count(),
--- a/drivers/xen/core/machine_reboot.c
+++ b/drivers/xen/core/machine_reboot.c
@@ -23,6 +23,7 @@
 
 #if defined(__i386__) || defined(__x86_64__)
 #include <asm/pci_x86.h>
+#include <asm/spec_ctrl.h>
 /* TBD: Dom0 should propagate the determined value to Xen. */
 bool port_cf9_safe = false;
 
@@ -259,7 +260,9 @@ int __xen_suspend(int fast_suspend, void
 		}
 
 		local_irq_disable();
+		x86_disable_ibrs();
 		err = take_machine_down(&suspend);
+		x86_enable_ibrs();
 		local_irq_enable();
 	}
 
