The Best of Both Worlds with On-Demand Virtualization Thawan Kooburat and Michael M. Swift On-Demand Virtualization On-Demand Virtualization allows systems to benefit from virtualization without paying for its overhead all the time. • Executing natively: No virtualization overhead when it is not being used. • Virtualize on-the-fly: Convert entire machine to VM only as the need arises. • No service disruption: OS/Application state and network connections remain intact after conversion. Motivation Cost of Virtualization Uses of On-Demand Virtualization There are many places where virtualization is not being used, such as datacenters running performance-sensitive applications or desktop environment due to: There are many use cases which can be enabled by on-demand virtualizations such as: • Lack of functionality: Devices such as GPUs, WIFI and ACPI devices are not emulated or supported by VMM. • IO overhead: Requires costly trap and emulation by host OS or VMM. • Memory overhead: TLB miss is more expensive due to extra level of page tables . • Resource consolidation: Execute natively during peak hours. Then, virtualize and consolidate resources to save power. • Debugging: Virtualize and use tools such as deterministic replay  to analyze issues. • Backup: Checkpointing can be used to create backups. Implementation Key Challenges Problem: Capturing OS and Process State Solution: Hibernation We rely on the hibernation mechanism to transfer OS state from a native machine to a virtual machine: we suspend the native OS to disk, and resume it inside a virtual machine. However, hibernation assume that the OS will resume on machine with same hardware profile. Physical Machine 3 Start VM Native Kernel Virtualize 1 Hibernate VMM Partition Guest Partition VMM Kernel Our prototype is implemented on Linux 2.6.35 using TuxOnIce  for hibernation. We use on KVM as our Virtual Machine Monitor. The only cost is (i) reserving disk space for the VMM and (ii) using logical devices. Our current prototype can perform one-way conversion from native to virtual execution while retaining SSH/SCP connections. The process takes about 90 seconds and majority of the time is spent in hibernate /resume process and machine reboot. Guest Kernel 2 Boot Other Devices 4 Resume VMM Partition Device Guest Partition CPU* Memory Problem: Discovering Devices inside the VM Display Solution: Hotplugging Hibernation Steps Virtualize Steps Prepare device and load drivers Machine Restart Hibernate For platform devices such as timers and interrupt routers that do not have hotplug support, the hibernation boot code passes on configuration information to the resuming kernel, so that it can reconfigure these devices. Load hibernate image Scan and reconfigure devices Resume user processes Solution: Logical Devices OS OS bond0 /dev/mapper/vdisk e1000e Phy e1000 VM CPU Hotplugging Memory Hotplugging Switch to VGA display mode * VMM may not be able to emulate hardware features found in a physical machine such as IOMMU. Thus, we need to disable these features in advance because the kernel and applications may rely on them. References and Related Work Boot up machine Problem: Preserving Device Bindings Logical devices act as an interposition layer between the kernel and device drivers. They are normally used to provide aggregation or high availability. Thus, they allow existing kernel structures to transfer their state from one device to another. We use the network bonding driver to preserve network connections and the device mapper to do the same for block devices. This allows us to preserve device binding even across different model/type of devices. Only one CPU is online Require VM to have the same amount of RAM Future Boot up VMM Kernel Start VM from Guest partition Guest Boot Kernel Kernel We rely on hotplugging support found in many devices to make transition from physical to virtual hardware. The system virtually unplugs physical devices and plugs in new virtual devices. We modify the kernel to rescan the PCI bus to discover virtual devices and attach them to the OS during the resume process. However, this mechanism only works on devices supporting hotplugging. Guest Kernel Legend: Current SATA Phy IDE VM Microvisor  presents the first step toward ondemand virtualization. However, they do not virtualize the entire system so their benefit is limited to online maintenance. VMWare Converter  can clone a physical machine into a VM. However, it does not preserve running state of the physical machine. OS live migration  adds whole-system migration to an OS without requiring virtualization. However, each class of device must provide an import/export interface to transfer device state. Our logical device approach do not require device driver modification.  R. Bhargava, B. Serebrin, F. Spadini, et al. Accelerating two-dimensional page walks for virtualized systems, ASPLOS, 2008.  J. Chow, T. Garfinkel, and P.M. Chen. Decoupling dy-namic program analysis from execution in virtual environments, USENIX, 2008.  Linux software suspend http://www.tuxonice.net  D.E. Lowell, Y. Saito, and E.J. Samberg. Devirtualizable virtual machines enabling general, single-node, online maintenance, ASPLOS, 2004.  VMware vCenter Converter http://www.vmware.com/products/converter  M. Kozuch, M. Kaminsky, and M.P. Ryan. Migration without Virtualization, HotOS, 2009.