I’ve been trying to work out how to remotely boot an encrypted computer system – in particular, one running Debian GNU/Linux. It turns out that this is not trivial. The reason is, a Debian system with an encrypted system drive expects the user to be able to enter a password to decrypt the drive before it can finish booting up. That’s fine if your system is a laptop and you’ve got it in front of you, but it won’t work if your system is a server located hundreds of miles away, because even if you can remotely switch the server on (e.g. you have a remotely operable power strip, or a friend/contractor who will press the power switch for you), you will have to enter the password into the console before your SSH, HTTP or other server software will be able to run. The reason they can’t run straight away is because they, or at least their configuration files, are on the encrypted disk.
Assuming you really do want to keep your system drive encrypted, and you do need to be able to boot it remotely, there are a couple of solutions.
- Keep your /boot directory in a partition that is unencrypted, and install Dropbear (or another small SSH client) on it, configured to start up on boot in a way that will let you log in via SSH to enter your decryption password. This apparently works (see a tutorial with details for Debian and Ubuntu here), but is complex to configure, fragile (it can break during kernel upgrades) and will not show the remote user any BIOS or POST menus, since the /boot is accessed only after those have run.
- Use KVM-over-IP. This appears to be the standard solution among professional sysadmins. It seems that several server manufacturers (I’ve looked at the HP, Dell, Fujitsu-Siemens and Apple websites) now incorporate KVM-over-IP directly into their servers. Even some of the low-end servers have this built in. Alternatively, external KVM-over-IP boxes can be used if the server doesn’t have one built in. At the cheap end of the market (my end!), I’ve seen models for sale online by brands like Lindy, Belkin, Adder and Aten.
I’ve not tried any of these options, and I don’t know how good they are, but judging by forum posts, other people are interested in this problem too, so I figured I’d share what I’ve learned so far.
One thing that concerns me about both classes of solution is that there’s a remote login prompt sitting online, ripe for brute-forcing. I’d feel much happier deploying one of these systems if it had a rate-limiting option. I suppose that if I deployed it behind a firewall of some kind (e.g. a router-based firewall), I might be able to configure the firewall to limit login attempts; it would depend on the firewall, and on my abilities. Rate-limiting login attempts isn’t something I’ve yet done.
I’ve just learned, here, that recent versions of HP’s iLO do provide rate limiting on login attempts; well done, HP! They also offer two-factor authentication, but unfortunately that’s Windows-only at the moment
Sam -
I don’t really know very much about the specifics of Linux or indeed remote booting an encrypted system, but it seems like an interesting problem.
Thinking about it for a couple of minutes, I had a couple of ideas of things you could do – they may well be inappropriate or utter nonsense but…:
1) If you’ll be consistently accessing the system from the same remote IP address, could you ‘pre-authenticate’ by hard-coding your IP address(es) into the Linux system and only accepting any remote commands (unencrypted) from those IP addresses? The weakness would obviously be the risk of someone spoofing the IP address. Though presumably you could put in some other security to protect against this – maybe password protect the BIOS? This may make no sense, of course –
I’m talking without any knowledge…
2) Use a Linux Virtual Machine in the cloud – I’ve not investigated those sorts of services yet, but I imagine that they are pretty configurable, as well as being scalable etc. Is this for work / Darwin Correspondence Project?
I hope that your thinking gets you to a workable solution…
N
Hi Nick,
1) One problem with this approach is that restricting logins to particular IP address(es) is something that is most easily done by a system that has got as far as booting its firewall. Iptables, for instance, can perform this function easily. If all you’ve got running is dropbear, it might be possible – I don’t know. The dropbear approach is fragile and convoluted enough that I’m disinclined to try it. Some of the KVM-over-IP products might be able to do IP address restriction. In any case, you’d probably want to whitelist more than one IP address, because otherwise if the system at your only whitelisted IP address went down, it would also prevent out-of-band comms with the target system.
2) Depending on the system requirements, and the choice of virtual machine, this may or may not be practical, and may or may not be secure. The aim of system disk encryption is to provide some security even from those who may physically obtain the system, e.g. by theft. Data centres are probably less vulnerable to theft than many other premises, but are not immune. That said, some virtualisation approaches, which distribute storage across spatially distant physical machines, may mitigate this somewhat. In any case, if you want to store large amounts of data (hundreds of GB), but need very little concurrent access, then VMs’ pricing structures may be unfavourable compared to maintaining the physical systems yourself. I investigated the VM route before deciding to go for hardware, though I hadn’t factored in the cost of KVM-over-IP, and am now re-assessing, though I think it likely I may not need out-of-band access, and that even if I decide that I really can’t do without it, the total cost of the physical approach is probably still lower, per year on average (assumming the infrastructure lasts for at least 3 years) than the cost of a comparable VM setup.
And finally… no, this isn’t for the DCP. It’s part of an effort to create a useful infrastructure for various things I’m doing/planning. If I manage to get it all working the way I want it to, I’ll probably blog about the whole arrangement and how to set it up yourself if you wish to.
Thanks for the suggestions,
Sam
Thanks for the clarifications Sam – sounds like you’ve thought it through! Looking forward to hearing more about it soon…
Nick
There’s a nice article about rate-limiting SSH login attempts over at HostingFu. However, it’s IP-based and probably wouldn’t slow down the kind of distributed/spoofed attack which is now in vogue.
It seems that in addition to the dropbear and kvm-over-ip options, there’s a third method: terminal servers. These serve the console over a serial interface to a management server, which can then be logged into to gain console access to any of the managed machines.
There’s a good article here about the benefits of this approach over the KVM-over-IP approach, at least in a professional environment, but it was written in 2002 and I get the impression KVM-over-IP systems have dramatically improved since then.
After (very) painstakingly winnowing down the best options for me to a mere five, it turns out that the greatest benefit/cost ratio is probably just for me to keep the server concerned switched on, and not to worry about out-of-band management.
Here’s the spreadsheet in which I ranked my final shortlist. One skew which needs explaining is that the quad-core ML310 has a free care-pack & software offer at the moment. Another skew is that I have to put pound coins in a meter for the electricity supply the server is running from (honestly!), which means high power consumption is more inconvenient for me than for most.
I’ll be looking into HDD spin-downs, CPU frequency scaling, etc, in a bid to keep the power consumption down, since I’m planning to keep the server up continually. I think even wake-on-lan wouldn’t work with an encrypted system drive and no OOB management, so getting power control settings right is going to prove significant.