Booting OpenIndiana on EC2
Since OpenSolaris was axed, we haven't had an option for running a Solaris-based distribution on EC2. Thanks to the hard work of Andrzej Szeszo, commissioned by Nexenta, now we do.
This should be considered a proof of concept, and is perhaps not ready for production. Use at your own risk!
Spin up ami-4a0df023 as m1.large or bigger. This is an install of OpenIndiana oi_147.
🔗Authentication
The image doesn't currently import your EC2 keypairs, so you'll need to log in as a user. root is a role in this image, so you'll need to log in as the oi user.
The oi user's password is "oi".
# ssh oi@1.2.3.4 The authenticity of host '1.2.3.4 (1.2.3.4)' can't be established. RSA key fingerprint is da:b9:0e:73:20:81:4f:a2:a7:91:0d:7d:3c:4b:cb:80. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '1.2.3.4' (RSA) to the list of known hosts. Password: Last login: Sun Apr 17 06:54:24 2011 from domu-3-4-5-6 OpenIndiana SunOS 5.11 oi_147 September 2010 oi@openindiana:~$ pfexec su - Password: OpenIndiana SunOS 5.11 oi_147 September 2010 root@openindiana:~#
To make root a real user again, use the following commands:
$ pfexec rolemod -K type=normal root $ pfexec perl -pi -e 's/PermitRootLogin no/PermitRootLogin yes/' /etc/ssh/sshd_config $ pfexec svcadm restart ssh
You can now log in as root as you'd expect. This behavior is changing all over the place (including Solaris 11 proper), but I don't mind being a dinosaur.
🔗Caveats
There are some limitations, however.
🔗Boot Environments
- You have no console access
- If an upgrade fails, you can't choose a working BE from grub
- For the same reason, you won't be able to boot failsafe
🔗Zones
- You won't be able to pull an IP from EC2 for your zones
- Only one Elastic IP can be assigned to an instance, so you won't be able to forward a public IP to an internal zone IP
- You'll be able to forward ports from the global zone to zones, of course, but this is less useful than zones having unique IPs associated with them
🔗EBS
- There is a bug in devfsadmd which doesn't like device IDs over 23. I describe how to deal with this below.
🔗Boot Volumes
There are two EBS volumes assigned with this AMI. The 8GB one is the root pool device. The 1GB is where the boot loader (pv-grub) lives.
Triskelios joked earlier that your instance got into a hosed state, you could mount the 1GB volume elsewhere, munge your grub config, then assign it back to your the busted instance. This theoretically gets around not having console access. It's also hilarious. But could work.
🔗Upgrading to oi_148
oi_147 has some known bugs, so we want to get up to oi_148. You could also update to the OpenIndiana illumos build (dev-il), but we'll stick with 148 for now.
The old opensolaris.org publisher is still available, as there is software on it not available in OpenIndiana's repo. However, we need to set that publisher non-sticky so it doesn't hold back package upgrades. If you don't set the repo non-sticky, you won't get a complete upgrade. You'll be running a 148 kernel, but lots of 147 packages. One symptom of this is zones won't install.
root@openindiana:~# pkg publisher PUBLISHER TYPE STATUS URI openindiana.org (preferred) origin online http://pkg.openindiana.org/dev/ opensolaris.org origin online http://pkg.openindiana.org/legacy/ root@openindiana:~# pkg set-publisher --non-sticky opensolaris.org root@openindiana:~# pkg publisher PUBLISHER TYPE STATUS URI openindiana.org (preferred) origin online http://pkg.openindiana.org/dev/ opensolaris.org (non-sticky) origin online http://pkg.openindiana.org/legacy/
Once that's done, we update the current image.
root@openindiana:~# pkg image-update Packages to remove: 4 Packages to install: 7 Packages to update: 531 Create boot environment: Yes DOWNLOAD PKGS FILES XFER (MB) Completed 542/542 11254/11254 225.5/225.5 PHASE ACTIONS Removal Phase 1882/1882 Install Phase 2382/2382 Update Phase 19953/19953 PHASE ITEMS Package State Update Phase 1073/1073 Package Cache Update Phase 535/535 Image State Update Phase 2/2 A clone of openindiana exists and has been updated and activated. On the next boot the Boot Environment openindiana-1 will be mounted on '/'. Reboot when ready to switch to this updated BE. --------------------------------------------------------------------------- NOTE: Please review release notes posted at: http://docs.sun.com/doc/821-1479 ---------------------------------------------------------------------------
A new BE has been created for us, and is slated to be active on reboot.
root@openindiana:~# beadm list BE Active Mountpoint Space Policy Created -- ------ ---------- ----- ------ ------- openindiana N / 94.0K static 2011-04-04 23:00 openindiana-1 R - 3.90G static 2011-04-17 07:20 root@openindiana:~# reboot
Once the instance comes back up, log in:
# ssh oi@1.2.3.4 Password: Last login: Sun Apr 17 06:54:25 2011 from domu-3.4.5.6 OpenIndiana SunOS 5.11 oi_148 November 2010 oi@openindiana:~$ pfexec su - Password: OpenIndiana SunOS 5.11 oi_148 November 2010
As you can see, we're running oi_148.
To be sure the upgrade is happy and we don't have any sticky 147 bits left:
root@openindiana:~# pkg list | grep 147 root@openindiana:~#
Awesome.
🔗ZFS on EBS
Create some EBS volumes and attach them to the instance. You'll need to specify a Linux-style device path for the volume. The older OpenSolaris AMI required a numeric device, as Solaris expects device IDs to be 0..23. This either seems to have been broken at some point in the last year, or doesn't work with pv-grub. Regardless, we can work around it.
$ ./aws create-volume --size 128 --zone us-east-1b $ ./aws create-volume --size 128 --zone us-east-1b $ ./aws create-volume --size 128 --zone us-east-1b $ ./aws create-volume --size 128 --zone us-east-1b $ ./aws attach-volume vol-vvvvvvvv -i i-iiiiiiii -d /dev/sdc $ ./aws attach-volume vol-vvvvvvvv -i i-iiiiiiii -d /dev/sdd $ ./aws attach-volume vol-vvvvvvvv -i i-iiiiiiii -d /dev/sde $ ./aws attach-volume vol-vvvvvvvv -i i-iiiiiiii -d /dev/sdf
Once the volumes are available, you'll see messages like this in dmesg:
failed to lookup dev name for /xpvd/xdf@2128 disk_link: invalid disk device number (2128)
Which is the devfsadmd bug I mentioned above. Solaris expects device IDs to be 0..23, and devfsadm doesn't know how to deal with anything higher.
There's very likely a way to automate this, but I just wrote a stupid script that creates links in /dev/dsk and /dev/rdsk for the devices we've attached to the instance. Until the devices have the proper links, you won't see them in format or iostat. And cfgadm doesn't work in a Xen guest, so.
The device IDs are consistent, however. The first two disks in the system (the rpool and the pv-grub volumes) are 2048 and 2064. The device IDs increment by 16:
root@openindiana:~# format /xpvd/xdf@2048 1. c0t1d0 ?????HH???X?[??????? cyl 4095 alt 0 hd 128 sec 32> /xpvd/xdf@2064 Specify disk (enter its number):
So now we link in the new devices:
root@openindiana:~# ./links.sh c0t2d0 2080 root@openindiana:~# ./links.sh c0t3d0 2096 root@openindiana:~# ./links.sh c0t4d0 2112 root@openindiana:~# ./links.sh c0t5d0 2128 root@openindiana:~# format /xpvd/xdf@2048 1. c0t1d0 ?????HH???X?[??????? cyl 4095 alt 0 hd 128 sec 32> /xpvd/xdf@2064 2. c0t2d0 ?????HH???X?[??????? cyl 16709 alt 0 hd 255 sec 63> /xpvd/xdf@2080 3. c0t3d0 ?????HH???X?[??????? cyl 16709 alt 0 hd 255 sec 63> /xpvd/xdf@2096 4. c0t4d0 ?????HH???X?[??????? cyl 16709 alt 0 hd 255 sec 63> /xpvd/xdf@2112 5. c0t5d0 ?????HH???X?[??????? cyl 16709 alt 0 hd 255 sec 63> /xpvd/xdf@2128 Specify disk (enter its number):
Create our ZFS pool:
root@openindiana:~# zpool create tank mirror c0t2d0 c0t3d0 mirror c0t4d0 c0t5d0 root@openindiana:~# zpool list tank NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT tank 254G 80K 254G 0% 1.00x ONLINE -
Once the EFI labels have been written to the disks, format stops throwing errors on them, as well:
root@openindiana:~# format /xpvd/xdf@2048 1. c0t1d0 ?????HH???X?[??????? cyl 4095 alt 0 hd 128 sec 32> /xpvd/xdf@2064 2. c0t2d0 <unknown-unknown-0001-128.00gb> /xpvd/xdf@2080 3. c0t3d0 <unknown-unknown-0001-128.00gb> /xpvd/xdf@2096 4. c0t4d0 <unknown-unknown-0001-128.00gb> /xpvd/xdf@2112 5. c0t5d0 <unknown-unknown-0001-128.00gb> /xpvd/xdf@2128 Specify disk (enter its number):
And, just for fun:
root@openindiana:~# dd if=/dev/urandom of=/tank/random bs=1024 count=204800 204800+0 records in 204800+0 records out 209715200 bytes (210 MB) copied, 9.14382 s, 22.9 MB/s
So now you have ZFS on EBS, with the ability to do anything ZFS can. Snapshots will be much, much faster than EBS snapshots (though not complete copies, and will obviously be lost if your pool is lost, whereas EBS snapshots are complete copies of the volume and can be cloned and mounted out of band), enable compression, dedup (though this would probably be terrifyingly slow on EC2), and so on.
links.sh is available here.
🔗Virtual Networking
This is fodder for another post, but something I've done elsewhere is to use Crossbow to create a virtual network with zones and VirtualBox VMs. The global zone runs OpenVPN, giving clients access to these private resources. This model seems perfectly suited to EC2, given the IP assignment limitations noted above. Unfortunately I don't' imagine VirtualBox is an option here, but even just a private network of zones would be extremely useful.
And perhaps someday EC2 will let you assign multiple Elastic IPs to an instance.
🔗Conclusion
While there are still a few bugs to be worked out, this proof of concept AMI does work and is ready to have its tires kicked.
I'm pretty stoked to have Solaris available on EC2. Thanks Andrzej and Nexenta both!