Booting OpenIndiana on Amazon EC2

Since OpenSolaris was axed, we haven’t had an option for running a Solaris-based distribution on EC2. Thanks to the hard work of Andrzej Szeszo, commissioned by Nexenta, now we do.

This should be considered a proof of concept, and is perhaps not ready for production. Use at your own risk!

Spin up ami-4a0df023 as m1.large or bigger. This is an install of OpenIndiana oi_147.

Authentication

The image doesn’t currently import your EC2 keypairs, so you’ll need to log in as a user. root is a role in this image, so you’ll need to log in as the oi user.

The oi user’s password is “oi”.

# ssh oi@1.2.3.4
The authenticity of host '1.2.3.4 (1.2.3.4)' can't be established.
RSA key fingerprint is da:b9:0e:73:20:81:4f:a2:a7:91:0d:7d:3c:4b:cb:80.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '1.2.3.4' (RSA) to the list of known hosts.
Password: 
Last login: Sun Apr 17 06:54:24 2011 from domu-3-4-5-6
OpenIndiana     SunOS 5.11      oi_147  September 2010
oi@openindiana:~$ pfexec su -
Password: 
OpenIndiana     SunOS 5.11      oi_147  September 2010
root@openindiana:~# 

To make root a real user again, use the following commands:

$ pfexec rolemod -K type=normal root
$ pfexec perl -pi -e 's/PermitRootLogin no/PermitRootLogin yes/' /etc/ssh/sshd_config
$ pfexec svcadm restart ssh

You can now log in as root as you’d expect. This behavior is changing all over the place (including Solaris 11 proper), but I don’t mind being a dinosaur.

Caveats

There are some limitations, however.

Boot Environments

  • You have no console access
  • If an upgrade fails, you can’t choose a working BE from grub
  • For the same reason, you won’t be able to boot failsafe

Zones

  • You won’t be able to pull an IP from EC2 for your zones
  • Only one Elastic IP can be assigned to an instance, so you won’t be able to forward a public IP to an internal zone IP
  • You’ll be able to forward ports from the global zone to zones, of course, but this is less useful than zones having unique IPs associated with them

EBS

  • There is a bug in devfsadmd which doesn’t like device IDs over 23. I describe how to deal with this below.

Boot Volumes

There are two EBS volumes assigned with this AMI. The 8GB one is the root pool device. The 1GB is where the boot loader (pv-grub) lives.

Triskelios joked earlier that your instance got into a hosed state, you could mount the 1GB volume elsewhere, munge your grub config, then assign it back to your the busted instance. This theoretically gets around not having console access. It’s also hilarious. But could work.

Upgrading to oi_148

oi_147 has some known bugs, so we want to get up to oi_148. You could also update to the OpenIndiana illumos build (dev-il), but we’ll stick with 148 for now.

The old opensolaris.org publisher is still available, as there is software on it not available in OpenIndiana’s repo. However, we need to set that publisher non-sticky so it doesn’t hold back package upgrades. If you don’t set the repo non-sticky, you won’t get a complete upgrade. You’ll be running a 148 kernel, but lots of 147 packages. One symptom of this is zones won’t install.

root@openindiana:~# pkg publisher
PUBLISHER                             TYPE     STATUS   URI
openindiana.org          (preferred)  origin   online   http://pkg.openindiana.org/dev/
opensolaris.org                       origin   online   http://pkg.openindiana.org/legacy/

root@openindiana:~# pkg set-publisher --non-sticky opensolaris.org

root@openindiana:~# pkg publisher
PUBLISHER                             TYPE     STATUS   URI
openindiana.org          (preferred)  origin   online   http://pkg.openindiana.org/dev/
opensolaris.org          (non-sticky) origin   online   http://pkg.openindiana.org/legacy/

Once that’s done, we update the current image.

root@openindiana:~# pkg image-update
                Packages to remove:     4
               Packages to install:     7
                Packages to update:   531
           Create boot environment:   Yes
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                              542/542 11254/11254  225.5/225.5

PHASE                                        ACTIONS
Removal Phase                              1882/1882
Install Phase                              2382/2382
Update Phase                             19953/19953

PHASE                                          ITEMS
Package State Update Phase                 1073/1073
Package Cache Update Phase                   535/535
Image State Update Phase                         2/2

A clone of openindiana exists and has been updated and activated.
On the next boot the Boot Environment openindiana-1 will be mounted on '/'.
Reboot when ready to switch to this updated BE.


---------------------------------------------------------------------------
NOTE: Please review release notes posted at:

http://docs.sun.com/doc/821-1479
---------------------------------------------------------------------------

A new BE has been created for us, and is slated to be active on reboot.

root@openindiana:~# beadm list
BE            Active Mountpoint Space Policy Created      
--            ------ ---------- ----- ------ -------      
openindiana   N      /          94.0K static 2011-04-04 23:00
openindiana-1 R      -          3.90G static 2011-04-17 07:20

root@openindiana:~# reboot

Once the instance comes back up, log in:

# ssh oi@1.2.3.4
Password:
Last login: Sun Apr 17 06:54:25 2011 from domu-3.4.5.6
OpenIndiana     SunOS 5.11      oi_148  November 2010
oi@openindiana:~$ pfexec su -
Password:
OpenIndiana     SunOS 5.11      oi_148  November 2010

As you can see, we’re running oi_148.

To be sure the upgrade is happy and we don’t have any sticky 147 bits left:

root@openindiana:~# pkg list | grep 147
root@openindiana:~#

Awesome.

ZFS on EBS

Create some EBS volumes and attach them to the instance. You’ll need to specify a Linux-style device path for the volume. The older OpenSolaris AMI required a numeric device, as Solaris expects device IDs to be 0..23. This either seems to have been broken at some point in the last year, or doesn’t work with pv-grub. Regardless, we can work around it.

$ ./aws create-volume --size 128 --zone us-east-1b
$ ./aws create-volume --size 128 --zone us-east-1b
$ ./aws create-volume --size 128 --zone us-east-1b
$ ./aws create-volume --size 128 --zone us-east-1b

$ ./aws attach-volume vol-vvvvvvvv -i i-iiiiiiii -d /dev/sdc
$ ./aws attach-volume vol-vvvvvvvv -i i-iiiiiiii -d /dev/sdd
$ ./aws attach-volume vol-vvvvvvvv -i i-iiiiiiii -d /dev/sde
$ ./aws attach-volume vol-vvvvvvvv -i i-iiiiiiii -d /dev/sdf

Once the volumes are available, you’ll see messages like this in dmesg:

failed to lookup dev name for /xpvd/xdf@2128
disk_link: invalid disk device number (2128)

Which is the devfsadmd bug I mentioned above. Solaris expects device IDs to be 0..23, and devfsadm doesn’t know how to deal with anything higher.

There’s very likely a way to automate this, but I just wrote a stupid script that creates links in /dev/dsk and /dev/rdsk for the devices we’ve attached to the instance. Until the devices have the proper links, you won’t see them in format or iostat. And cfgadm doesn’t work in a Xen guest, so.

The device IDs are consistent, however. The first two disks in the system (the rpool and the pv-grub volumes) are 2048 and 2064. The device IDs increment by 16:

root@openindiana:~# format < /dev/null
Searching for disks...
Failed to inquiry this logical diskdone


AVAILABLE DISK SELECTIONS:
       0. c0t0d0 <drive type unknown>
          /xpvd/xdf@2048
       1. c0t1d0 <??????HH???X?[??????? cyl 4095 alt 0 hd 128 sec 32>
          /xpvd/xdf@2064
Specify disk (enter its number):

So now we link in the new devices:

root@openindiana:~# ./links.sh c0t2d0 2080
root@openindiana:~# ./links.sh c0t3d0 2096
root@openindiana:~# ./links.sh c0t4d0 2112
root@openindiana:~# ./links.sh c0t5d0 2128

root@openindiana:~# format < /dev/null
Searching for disks...
Failed to inquiry this logical diskFailed to inquiry this logical diskFailed to inquiry this logical diskFailed to inquiry this logical diskFailed to inquiry this logical diskdone


AVAILABLE DISK SELECTIONS:
       0. c0t0d0 <drive type unknown>
          /xpvd/xdf@2048
       1. c0t1d0 <??????HH???X?[??????? cyl 4095 alt 0 hd 128 sec 32>
          /xpvd/xdf@2064
       2. c0t2d0 <??????HH???X?[??????? cyl 16709 alt 0 hd 255 sec 63>
          /xpvd/xdf@2080
       3. c0t3d0 <??????HH???X?[??????? cyl 16709 alt 0 hd 255 sec 63>
          /xpvd/xdf@2096
       4. c0t4d0 <??????HH???X?[??????? cyl 16709 alt 0 hd 255 sec 63>
          /xpvd/xdf@2112
       5. c0t5d0 <??????HH???X?[??????? cyl 16709 alt 0 hd 255 sec 63>
          /xpvd/xdf@2128
Specify disk (enter its number):

Create our ZFS pool:

root@openindiana:~# zpool create tank mirror c0t2d0 c0t3d0 mirror c0t4d0 c0t5d0
root@openindiana:~# zpool list tank
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
tank   254G    80K   254G     0%  1.00x  ONLINE  -

Once the EFI labels have been written to the disks, format stops throwing errors on them, as well:

root@openindiana:~# format < /dev/null
Searching for disks...
Failed to inquiry this logical diskdone

AVAILABLE DISK SELECTIONS:
       0. c0t0d0 <drive type unknown>
          /xpvd/xdf@2048
       1. c0t1d0 <??????HH???X?[??????? cyl 4095 alt 0 hd 128 sec 32>
          /xpvd/xdf@2064
       2. c0t2d0 <Unknown-Unknown-0001-128.00GB>
          /xpvd/xdf@2080
       3. c0t3d0 <Unknown-Unknown-0001-128.00GB>
          /xpvd/xdf@2096
       4. c0t4d0 <Unknown-Unknown-0001-128.00GB>
          /xpvd/xdf@2112
       5. c0t5d0 <Unknown-Unknown-0001-128.00GB>
          /xpvd/xdf@2128
Specify disk (enter its number):

And, just for fun:

root@openindiana:~# dd if=/dev/urandom of=/tank/random bs=1024 count=204800
204800+0 records in
204800+0 records out
209715200 bytes (210 MB) copied, 9.14382 s, 22.9 MB/s

So now you have ZFS on EBS, with the ability to do anything ZFS can. Snapshots will be much, much faster than EBS snapshots (though not complete copies, and will obviously be lost if your pool is lost, whereas EBS snapshots are complete copies of the volume and can be cloned and mounted out of band), enable compression, dedup (though this would probably be terrifyingly slow on EC2), and so on.

links.sh is available here.

Virtual Networking

This is fodder for another post, but something I’ve done elsewhere is to use Crossbow to create a virtual network with zones and VirtualBox VMs. The global zone runs OpenVPN, giving clients access to these private resources. This model seems perfectly suited to EC2, given the IP assignment limitations noted above. Unfortunately I don’t’ imagine VirtualBox is an option here, but even just a private network of zones would be extremely useful.

And perhaps someday EC2 will let you assign multiple Elastic IPs to an instance.

Conclusion

While there are still a few bugs to be worked out, this proof of concept AMI does work and is ready to have its tires kicked.

I’m pretty stoked to have Solaris available on EC2. Thanks Andrzej and Nexenta both!

Posted Sunday, April 17th, at 7:18 AM (∞).
Comments
blog comments powered by Disqus

Powered by Tumblr; themed by Adam Lloyd.