When I first joined D2SI a few months ago, I was thrown into AWS pretty much right away. After playing Cloud Cowboy for a few days, something kept bothering me: OpenBSD my OS of choice was not available ! By not available I mean, there was not even a public/community/unsupported AMI (Amazon Machine Image) to play with.

Outrageous!

For those of you who do not know about OpenBSD, it is a free multi-platform 4.4BSD-based UNIX-like operating system with a strong focus on security and simplicity. It’s worth mentioning that it is responsible for the well-known OpenSSH; a detailed list of other software developed or maintained by the OpenBSD team can be found here.

In a nutshell, it’s a great operating system and I was missing my favorite toy!

It’s alive… not!

So I started looking at creating an AMI from scratch. Surprisingly that process ended up being very easy:

• create and loopback-mount a raw image containing a UFS filesystem
• extract the OpenBSD base sets (which are just regular tarballs) and kernel
• enable console output (so that one could « aws ec2 get-console-output »)
• install the boot loader on the image
• then use the ec2 tools to import the RAW image to S3, convert it into a volume (ec2-import-volume) which we can snapshot (ec2-create-snapshot) and create an AMI from (ec2-register)

A script that automates this is available on GitHub. It’s still rough around the edges and must be run from a -current OpenBSD machine but it does the job.

$ ./create-ami.sh -h 
./create-ami.sh[260]: -h: unknown option
usage: create-ami.sh [-in]
 -i /path/to/image
 -n only create the RAW image (not the AMI)
 -s image/AMI size (in GB; default to 1)

$ ./create-ami.sh 
===> creating image container
===> creating image filesystem
===> mounting image
===> fetching sets (can take some time)
===> fetching ec2-init
===> extracting sets
===> installing MP kernel
===> installing ec2-init
===> removing downloaded files
===> creating devices
===> storing entropy for the initial boot
===> installing master boot record
===> configuring the image
===> unmounting the image
===> image available at:
     /tmp/aws-ami.yIWPLf9Iu2/openbsd-20160210T223138Z

===> uploading image to S3 (can take some time)
Requesting volume size: 1 GB
<snip>
Uploading the manifest file
Uploading 1073741824 bytes across 103 parts
<snip>
The disk image for import-vol-fgd2p1en has been uploaded to Amazon S3 where it is being converted into an EBS volume. You may monitor the progress of this task by running ec2-describe-conversion-tasks. When the task is completed, you may use ec2-delete-disk-image to remove the image from S3.

===> converting image to volume (can take some time)

===> creating snapshot (can take some time)
<snip>

===> registering new AMI: openbsd-20160210T223138Z-20160210T223446Z
IMAGE ami-45358536

Using this newly created AMI, the machine booted fine but unfortunately no NIC (network interface card) was detected which left us with no way to access it. While AWS hypervision is based on Xen, it does not allow you to configure anything on the hypervisor side for obvious reasons — so there was no way to modify the Xen configuration to emulate a known network chip. Yes, that’s probably the reason I could not find any AMI in the first place… 😉

I can has net?

Not being a hardware nor kernel developer, my poor attempt at bringing my new toy on AWS came to an end pretty fast.
At that point I started remembering a few informal talks from fellow OpenBSD developers about native Xen domU support. Armed with my best impression of Shrek’s Puss in Boots I started begging around only to find out that OpenBSD developer Mike Belopuhov had already started working on it. His work is sponsored by Esdenera Networks (a German company that builds OpenBSD-based firewalls for Software Defined Networks).

Less than a couple of months later, Mike granted the community with this gift (extracted from the xen(4) man page):

xen driver performs HVM domU guest initialization, provides abstraction for virtual Xen interrupts, access to the XenStore configuration storage as well as a device probing facility for paravirtualized devices such as disk and network interfaces.

Ah, the playground is now opened o/

Let’s ride!

There was one last step needed before we could enjoy our shiny and new OpenBSD installation: inject the SSH public key into the instance to be able to connect. Most Linux distributions use cloud-init or that purpose. After a quick look at the code, I decided it was not worth porting this to OpenBSD. The code was already bloated enough trying to deal with most distributions out there and several features were out of scope at this point. Not to mention it is written in Python which would have meant adding several external packages to the default image.

Instead I came up with a very slim and stupid shell script that would only ask the AWS meta-data mock server for the required information. Under Linux, it would be similar to running curl(1) or wget(1) against http://169.254.169.254/latest/…
Code is available on GitHub and only supports the following for now:

• setting the hostname
• installing the provided SSH public key to /root/.ssh/authorized_keys
• executing user-data (if it starts with a shebang)
• displaying the host SSH fingerprints on the console (to match cloud-init)

I also imported boto3, awscli and the EC2 API Tools into OpenBSD ports so that they could be installed using our native package manager.

$ aws ec2 describe-instances 
  --filters "Name=tag:Name,Values=openbsd" 
  --query "Reservations[].Instances[].InstanceId" 
  --output text
i-0dd5a20a9d3e65767
$ aws ec2 get-console-output --instance-id i-0dd5a20a9d3e65767

i-0dd5a20a9d3e65767 +2173200+267272+0+647168 [72+577248+384230]=0xa6ce90
entry point at 0x1001000 [7205c766, 34000004, 24448b12, e040a304]
[ using 962192 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2016 OpenBSD. All rights reserved. http://www.OpenBSD.org

OpenBSD 5.9 (GENERIC.MP) #1869: Thu Feb 4 09:50:59 MST 2016
deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 1056964608 (1008MB)
avail mem = 1020796928 (973MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xeb01f (11 entries)
bios0: vendor Xen version "4.2.amazon" date 12/07/2015
bios0: Xen HVM domU
acpi0 at bios0: rev 2
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP APIC HPET WAET SSDT SSDT
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 11, 48 pins
ioapic0: misconfigured as apic 0, remapped to apid 1
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz, 2400.32 MHz
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,LONG,LAHF,ABM,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 100MHz
acpihpet0 at acpi0: 62500000 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
pvbus0 at mainbus0: Xen 4.2
xen0 at pvbus0: features 0x705, 32 grant table frames, event channel 3
"vfb" at xen0: device/vfb/0 not configured
"vbd" at xen0: device/vbd/768 not configured
xnf0 at xen0: event channel 5, address 06:da:1c:c0:fe:13
"console" at xen0: device/console/0 not configured
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
wd0 at pciide0 channel 0 drive 0:
wd0: 16-sector PIO, LBA48, 10240MB, 20971520 sectors
wd0(pciide0:0:0): using PIO mode 0, DMA mode 2
pciide0: channel 1 disabled (no drives)
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x01: SMBus disabled
vga1 at pci0 dev 2 function 0 "Cirrus Logic CL-GD5446" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
xspd0 at pci0 dev 3 function 0 "XenSource Platform Device" rev 0x01
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: density unknown
fd1 at fdc0 drive 1: density unknown
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
nvram: invalid checksum
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on wd0a (8f65337136a44ad6.a) swap on wd0b dump on wd0b
clock: unknown CMOS layout
Automatic boot in progress: starting file system checks.
/dev/rwd0a: file system is clean; not checking
setting tty flags
pf enabled
starting network
DHCPDISCOVER on xnf0 - interval 3
DHCPOFFER from 172.31.16.1 (06:90:d1:8f:bf:e1)
DHCPREQUEST on xnf0 to 255.255.255.255
DHCPACK from 172.31.16.1 (06:90:d1:8f:bf:e1)
bound to 172.31.16.127 -- renewal in 1800 seconds.
openssl: generating isakmpd/iked RSA keys... done.
ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519
starting early daemons: syslogd pflogd ntpd.
starting RPC daemons:.
savecore: /dev/wd0b: Device not configured
checking quotas: done.
clearing /tmp
kern.securelevel: 0 -> 1
creating runtime link editor directory cache.
preserving editor files.
starting network daemons: sshd smtpd.
ec2: #############################################################
ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
ec2: 1024 SHA256:ec6LNxkAlF5zZrdg8q01D2TCZ7rEeM6hVE17UM/y+PM root@ip-172-31-16-127.eu-west-1.compute.internal (DSA)
ec2: 256 SHA256:pbxs2mSzBsUbD48UwJeVkDFrN9TfHk5/6fDGLO0bouA root@ip-172-31-16-127.eu-west-1.compute.internal (ECDSA)
ec2: 256 SHA256:8jo+VAjDuQSZZ79ReNRTt6xdCwkdoJgUVewEcgcWFyc root@ip-172-31-16-127.eu-west-1.compute.internal (ED25519)
ec2: 2048 SHA256:4um8T6zdyq/jD+dYz8+PXL3285ja/WQy0K/SqE8XQY8 root@ip-172-31-16-127.eu-west-1.compute.internal (RSA)
ec2: -----END SSH HOST KEY FINGERPRINTS-----
ec2: #############################################################
starting local daemons: cron.
Wed Feb 10 10:40:29 UTC 2016

OpenBSD/amd64 (ip-172-31-16-127.eu-west-1.compute.internal) (tty00)

login:

While there’s still work to be done and testing to be made, we now have native support for the netfront Xen paravirtual networking interface. Currently block devices (i.e. hard drives) are still using the IDE emulation that Xen exposes, but no doubt work will happen sometimes to support the native blkfront interface.

– So was this all just for fun? – No!

There are very good reasons for wanting to run OpenBSD on AWS and in the Cloud in general.

Diversity is good: we don’t want to end up in a world where the only UNIX-like operating system left is Linux.

A typical OpenBSD installation has a very small footprint in term of storage size and memory consumption and makes it a very good candidate to build application stacks on.

OpenBSD is also already Cloud-friendly: we’ve had VirtIO (KVM, QEMU, VirtualBox) and VMT (VMware Tools driver) support for years. The system comes with IPsec and a BGP daemon by default which makes it a strong candidate for creating very secure virtual private network (VPN) gateways or for interconnecting VPC regions. Not to mention out-of-the-box vxlan(4) support for building overlay networks (virtualized layer 2 network over layer 3 network)…

Oh and NAT gateway anyone? I would trade iptables for PF anytime! (PF being the OpenBSD packet filter which made its way into Mac OS X and soon Oracle Solaris).

Nowadays, a typical online service is often a complex machinery involving lots of different components to achieve performance, redundancy, resiliency and elasticity. OpenBSD is a strong advocate for simplicity and building complex infrastructures out of simple and secure components is usually a good bet.

Same player shoots again

Can you share your toy?

While it may look familiar at first glance, OpenBSD is not yet another Linux distribution. It is a completely different operating system so do not be surprised if some standard commands syntax differs a bit. After all we are running a BSD userland, not a GNU one. That said, most opensource software are available through packages and these are the same as on Linux (e.g apache, nodejs, PHP, varnish, haproxy, squid, ansible, puppet, GNOME, KDE, chromium, firefox…). On OpenBSD (and on BSDs in general), there’s a clear separation between the base system and the packages: the base system is the complete operating system (kernel, userland, standard daemons…) and unlike Linux, it is developed and installed as a whole and not « packaged » from different projects.
External packages are installed under the /usr/local hierarchy and configuration files lie under /etc.

To wrap up, let’s install, enable and start the nginx web server. Note that we will be setting our package mirror path manually because we are running a development release and we want to make sure we point to a snapshot directory.

# export PKG_PATH=http://ftp.fr.openbsd.org/pub/OpenBSD/snapshots/packages/amd64
# pkg_add nginx
quirks-2.197 signed on 2016-02-08T22:17:20Z
Ambiguous: choose package for nginx
a    0: <None>
      1: nginx-1.9.10
      2: nginx-1.9.10-lua
      3: nginx-1.9.10-naxsi
      4: nginx-1.9.10-passenger
Your choice: 1
nginx-1.9.10: ok
The following new rcscripts were installed: /etc/rc.d/nginx
See rcctl(8) for details.
Look in /usr/local/share/doc/pkg-readmes for extra documentation.
# find /etc/nginx/ -type f
/etc/nginx/fastcgi_params
/etc/nginx/koi-utf
/etc/nginx/koi-win
/etc/nginx/mime.types
/etc/nginx/nginx.conf
/etc/nginx/scgi_params
/etc/nginx/uwsgi_params
/etc/nginx/win-utf
# rcctl enable nginx
# rcctl start nginx
nginx(ok)
# echo 'Welcome to nginx on OpenBSD!' 
    >/var/www/htdocs/index.html
# ftp -MVo - http://localhost
Welcome to nginx on OpenBSD!

Have fun!