On Learning Vagrant

tl;dr I wanted to learn Vagrant so I would know what Packer does and what Tim's templates and Joe's and Rich's scripts are really doing to improve the process of working with many virtual machines. Along the way I learned to make a catalog of OS X versions for Vagrant allowing me to spin up OS X VMs of any version at will and I will show you how to do that too.


This post will begin a series of articles on using Vagrant to quickly and easily create OS X based virtual machines.

I was first introduced to Vagrant in 2013 by either one (or both!) of Gary Larizza's or Graham Gilbert's posts. I remember thinking it was kinda cool but not really something I needed to put into use (also they were both advocating tools that help with Vagrant but I still didn't know Vagrant). I have access to between 4-8 machines on a daily basis for testing purposes and at the time my portable was a MacBook Air and my iMac still had spinning rust. Not ideal for VMs compared to the i7 with SSD and 16Gigs of RAM I have now.

Fast forward to PSU 2015, both Joseph Chilcote and Rich Trouton are doing presentations on using virtual machines for testing etc. This Vagrantfile Joe showed during the demo that spit out a list of available updates particularly caught my interest because I had gone through the process of updating AutoDMG's UpdateProfiles.plist for new items in the past.

Once I arrived back home I briefly played with vfuse and several other scripts etc. for speedily creating VMs. I didn't really understand what was happening with Packer, Tim's templates or any of that but I managed to find a fix a few problems I was having with some quick Google searches. Somewhat frustrated I put Vagrant away. I was still using the low powered Air and I had to get ready for the start of school. "This won't help me do anything now."

IT work in a school district, in my experience, is pretty cyclical in nature. There are some rough spots at the start and end of the school year and there is a bit of a slowdown in winter. This slowdown is great for research, development, planning, and experimentation. This year I've been feeling things wind down for two weeks or so, and finally at the end of last week I started putting a plan together.

Ultimately I want to get several things going here; Imagr, BSDPy, osquery with logging, and some tweaks to munki and munki-enroll. I knew that one way to get these things going more quickly was Docker and I also knew that a quick way to learn Docker would be to spin up Linux VMs. This is where Vagrant comes back into the picture.

The Vagrant documentation is pretty amazing and I tore into it. I studied it and took notes (physical ones, with a pen). I created some Linux boxes from scratch and I played around. I wanted to create some OS X base boxes for myself as well. I googled and found nothing on creating an OS X base box from scratch (until clicking links further down the results page while creating this post :/ ). I wanted to create some base boxes by hand in Vagrant before learning how Packer improves that process, before learning what Tim's templates do for me so, ultimately, I would know why vfuse + AutoDMG makes that even better.

On the way I learned something pretty cool. Namely that you can create your own catalog of Vagrant base boxes allowing you to spin up, in moments, OS X virtual machines for any OS version you have an installer app for.

The next few posts will show you how to do that too.

Making Shared Configuration Profiles Unique

Several people in the community have, from time to time, shared Configuration Profiles via Dropbox links, forum posts, or github. Nick McSpadden has graciously shared a fairly large collection.

There are a few problems one is faced with after copying so many profiles from someone else this way.

1. The top-level PayloadIdentifier is not ours
2. The PayloadUUIDs are no longer unique

The first problem is easily solved with existing tools.

sed -i '' 's/org\.sacredsf/org\.glenbrook225/' ./*.mobileconfig

The second problem…not so much.

I wrote a little something to create new UUIDs for the configs.

It works but it needs two things that I can think of.

1. the nested PayloadIdentifiers should update. I'm thinking something like Apple does but without the text. Maybe formatted like so: "new-top-UUID.new-payload-UUID"
2. sometimes the outer-level moves from the top of the file to the bottom. I don't know if this is because dictionaries in python have no order or some other reason but it makes them less human-readable.

Voice Updates repo_sync Stats

After Tim Sutton hopped into ##osx-server and freaked everyone out yesterday afternoon I decided to gather some before and after info.

Before:

df -H

Filesystem                 Size  Used Avail Use% Mounted on

/dev/mapper/vgpool-lv_var  476G  239G  215G  53% /var

After repo_sync run (deprecated pkgs NOT purged):

/dev/mapper/vgpool-lv_var  476G  276G  179G  61% /var

I ran the repo_sync by hand before leaving work for the evening in a screen session. Here is the time info. YMMV depends on bandwidth etc. etc.

repo_sync run ended

real    189m24.841s

user    7m26.358s

sys     7m15.786s

There were 22 deprecated pkgs that were not voices cached (mostly xprotect etc.) I did a 

repoutil --purge-product all-deprecated

Which removed those 22 updates. This left only the voices from 2013-07-24, 2012-05-13, and 2012-03-16 as deprecated items.

Filesystem                 Size  Used Avail Use% Mounted on

/dev/mapper/vgpool-lv_var  476G  274G  181G  61% /var

I added all the new voices in one shot with

repoutil --add-product all stable

Then got rid of the deprecated ones without having to remove them from the catalogs via

repoutil --purge-product all-deprecated --force

This finally brought the final disk usage to:

Filesystem                 Size  Used Avail Use% Mounted on

/dev/mapper/vgpool-lv_var  476G  222G  232G  49% /var


tl;dr You need a few hours and about 37GB to download all the new voices. But the new voices alone actually take up less space than the old ones did.

Yosemite SUCatalog Size

When I noticed repo_sync downloading all the OS X voices again I got mildly concerned about storage space.

Luckily, I learned that lesson once already and /var is mounted on top of an LVM storage pool in the Ubuntu VM that runs reposado. After a quick 'df -h' I switched over to ##osx-server and asked about the yoyo's storage requirements. gneagle suggested I do some investigation on my own and publish what I discovered.

I decided to stop the repo_sync run for the time being and disable the cronjob.

Later, I got curious and whipped this up in an iPython notebook after briefly taking a look at the catalog itself.

#!/usr/bin/python

import plistlib
import urllib2

catalog_url = 'http://swscan.apple.com/content/catalogs/others/index-10.10-10.9-mountainlion-lion-snowleopard-leopard.merged-1.sucatalog'
total_size = 0.0

yoyo_sucatalog = urllib2.urlopen(catalog_url).read()
yoyo_plist = plistlib.readPlistFromString(yoyo_sucatalog)
yoyo_prod = yoyo_plist['Products']

for k in yoyo_prod:
    for item in yoyo_prod[k]['Packages']:
        total_size += item['Size']

print str(total_size / 1073741824) + ' GiB'
print str(total_size / 1000000000) + ' GB'

And the output?

128.361939469 GiB
137.827583018 GB

So there you have it. Approximately 130-140 Gigs (for varying definitions of Gig).

There is probably some insignificant amount of overhead in everything that comes down from the server too. *shrug*

Looks like I need to add a drive to the VM and expand the pool. :)

munki is so damn cool

This post only exists because twitter's character count is too small…

but FYI munki is so damn cool.

The following items will be installed or upgraded:
        + duti-1.5.2
            set default applications for doc types
        + duti_launchagent-1.0
            runs duti at login on dir /Library/Glenbrook225/duti
        + AcrobatXProCS6-10.1.1
        + AcrobatUpd-10.1.10
        + duti_acrobatpro-1.1
            sets Acrobat Pro as editor for pdf at login via duti and duti launchagent

Only one of those things was actually in the manifest.

NICE.

pkgbuild many MAS apps with bash

I wanted to quickly create pkgs for iWork and iLife from apps with an institutional ID receipt because I am experimenting with AutoDMG and Lightning.

It takes a bit of time for some of these to build and I didn't want to wait around so here is what I did…

Start an array by just typing array=( into the command line and then hitting <return>

admin_dist@golden ~ $ array=( 

Drag and drop the apps into the Terminal window hitting <return> after each

> /Applications/GarageBand.app 

> /Applications/iMovie.app 

> /Applications/iPhoto.app 

> /Applications/Keynote.app 

> /Applications/Numbers.app 

> /Applications/Pages.app

When they are all in finish off the array

> ) 

Now for the good stuff.  :D

For each path in the array

admin_dist@golden ~ $ for path in "${array[@]}"; do

 greedily chomp from the front to the last / using parameter expansion

> app_name="${path##*/}"

chomp from the end to the first period

> name="${app_name%.*}"

get the version of the thing at $path using command substitution and an assignment

> version=$(defaults read "${path}"/Contents/Info CFBundleShortVersionString)

build it with a pretty name

> pkgbuild --component "${path}" $HOME/Desktop/"${name}-${version}".pkg
> done

This will give you nice installer packages on the Desktop of the form "GarageBand-10.0.2.pkg".

Here is gist with a full script instead of interactive command line entries.

Quick Tip - updatejournal.plist

I just migrated from an old 13" MacBook Pro to a new 13" Air.

I quickly found out that Migration Assistant in Mavericks over Thunderbolt is mostly broken.

Not a problem for me, I know how to use rsync with --exclude!

After getting my user directories moved over I found something rather annoying. MAS still showed that I had installed a bunch of updates, particularly developer seeds, that I most certainly was not running on this new machine.

I poked around for a few minutes with pkgutil, ls'd a bunch of folders, poked & prodded until finally throwing my hands up in exasperation.

Where the heck is this coming from!?!

I methodically started going through my User-level Library (as I now knew the updates only showed up for my standard user and not the admin) and found the offender.

After banishing this annoying little bugger to to the Trash my update list is free and clear.