unimplemented trap

Migrating from Posterous to Flickr

Thu, 27 Sep 2012 15:29:00 -0500

I have a Posterous blog to post pictures and videos of my kids so the Grandparents can see them. When I signed up for Posterous, the agreement said you own the data. They haven’t changed that (yet), but seeing how the evil Twitter overlords are treating their developers, I don’t want to find out too late in the game.

Our site has just short of 3000 posts, containing both photos and video. We also have a Flickr Pro account, but at the time I set it up in early 2008, Flickr didn’t have video hosting yet. I don’t think Posterous had autopost to Flickr yet anyway, but I don’t remember. Needless to say, I never turned it on.

So now I’m stuck trying to get data off of Posterous. I tried several autoimports from Wordpress and Squarespace, but the APIs must have changed and there are problems. Wordpress took theirs down to fix it, and when I used the SquareSpace import, all my photos came in sideways.

I looked into autopost, thinking I could script it to move it to another site. But when I tested Wordpress, it only linked back to the original on Posterous, leaving my data in the hands of the aforementioned evil Twitter overlords.

I tested Flickr… and success! It uploaded the image to Flickr. Now how to automate this. I found a posterous backup script on github (edit for your hostname and backup directory path), and was able to successful get the XML, but many of the media downloads failed. When looking into the autopost functionality, I noticed a common URL structure for it.

http://posterous.com/posts/autopost/000000000?external=11111

In that example string, the zeroes are the post ID, and the ones are the source we’re exporting too. Armed with that knowledge we now have to get the media IDs out of the XML files.

OS X now has a program called xmllint. I don’t know when they got this, but we’re going to use it to strip out only the contents of the id tag. First concatenate all your XML files pulled from the posterus_backup.py script into one file.

cat page* > collectedxml.xml

Now we’ll run xmllint on the XML file. This will strip out the contents of the id tag, and clean it up.

echo "cat  //id/text()" | xmllint --shell collectedxml.xml | egrep '^\w' >> media_ids.txt

Now we’ve got all the media IDs inside the file media_ids.txt, each on it’s own line. I tried using a shell variable to complete the URL and hit it via curl, but as you need to be logged into Posterous, I couldn’t make it work. So I tried AppleScript.

I found a script on StackExchange that will open a file with URLs in it, line by line and then close them. Bingo!

Only my file doesn’t contain complete URLs, it only has IDs. We’ll use sed to append the needed text before and after the ID. (Be sure to replace 111111 with the ID for your external source (drag an autopost link from a Posterous post into the location bar in your browser, and you’ll see it).

sed 's/^/http:\/\/posterous.com\/posts\/autopost\//g;s/$/?external=111111/g' media_ids.txt > media_urls.txt

Now we have a file with complete autopost URLs on each line. Let’s look at the AppleScript. The JavaScript check was glitchy for me, so I used a different method to check for the text on the page when it’s loaded. If it can’t find the text, the page hasn’t finished loading and it doesn’t close the window.

set thePath to (path to desktop as Unicode text) & "media_urls.txt"
set theFile to (open for access file thePath)
set theContent to (read theFile)
close access theFile

set theURLs to every paragraph of theContent


tell application "Safari"
    repeat with theURL in theURLs
        make new document
        set URL of front document to theURL
        delay 2
        set PageText to the text of front document
        if PageText contains "Flickr" then close front document
    end repeat
end tell

I copied this into a new document in AppleScript Editor, and saved my media_urls.txt to my ~/Desktop folder. Click Run. If you have 3000 images/videos to migrate like me, it will take a while. But at least it will be out of the hands of the evil Twitter Overlords.

Getting started with Puppet

Wed, 29 Aug 2012 00:13:00 -0500

Gary Larizza and Chris Barker of Puppet Labs hosted a webinar today on a special use case of using Puppet, namely to manage Mac OS X workstations. I don’t see it posted online yet, but check here for Puppet Labs to post the webinar later on.

This post for now is going to assume you have a Puppet server up and running. If this is an obstacle, I’d recommend downloading the Learning Puppet VM, which supports up to 10 nodes.

Installing and configuring the agent (client)

Every server and client needs puppet and facter installed. You can download the Mac packages here, and deploy them however you would install a package (Apple Remote Desktop, Munki, Casper, etc). I used facter-1.6.11.dmg and puppet-2.7.19.dmg.

The glaring bug in the Mac package of puppet is that the puppet user and group are not created upon installation. Fortunately enough, a couple puppet one-liners easily remedy that.

sudo puppet resource user puppet ensure=present

And again for the group.

sudo puppet resource group puppet ensure=present

You’ll see output like this showing that it’s been created:

notice: /Group[puppet]/ensure: created

group { 'puppet':

  ensure => 'present',

}

Now we add the user puppet to the group puppet.

sudo dscl . append /Groups/puppet GroupMembership puppet

I also want to hide this user from the login window. If you use a blank login prompt, you don’t really need this.

sudo defaults write /Library/Preferences/com.apple.loginwindow HiddenUsersList -array-add puppet

Now that we’ve got that done, lets create our puppet.conf file. I like to use vim in Terminal, but you can use any editor you like. Create a file in /etc/puppet called puppet.conf. We want to define our server name in this file. I also like to define plugin sync, and if you have wonky DHCP/DNS issues as described here, you can even define certname. Here’s what I started with.

[main]
pluginsync = true
server=master.puppet.domain
[agent]
certname=machost.domain

After that’s done, we need to set ownership of the /etc/puppet directory.

sudo chown -R puppet:puppet /etc/puppet

Next, let’s run puppet to generate the SSL cert request to the server. We’ll get back to this later.

sudo puppet agent -t

Config the node on the server

For this next section, I’ll use Gary’s module he introduced today, which includes a class for installing Firefox.

From your Puppet server, you can install it with this command:

sudo puppet module install glarizza-osx_management

You can also download it from Puppet Forge.

To keep things simple, I’ll only use the /etc/puppet/manifests/sites.pp file. It’s more likely that you’ll segment into other nested files, but for this example, we can start here.

Inside this file, lets define a node. In this case I’m defining this as default, but for a specific node, you would replace default with the name of the node (either DNS or defined as certname in puppet.conf) wrapped in single quotes (‘nodename’).

node default {
include osx_management::firefox
}

This shows that I’m including the manifest firefox, included within the osx_management module we just installed. Let’s look at this manifest which is installed in /etc/puppet/modules/osx_deployment/.

class osx_management::firefox {
  $package_name = 'Firefox%2015.0.dmg'

  package { $package_name:
    ensure   => present,
    provider => appdmg,
source => "http://download-origin.cdn.mozilla.net/pub/mozilla.org/firefox/releases/15.0/mac/en-US/Firefox%2015.0.dmg",
  }
}

I modified this from the original, which includes version 14 of Firefox, but it does the same thing. It will pull down the .dmg file using curl, it will use the provider appdmg to tell it to copy the .app file to the /Applications directory.

Last, we need to sign the certificate from our node. You can list the certs waiting to be signed with this command.

puppet cert list

Now sign the cert.

puppet sign certname

Back to the client

Now lets go back to our client and see if this all works. Let’s run the same command again. This time we have the SSL cert signed, so it will actually do something.

sudo puppet agent -t

Now we see puppet do it’s thing. Lots of lines referencing .rb files rush past in Terminal and we eventually get to the catalog run at the end. You should see at the end of it all a notice line:

notice: /Stage[main]/Osx_management::Firefox/Package[Firefox.dmg]/ensure: created

Success!

Universal Type Server not accepting any logins

Wed, 22 Jun 2011 12:59:00 -0500

We had a server crash, and went it came back up, Universal Type Server 2.1 wouldn’t accept any user logins. When I went to the UTS Users & Groups administration (port 8080) site, I couldn’t login with administrator. I know the password wasn’t changed.

Sounds like something got corrupted. I tried reinstalling UTS, but it wanted all UTS processes to be shutdown first.

First unload the launchd process:

sudo launchctl unload /Library/LaunchDaemons/com.extensis.server.web.admin.launchd.plist

Then stop the Java runtime:

sudo java -jar /Applications/Extensis/Universal\ Type\ Server/applications/jetty/stop.jar

This one spit out errors on authentication, confirming the database is likely corrupt.

Lastly, we shutdown JBoss:

sudo /Applications/Extensis/Universal\ Type\ Server/applications/jboss/bin/shutdown.sh -S

Now, following this Extensis KnowledgeBase article, I was able to reinstall UTS, get into the Administration Site (port 18081) and restore the database from backup.

Reposado

Tue, 21 Jun 2011 10:23:00 -0500

I’ve replaced our Mac OS X Server’s Software Update Server (SUS) with an open source alternative done by Walt Disney Animation Studios, or more specifically, Greg Neagle.

The software is called Reposado, and it uses Python, curl and a web server (Apache in my case) to replicate Apple’s SUS. It also offers furthur capability allowing you to create branches, allowing control over testing before releasing to all.

The biggest benefit at least for me, is that it will run easily on Linux. Which means you can easily run it on a VM. I setup a Debian VM with the default web server selection, and only needed to add ssh, curl and the reposado software itself to get up and running. You’ll want to kick off your initial repo_sync at night or on the weekend though, the full library of Apple’s updates is ~44 GB.