It can be a challenge to administer multiple Linux machines, particularly when this isn't your primary job.  I deal with about 6 machines on a daily basis, and the various configurations tend to blur together.  If I didn't have a system for keeping it straight, I'd be lost.  I know because I still have memories of "the time before the system" and it was madness.

I tried a few existing configuration management tools (cfengine, puppet, bcfg2) and everything seemed to be more complex than I wanted, so I wrote my own: Xenadu. Originally, Xenadu was specialized for creating Xen guest images, but I now use it to administer both virtual and physical machines.  It's one of those "makes the impossible possible" tools, for me, and I'm actually using it on a few production systems.

My primary goal with Xenadu is to track my system administration with a version control program like git.  By doing this, I essentially get automatic documentation and backups of my sysadmin activities.  If I made a change to a system and it broke, I would always be able to get a previous version of the system configuration out of git.

Once you have fully defined a system with Xenadu, it becomes quite easy to create other machines that have similar configurations.  This is great for keeping a stage/production environment in sync, or for creating several instances of a particular production server.

An example Xenadu configuration for a computer named "davidbowie" might look like this:

#!/usr/bin/env python
from Xenadu import XenaduConfig, Perm
mapping = [
    ['/etc/hosts', "hosts", Perm.root_644],
    ['/etc/network/interfaces', "interfaces", Perm.root_644],
    ]
env = { 'ssh': { "user": "root", "address": "davidbowie.example.com" } }
XenaduConfig(env, mapping)

Save the file as davidbowie.py, and you're ready to manage that computer.  In this case, two files (hosts and interfaces) are going to be tracked by Xenadu.  Now you can edit the files locally, check them into version control, etc.  If you've edited your local copy of hosts and want to send it to your remote system:

./davidbowie.py --push hosts

A particularly useful feature is that you can specify a file according to its local or remote filename, so the following are equivalent:

./davidbowie.py --push hosts
./davidbowie.py --push /etc/hosts

You can also --get, which downloads a file, and --getall, which downloads every file specified in your Xenadu system definition.  Check out the readme file on the Xenadu website for more information.

In the olden days, I would perform system administration by logging onto the system itself, editing files as I went along.  You can still do this with Xenadu, but make sure to --get your changes from the remote host, so that you can check it into version control.  My current workflow is to make changes locally, then --push them to the remote host.  Give it a try!

As John Vinson reports on WebProNews: "If you ever need proof that truth is stranger than fiction, simply give this story a read." He's referring to a crazy series of events involving a New York-area newspaper and an article I posted to my blog on Tuesday titled, "When a newspaper rips off your blog, then taunts you about it". (more coverage here, here, and here) There was a pretty vigorous response to the story, to put it mildly!  Since everyone has now had 24 hours to calm down, I'm going to present the complete sequence of events, and close with a brief update on the current state of the situation.

This story really starts at 8:59PM on March 23rd, when I noticed this tweet from Mark Cuban:

Got One !!! Finally..http://www.petitelapgiraffe.com/

Since I had never seen any advertising related to this website, I was completely floored.  If you have no idea what this is about, take a minute to check it out.  Immediately, I had questions: Could these animals really exist?  Are they so rare that only billionaires can afford them?  How does a Russian farm come up with such a polished website?  (that work wasn't cheap) What of the live video feed!? I'll admit it straight-up: I totally bought into the illusion.  It was brilliant!  As I tried to learn more, I grew increasingly (and tragically) skeptical, eventually composing a blog entry at 9:56PM debunking the advertisements. Then, at 10:01PM on March 23rd, I posted this update to twitter:

I want a Petite Lap Giraffe! http://www.iandennismiller.com/blog/2011/03/petite-lap-giraffes/ #petitelapgiraffe

It turns out I wasn't the only person who wanted to know if these animals were real, and thousands of people started visiting my blog to find out the answer.  I had approached the viral ads like a puzzle, so it was a lot of fun for me to debunk the myth.  A lot of people seemed to be genuinely disappointed that the giraffes weren't real, so I spent some additional time consoling the desolate and despondent, who were populating my blog's comments section.

For the next few days, I Googled for "petite lap giraffe" about once per day, and idly clicked through the other coverage of the meme.  I had been following a sequence of articles posted by the LongIslandPress.com (since they seemed to be the only newspaper publishing anything about it), but around 2:00PM on Tuesday March 29th, one of their articles particularly caught my eye.  It had been published Monday, March 28th at 5:02PM.  Remember that time, because it's going to become really important a little later on.

As with the original Petite Lap Giraffe debunking, I started sensing that something was wrong with this article.  I wish I could link to the original article, but alas, it has since been swallowed up by the memory hole.  Fortunately, I created a PDF archive, so for posterity's sake, this screenshot will serve as a mirror of the original LongIslandPress.com article:

There were three sentences that jumped out at me, particularly the one about the stock image.  In my article debunking the giraffes, that was the detail that had sealed the deal for me.  But had they just claimed they were the ones who performed this research?  It sounded that way to me, so at 2:07PM I posted the following comment in response to the article:

I’m a little disappointed that you didn’t city [sic] my March 23 blog post on this topic, since it is the original source of the information you mention in this article. I thought it was standard to provide an attribution?

http://iandennismiller.com/blog/2011/03/petite-lap-giraffes

No big deal.  I checked back around 2:40PM, and it appeared as if my comment had been deleted.  That kindof irked me, so at 2:44PM, I tried again (this time, a little more forcefully):

I’m disappointed you didn’t city [sic] my March 23 blog post, where I actually conducted the research you are taking credit for. Specifically, I uncovered the link to the Grey Group, and I also discovered the “hot tub” stock image. At a minimum, you should provide attribution:

http://iandennismiller.com/blog/2011/03/petite-lap-giraffes

...but it’s dishonest to claim this research as your own. Usually, it’s considered plagiarism.

Again, it seemed like my comment had been deleted, but I soon realized that it was actually being held for moderation, which is a pretty normal thing in the blog world.  Again, no big deal.  More waiting. When I checked back at 3:05PM, there was a response! However, this was not the response I was expecting.  Someone at LongIslandPress.com had altered the original article, replacing this sentence:

And the cute little guy in the bath tub? Well, that’s a stock image with the cute little guy added in.

with this sentence:

A quick domain name lookup...which is free and public information...will give you those details.

Here is my record of their article, at that time (3:05PM).

By this point, I was pretty sure someone was trying to cover something up, so I told some of my friends about the situation and began summarizing my findings in a new blog article, which I published at 3:39PM.  The next step happened when a friend pinged me, pointing out that the article had been updated yet again.  By 4:23PM, the article included this sentence:

A quick domain name lookup...which is free and public information...will give you those details, which we acquired–you know, being a newspaper with research capabilities and all–of our own accord (although some are trying to claim this information as their own “discovery” as a way to promote their own personal website! But enough of that...)

Here is a screenshot of the article at 4:23PM:

It seemed like they had clearly received my comment, and although they were refusing to publish it, they were certainly responding to it!  Shortly after I was notified of this latest edit, I posted my final comment on their article:

Whatever - I think you did a pretty lame thing here.  You deleted the detail about the stock photo and are trying to make it sound like you did the rest on your own.  ...but I saved a copy of all 3 versions of your article, and it's pretty clear you know which details you lifted.  I'm kindof amazed at how shameless you are about this (really, all I asked for was proper attribution) but I actually don't have time to pursue this further.

Captcha: You win - lol

This comment seems to have been deleted outright, rather than being held for moderation,  but I did have a ton of work to do, and I didn't want to deal with this right now.  But hey: this is what friends are for.  They kept asking me questions about the article, and I updated my own article to mention LongIslandPress.com's inflammatory remark.  It seemed to me like LongIslandPress.com had provided a de facto admission of their deeds, so I started asking some forums for advice about how to report a journalistic ethics complaint.  At 8:46PM, I submitted the following blurb to slashdot.org:

"I've been keeping an eye on this viral marketing campaign called Petite Lap Giraffe — it's the DirecTV ads with the Russian guy and the tiny giraffe. I was pretty quick to debunk the existence of the giraffes, so a lot of people have been visiting my blog as a result. Today, I noticed a New-York area newspaper that was represented my research as their own, so I asked them to link to my blog (i.e. provide attribution). What ended up happening perfectly illustrates that newspapers just don't understand how the Internet works..."

The real break occurred at 11:27PM, when the story was featured on the front page of slashdot.  This brought on a flood of attention, but it seemed like many people weren't buying my account of events.  I have to admit: I thought these critics had a point. Again, questions crept into my consciousness: What if I got it all wrong?  What if LongIslandPress.com really did conduct their own research?  Freak Out!!

I spent the next 3 hours responding to criticisms, but at 2:29AM on March 30, Another Slashdotter posted the following comment:

Have you looked through your logs to see if anybody from their domain name/IP address visited your blog right before the article was published?

Obviously!  The Internet horde definitely needs to hear about the logs!  Earlier in the evening, I had been watching the real-time logs for a project, and I remembered seeing a visitor from the Long Island area.  I had actually done a reverse-DNS lookup at that time, and it turned out to have originated at the hostname mail.longislandpress.com, so this detail was lurking in my memory. In other words, I had a pretty good hunch about what IP address to look for in my personal blog's server access logs.

The logs contained the smoking gun, and these are the two entries originating from mail.longislandpress.com that sealed the deal:

XXX.XXX.XXX.XX – - [28/Mar/2011:20:56:31 +0000] “GET /favicon.ico HTTP/1.0″ 304 – “-” “Mozilla/5.0 
    (Windows NT 5.1; rv:2.0) Gecko/20100101 Firefox/4.0″</code>

XXX.XXX.XXX.XX - - [29/Mar/2011:19:40:30 +0000] "GET /blog/2011/03/total-bummer-longislandpress-
    com-plagiarism-and-coverup/ HTTP/1.0" 200 13398 "http://www.longislandpress.com/[redacted 
    wordpress admin.php]" "Mozilla/5.0 (Windows NT 5.1; rv:2.0) Gecko/20100101 Firefox/4.

In English, my logs contained records indicating LongIslandPress.com had visited my website at 4:56PM, just 6 minutes before they published their article.  This is why it was so important that the LongIslandPress.com article was published at 5:02PM on March 28th.

Here is the technical interpretation of the first log entry:

[the favicon.ico] was served with an HTTP 304 code (meaning “unmodified”) which suggests the favicon was already in someone’s cache. That means the page had previously been loaded.  The timestamp is 20:56:31 UTC, meaning it was 4:56PM in New York. The timestamp on the original Long Island Press article is 5:02PM.

And here is the interpretation of the second entry:

Someone:

• using the same IP address as the [favicon.ico] log entry
• using the same browser as before (or at least providing the same UserAgent)
• using the LIP wordpress admin interface (as indicated by the Referer field)
• ...clicked through to my site, in order to read [the article about LongIslandPress.com]

This satisfied everybody, and finally I could get some rest.  The next major event occurred around 9:29AM on March 30, when LongIslandPress.com took their article offline.  At 3:45PM, LongIslandPress.com put the original article online again (minus the remarks), which included attribution.

Since then, I've been keeping an eye on LongIslandPress.com, and at 10:27AM on March 31, they posted an article detailing a local Long Island connection to the Petite Lap Giraffe thing.  So that's why LongIslandPress.com had written so many articles about the Petite Lap Giraffe!  (this really satisfied a nagging question I had been puzzling over for several days).

Freedom, glorious freedom.

Once upon a time, I took a class based on  a single question: "what is freedom?"  We meandered through US history, identifying several distinct stages in the evolution of the definition of "freedom."  I was horrified to learn, during a discussion, that so many of my classmates wanted what I will call "freedom from information." Ah yes - Professor Sandage had a way of bringing the ugliest truths to the surface, for all to witness.

On the one hand, I can understand this desire for freedom from information: telemarketing, advertising, spam, the scrolling headlines at the bottom of a newscast...  well, any unsolicited attempt at selling things you don't care about.  On the other hand, I think we need more information instead of less, and we need effective tools to filter and manage that information so we only see what we care about.

The term "freedom" is muddied by historical contexts, but also through the process of etymological erosion.  With that said, I want to take a moment to discuss the expression, "free as in speech, not beer."

Free as in speech, not beer

"Free as in speech, not beer" is an expression that comes up in open source discussions all the time.  It's a little hard to unpack, unless you really dig into the dual meaning of the word "free."  Thanks to Wikipedia, we're part of the way there: the word "free" is used to mean two things: Gratis versus Libre.  We call both of these terms "free" nowadays, but once upon a time, there were different words because they are totally different concepts.  Gratis means "without charge" whereas Libre is more like "liberty" or "freedom."

So what is free speech?  Of course, that's the freedom to say what you want (so long as you accept the consequences for what you've said).  And free beer?  Well, that would mean beer that is provided at no cost.  I think the key is this: although you are free to say what you want, you could well end up in court for it (e.g. slander) and your expression won't come free of charge.  On the flipside, you can provide beer free of charge, but not to someone who is 15 years old, so you may not freely provide beer to anyone you wish.

In other words, speech embodies Libre (but not necessarily Gratis) perfectly.  Likewise, beer embodies Gratis very well, at the same time that beer is so closely regulated by many governments that it is hardly "libre."  Nevertheless, everybody likes a good party with some beer pro gratis.

The Free House, and the Public House

Speaking of free beer, the Free House is definitely not a place to find such a zero-cost beverage.  For starters, the term Free House is mostly British, and always beer-related.  It refers to a Public House (which you may know as a "pub") that will sell any kind of beer they can get people to buy.  Contrast this with a Tied House, which sells beer manufactured by a single brewer, and you find that the Free House will have several brands on tap.  Here, the term "Free" is more like Libre, and is used in the context of the "free market."  ...and we all know that the free market isn't composed of things that are zero-cost.

When I was living in Berkeley, California there were two particularly good "Tied House" pubs that brewed and sold only their own brands of beer: Jupiter and Triple Rock.  I should also mention Pyramid, which had a pretty cool restaurant with their own beverages on tap.  This kind of pub is fun because they'll often have a sampler option to let you taste a small glass of everything they brew.  It's a great way to experience the full spectrum of beers, but a word of advice: start with the lightest stuff and progress towards darker.  The one exception to this rule is for hoppy beverages (e.g. IPA or APA), which might be light but which may have a pronounced bitter taste.  You might want to close it off with an APA, even after drinking the stouts.

Open Source Software

There's nothing that goes quite so well with open source software as a tasty hoppy beverage.  I like pairing Stone Brewing Company's Arrogant Bastard with GnuPG, the open source implementation of Phil Zimmerman's PGP (pretty good privacy) software.  Another favorite of mine is the Spaten Optimator paired with Wordpress.  More recently, I've taken a liking to Unibroue, the French Canadian brewer, who offers such brews as Tres Pistoles, which is an excellent complement to Python.  This last combination is probably the most dangerous of the group, because you might end up with excellent code, and you might end up with British comedy.

In the end of the day, free speech and free beer have a lot to do with open source software.  You see, licenses such as the GNU General Public License actually permit developers to charge for their software, while simultaneously requiring all GPL software to be published with its source code.  In this sense, the "free beer" part means the software isn't necessarily without cost, and the "free speech" part means you are required to publish the source code.  In other words, the Libre aspect of the GPL has an important restriction: you are not free to not publish the source code, which in turn provides the most fundamental tenet of open source software: you are free to read and distribute the source code.

I want to hedge my previous statement: the GPL is a famous topic of debate, so there's plenty of room to criticize anyone who says anything - at all - about the GPL or about open source software, either according to the letter of the license, or according to the spirit of the movement.

Let me sum it up like this: "free" means many things to many people throughout many time-periods, but for some reason, it almost always comes down to a matter of speech and beer.