MENU

by • January 23, 2014 • Apple, Drive Arrays Comments2868

“Permanent” vs “Temporary” files (a thesis)



Looks like Apple is having hard time keeping up with the demand for the new Mac Pros.

Word on the street is that shipping times have slipped to March. I’m hoping that the demand is far outweighing the supply. If this is indeed the case It would be a bitter sweet pill for Apple to swallow knowing that they let their most loyal users languish for so long. The questions and concerns from the digital content creation community was falling on deaf ears for far too long. I’m hoping that the demand has taught Apple (and the rest of the Apple media bubble) a lesson. Digital content creators still exist and we choose to work on the Apple platform. Please don’t ignore us any longer.

Of course I could be completely wrong.  The issue could be that the new US assembly plant has been really slow to meet even the modest demands. Ugh … scratch that last line. Lets keep to the idea that the entire Apple world starts and stops with us, the digital content creators. We need to stay in our Happy Place people.

I still haven’t made a new Mac Pro purchase yet. I’m a bit embarrassed. I kind of spearheaded the campaign to bring attention to Apple’s neglect and now that the new machines are finally available for purchase I haven’t taken the plunge.

One by one the barriers between my wallet and the purchase are disappearing.

I’ve gotten over the thought of purchasing an iMac. Yes I do think than an iMac would work fine for the HD editing I’m doing today. Heck, I know some people that are editing HD with a Mac Mini!! The modern codecs have kept the data rates small enough with HD footage that an iMac is plenty fast. But if I bought an iMac today I would probably be buying another iMac two years from now because two years from now I will probably be seeing a lot more 4k footage. I don’t think today’s iMac has enough headroom to handle that kind of data rate that 4k footage will demand.

The biggest issue I’m still having is the external storage solution. First off, the cost is a concern. I always expected to pay about $4500 for my new Mac Pro (even though I thought my specs would be higher for that price). I didn’t think I would have to spend another $1000 on an external storage device. I’m sure I’ll have whatever storage array I choose for the next 5 years but the initial cost (along with the new computer) is a sticking point.

Then there is the workflow issue. To me, RAID was a great technology before the dawn of cheap redundant backups to multiple physical devices was feasible. It provided redundancy and speed long before we all had NAS devices and networked Time Machine backups. Having your data exist in multiple physical and virtual locations has become incredibly easy. But I don’t believe that all data should be treated the same way. I see a huge difference between “temporary” storage and “permanent” storage. I also see a huge difference between “temporary” files and “permanent” files. (I’ll explain this in a second)

RAID devices scare me. I hate having “permanent” data tied to multiple drives at one time. I have a fear of drive failure destroying what I consider “permanent” data. And if my internal boot drive is not big enough then I will be forced to store “permanent” data on the RAID array. I understand the speed benefits. I understand that a RAID can be rebuilt if a drive fails. But I don’t trust it. It seems too fragile. Drobo is a bit different but I don’t think the Drobo’s data speeds are fast enough for editing.

Now for my explanation. Bear with me on this …
This is my thinking and of course YMMV.

My internal boot drive with all of my program files and applications is considered “permanent” storage. This drive is consistently backed up to a separate physical NAS drive via Time Machine. All important “working” files on this drive that are consistently changing during a project are also backed up to the cloud via Dropbox. To me, “permanent” files types are Lightroom libraries, any files that are created during a project, After Effects and Premiere project files, rendered Quicktimes, etc. All of these “working” files are stored on my boot drive, in my Dropbox folder, while being backed up to Time Machine. That means all of these important files exist in three physical places at all times as well as the cloud. If the “permanent” drive fails then I can work from my Dropbox on my laptop or another computer with my external media drive. Or if I have time I can restore via Time Machine.

I consider RAID arrays to be used as “temporary” storage. This is the place for Raw camera footage and photographs. These types of items get immediately backed up to a separate hard drive at the beginning of a project. They get deleted from the working RAID drive at the project completion. If the RAID crashes during a project then any media can be restored from the backup or, even better, work can continue with the archived footage via firewire or USB3. These files do not need to be backed up to Time Machine on a consistent basis. These files do not change. Backup once and forget about it. Plus, restoring a RAID array from a Time Machine backup would take FOREVER!!!

The problem with this workflow and the new Mac Pro is that my internal boot drive needs to be fairly large. The 256GB flash drive that is standard with the entry level model doesn’t work for me. I don’t want to mix my “permanent” and “temporary” files on a RAID array. I want my Dropbox folder and my photo libraries to exist on my “permanent” drive so that Time Machine and Dropbox are constantly backing up any changes. I also want to be able to restore from Time Machine if needed with one click. If you plan the workflow for the worst case scenario then there will be no need to juggle folders during a restore because the destination drive is too small.

My internal hard drive on my 2008 Mac Pro is 2TB. Its $1000 for 1TB on the new Mac Pro. Can you see my conundrum? I’m going to have to buy an external array that will allow me to mount some drives individually while RAIDing the others. I haven’t had time to do any research on this yet. After I figure out this issue then I will be able to buy my new Mac Pro!!!

Lou Borella


Related Posts