A Digital Place for Everything, and Everything in its Digital Place

How a global video production company keeps track of its ever-growing digital library

by Jason Hester

Jason Hester: Ace Video Hoarder

Confession:  I’m a digital hoarder.  I have terabytes of data in my personal library.  Music, movies, pictures, sound effects, apps, you name it and I’ve probably got it sitting on a hard drive.  In most cases, being a hoarder is frowned upon, but in my world as a video editor I wear it as a badge of honor. 

As any video editor will tell you, clients love to revise old videos. 

Typical Client Request: 

“You remember that company overview video we did four years ago?  We just got a new CEO, so we need to replace all of the former CEO’s sound bites.  Can you guys shoot that and have an updated video ready to go by tomorrow’s close of business?”

In the “old” days (10-15 years ago) that would mean finding your Avid or Final Cut Pro project file and all of the tapes used in the original project and setting up a large batch capture to rebuild the video clip-by-clip in real-time using a tape machine.  Since then, a lot of cameras have gone tapeless, meaning they record to an internal hard drive or removable-media, such as P2 or SD cards.  That’s great for ingesting footage.  No more digitizing tapes in real-time. Instead, you just copy data right from your camera’s removable-media onto your computer and start editing, often much faster than the real-time digitizing of tapes. 

Re-connecting your media to make changes to an old project is much easier, too.  For the most part, you can just highlight your sequence or timeline, point it to a folder of footage and it will rebuild it for you almost instantaneously. 

One downside to this is you no longer have a collection of tapes holding all of your raw footage to store in a library after the project has been completed.  The footage only exists as data on a hard drive, and hard drives fill up fast.  Here at Firstline, we have a 16 terabyte drive array dedicated to video (that’s 16,000 gigabytes) and it still fills up way faster than I would have ever imagined.  Keeping it backed up and properly archived is a must.

Carbon Copy Cloner:

First, let’s talk a little bit about maintaining a backup.  Attached to our 16TB array, we have an additional 12TB array attached via Firewire 800.  We use an excellent app called Carbon Copy Cloner (http://www.bombich.com/) to create an exact clone of our video drive that updates every night.

 Here’s the way Carbon Copy Cloner works for us: we’ve created a scheduled task that runs every night at 10 pm.  The very first backup took almost 48 hours to copy all of our data over to the new backup drive.  After that, backups have been much faster because Carbon Copy Cloner compares the two drives every night and only backs up files that have changed or been added since the last scheduled backup.  So, we always have a cloned backup of our video drive in the case of a catastrophic failure on our main array.

Archives: 

You may be wondering why we’ve only got a 12TB drive to back up a 16TB drive. The answer to that is simple: just because you’ve got 16TB’s available, you should never completely fill it up.  Drives run better when they’ve got space available and by having our backup drive only be 12TB’s, this helps us to police ourselves a bit and make sure we’re properly archiving our data after projects are complete.

When it comes to archiving, we use a hard drive docking station that connects to our individual computers via USB-2 or E-SATA.  These docks allow us to buy raw, internal hard drives, which are considerably cheaper than external hard drives, and just swap them in and out whenever we need to archive footage, or retrieve it after it’s been archived.

“That royalty free track I used eight years ago in a TV promo will be perfect for this new web video.  Now where did I put it?”

Earlier in this blog, I likened myself to a hoarder, but that’s not 100% true.  Most hoarders have no idea what all they have.  I, on the other hand, know exactly what I’ve got archived and where it lives.  At the time of this writing, we currently have 56 hard drives in our Firstline archive.  That’s a lot of drives to have to manually pull out, attach to the docking station and search when I’m looking for a specific file or batch of footage. 

NeoFinder: 

NeoFinder

Luckily, I found this pretty cool app a few years back called NeoFinder (http://www.cdfinder.de), which acts a cataloging system for hard drives and optical discs.  Every time I archive something to a hard drive, I open NeoFinder and tell it to “catalog” the hard drive if it’s new, or “update” if I’ve previously archived files to it, and the process is almost instantaneous.  Then, when I’m looking to retrieve something from my archives, I just open NeoFinder, do a simple search and it tells me exactly which hard drive I need to pull off the shelf.

Like most things, there are multiple ways of handling backups and archiving, and what I do may not be perfect for you.  In fact, it’s not 100% perfect for me, yet.  But I’m working on it. 

The next step in my archiving plan is to make backups of each of my archive drives and store them off-site, in case of a seriously catastrophic event at our studio.  With 56 archive drives (and counting) that’s a pretty daunting task, but one I feel is vital.  Hard drives die and the time and cost associated with making these backups of backups will be well justified the first time I go to spin up an archive and it fails. 

So, how are you handling backups and archiving?

Jason Hester is a Video Editor with Firstline Creative & Media who has spliced together well over 1,000 creative video projects for television networks, Fortune 500 companies and major non-profits across the globe. Jason lives (and hoards digital data) near Atlanta with his wife and son.