NOTE: Updated to correct the cards brought for the install.
As I reported a few weeks ago, Small Tree replaced our original ethernet SAN we’ve been running since December 2008 with a new system including an all new 48TB Granite Stor RAID II. I also reported that we discovered some things about the Macs that Steve and Chris wanted to go back and test even further.
During the original install, the new Edge Core Switch turned out to have an issue which caused one of the 10Gig ports to fail. But we all expected that the system would work well for us while a solution was found to that issue. Unfortunately, though we tested the system for a full day with all the systems running and doing some editing, after the first full week, we knew we had to improve the performance of the system in a hurry. There just seemed to be a fine tolerance to what could be happening at any given time to ensure that all the systems worked properly, particularly the ability to master shows without the edit to tape aborting for dropped frames. It really came to a head one afternoon when we were trying to get two network shows out at the same time and of course, dropped frames turned everything into a race against the clock to make overnight shipping. We made it, but it was way too close for comfort.
We remained in almost daily contact with Steve, Chris and the rest of the Small Tree engineering team and they repeatedly would log into our various computer systems, make some tweaks, take some notes and keep working at it. Then last week Steve, Chris and Corky Seeber made a return visit to our facility, but this time they brought a brand new Small Tree 10gig switch along with a quad port 10 Gb card (installed in the server to take the best advantage of the new Small Tree 10 GB switch), and 4 Single port Ethernet cards. Small Tree had noticed during testing in their offices that they were able to get better performance from the Small Tree 1 GbE cards than the internal Ethernet ports of the Apple systems when using the Small Tree 10 Gb Switch.
Basically what Small Tree has been able to do in the past is make high speed, off-the-shelf network switches work for high speed video editing. But from what I understand we’ve hit the breaking point where if ethernet shared storage is going to continue to evolve and move forward, we need switches designed more for that task than for regular network traffic. Enter the new Small Tree 24 Port 10Gigabit Ethernet switch.
This thing was built with complete 10GigE infrastructure within to ensure that the maximum data throughput is achieved at all times. Designed from the ground up by Small Tree, this unit is something that should be able to withstand the constant beating that a shop like ours throws at it. So that was step one, install the new switch to give the entire system a speed boost right off the bat.
The next step was to go through each individual computer in our facility and individually tune them further. The first time they were here they did the first round, but since discovered some more “secret sauce” to make things work more smoothly. They go into the Terminal of each machine and make some internal tweaks to the setup of each system. There’s no “one setting fits all” as I found out. Each systems has to be configured based on all the particulars of how the machine is set up including the software it runs and the third party hardware installed. In the end, I believe Small Tree only installed one of the new 10GigE cards into the machines because quite frankly, the others didn’t need them once they were tuned correctly.
In short order we had the SAN up and running and every edit suite playing down video timelines along with all of our iMacs. So that’s 5 Mac Pro workstations and 7 iMacs all playing 720p or 1080i ProRes video. The Mac Pros were were all playing FCP 7 timelines in a loop and the iMacs were all playing 20 to 30 minute clips in a loop. The iMacs don’t have editing software on them, they’re used by Producers to review footage as necessary.
Once again, the system in Edit 1 was the most vexing because it’s one of the fastest systems in the entire facility and the one we use to cut the feature documentaries, yet it would drop frames playing the same timeline that the slowest system in the shop could play with no problems. Normally you would say “Add more Ram!” “Add a faster GigE Card!” and those might fix the problem temporarily. But there was something fundamental with the way this machine was configured that needed to be addressed. This was a super fast 8 core machine being outperformed by a much slower four core machine. I would guess the pair of them spent about 6 hours just on that one Mac Pro but they finally hit on a combination of settings that made a major change in the behavior of the machine. We did not touch the RAM, we did not change the Ethernet Card (A Small Tree Peg1 card that’s been in there all along), Steve and Chris simply kept going into the Terminal and making adjustments to the way the Mac Pro operates. I honestly don’t know all the particulars but by the end of the day, the Edit 1 system was behaving better than it has in a long time.
So when we started this whole process of moving from the older SAN configuration to the new SAN configuration we could lay off a half hour show to tape, but we would have to carefully manage what all the clients on the SAN were doing. Even then, we could get those aborts due to dropped frames.
Now, we can literally lay off two shows simultaneously and not pay any attention to what any of the other clients are doing. We’ve never been able to do that. In fact we did it 4 times in a row as a test. Two Mac Pros laying off 30 minute 720p HD shows being converted to 1080i via the AJA Kona 3, Three other Mac Pros playing 5 to 30 minute timelines in a loop or editing / scrubbing video (I was jumping from edit suite to edit suite to take over) and the 7 iMacs all playing long video clips in a loop. In fact I even laid off 3 shows simultaneously in one test. We’ve never been able to do any of this in the past. Oh we could edit in all the suites at the same time no problem, but mastering two shows at the same time to tape, that’s never happened, but we always knew that and planned for it.
Then on Monday the editors came in and without any prompting, one of them said, “Everything is snappier today.” All three of my editors noticed immediate improved performance from the system. More realtime playback, dropped frames non-existant and absolutely no concern for mastering off shows anymore, even two at a time. After a full week of hammering on the system, I’m glad to say that the system is proving itself on a day to day basis.
And it’s not just that we have some faster products now with the new 48TB RAID and the 10GigE switch from Small Tree. It’s the technical experience by Steve, Chris and all the engineers at Small Tree to completely understand the inner workings of all the machines that are connected to the system. Not accepting that we can “just throw more RAM or another card” at the problem and try to make it go away. It’s getting to the heart of the problem, understanding it and then taking the correct course of action to solve it. Sometimes that means adding more hardware / RAM and other times it simply means tweaking the inner workings of the system.
It’s very exciting to see what started out really as a cheaper alternative to a fibre channel SAN now evolving into a much more robust and fast system. Oh and don’t think this is a Mac based solution, this concept can run on Windows as well, these guys have expertise in both platforms and of course Linux too. That’s VERY important to me right now as we ponder the future course of our company and the NLE solution(s) we go with. We have to be prepared for the possibility that a Windows workstations (or two, or three) could start appearing in our shop. Thankfully, the guys at Small Tree will be ready to take our SAN in whatever direction we need to go.
Ok, about that * in the title – Technically what we are using is a NAS, not a SAN. But when Bob Zelin and I first started talking about it publicly we both referred to this configuration as a SAN because that’s what we call shared storage in video production. Steve Modica got tired of correcting us and just went along with our (incorrect) terminology. So if you’re fussy about the correct terminology, what we are using to edit video is a NAS. Happy now? Good!
Hi Walter,
Thanks for the interesting and informative post. I am curious if you would go into more geeky detail about what settings are being tweeked (packet/frame size?) on each machine and what sort of factors are in play that lead to the conclusions on settings. I know this stuff gets really geeky but I for one am not ashamed to “get my geek on” from time to time.
Nope, not going to do that for two reasons:
1 – That’s what you pay Small Tree for. Their knowledge and experience in setting all of this up.
2 – That information will vary depending on your precise set up. The RAID, the SAN Controller, the Clients, how many clients, how the clients are configured, etc…. There is no “one size fits all” when setting these things up, Small Tree has to adjust and set up each Client individually to ensure it works correctly.
So even if I told you how specifically they set all of this up, it could utterly fail in your situation. While the basics of this configuration are the same no matter where it’s installed, the fine tuning in the configuration will be unique to each installation.
Small Tree’s technical spec aren’t that big of a secret, they post it in their support page on their website for anyone to see.
http://www.small-tree.com/kb_results.asp
I never said they hide their specs, I just said they know how to tweak each machine.
I will also say that after installing their switch, the entire system improved speeds dramatically as I spoke of in the original article. There is a huge difference in performance between this switch and the original EdgeCore switch that was originally installed by Maxx Digital. In fact there’s a huge difference period between the performance of this entire system vs. the original Final Share system we used to run. But then that’s not surprising since Small Tree basically created and perfected the concept of shared storage via Ethernet.
Hey Walter,
Great article,
We are pretty much in the same boat with shared storage and have really liked out original small tree system but have to upgrade to 10GbE with all the work we are doing.
I am a little frustrated that one of small tree’s advantages was how inexpensive solutions could be.
My quick math:
48TB ST RAID – $12,000-15,000
10GbE Switch – $15,000
10GbE Cards, Transceivers, connectors – $2,000+
New Server w/32GB RAM – $5,000+
Equals – $34,000, Plus the money to fly them out multiple times for setups, small post houses just can’t afford this!
We all know their ST RAID is just a istoragepro chassis with enterprise drives connected to an ATTO R680 Controller card. There are many other 10GbE switches from maxxdigital and Intel for a 1/3 of the price of their 24 port switch.
While I think it’s great that Small Tree can tweak their systems, I can’t afford to fly them out and really want the system to just work once it’s installed and not have to tweak it.
I would like to think of spending less than 20K on this type of system, if I was going to spend 30K+ I’ll just go to fibre again.
Just my two cents, I’m glad you are running great and appreciate the post.
-Jeff
Since you’re doing the math, then go Fibre and price that out using the exact same specs. 48TB to as many seats as you need to drive. In my case the price difference was $64,000 to $22,000 when we did our original install with just 16TB of Shared Storage. Not sure what that would be today with 48TB going to 12 seats. It would be a LOT more than $30k.
As for Small Tree tweaking your system, they can do that all remotely. They just wanted to come to my shop and do this install personally because it was their brand new switch. Plus they had never been to the shop before and they wanted to come in. But they have spent many hours tweaking my system remotely long before they ever came to the shop and they continue to tweak to this day. They’re never satisfied and always trying new things.
So in your case, if you wanted to go their route, you would simply purchase the gear, they would pre-configure everything to a point at their location, and then log in to your systems remotely to do all the fine tuning.