Decklink Duo

[ajmaudio]

I built two identical machines from scratch--all parts brand new--in April. Each machine is equipped with two Phoenix cards (4 inputs each box), SSD for content, and Radeon 4870 output card.

The macs as a whole have had zero problems. Playback has been stable, despite operating in very hot and humid conditions.

The Phoenix cards, on the other hand, have been a letdown. Capture has seized/frozen on numerous occasions, often requiring a reboot to reacquire a signal; at one recent show we shut a machine down for an hour before the our set to give it time to cool down but one of the cards never came back, and I had to set Catalyst to copy one of the other inputs.

At first we thought this was a heat issue across the entire system but one card in particular seems to seize a lot more than the others. This is troubling to me because the two servers on this show are built identically. It seems to me that if it was a heat issue, I would be having similar failures on both machines. As it is, I suspect a defective card.

Considering the price of the cards, and being only 3-4 months old, this is a disappointment. To their credit, AS seem keen to help and are sending a new card out this week. But I remain nervous, especially after reading Oli's posts...

I doubt its heat related... I have been running mine outside in 100+ degree weather all summer with no issues... assuming all your cooling parts are working I'd look elsewhere.... unless perhaps you are leaving them out in direct sun in a black case etc.. mine have been in that situation a couple times and made out ok... I was suprised... my cases are blue though lol

[samsc]

I figured out yesterday - Oli had blown up 11 cards ( out of 5 systems with 2 in each ) - thats a 110% failure rate-
No wonder he is upset- i have every sympathy for him-

But this is completely unprecedented - and means something else must be wrong with the touring system-
it just cannot be the cards with this failure rate - it has to be something electrical or physical or something in the rest of the system-

sdi cards - because they connect to outside systems which can be on a separate power supply -are uniquely vulnerable to power and electrical spiking-
sdi is not isolated by default- and as we all know there are many many power things going on in shows - surges - noise - spikes- ground differences - which can cause problems.
And also lighting video and sound power usually run off different phases - or different generators...

--

AJM- I dont think its heat either - otherwise the failure rate would be high for everyone - and it isnt.
Touring in the USA is a very hot environment - and there are a lot of tours out there.

--

We work in an exceptionally tough environment for electronics.
Constant shipping constant unplugging-
And equipment does fail- every component-
Macs graphics cards etc etc
I dont think AS has any higher failure rate in general than any other part of the electronics
The failure modes are all accelerated due to usage patterns etc.

my own macs are vulnerable to failure because im constantly putting different sets of pci cards inside the machines.

[OliLights]

Thanks for all the replies once again, seems like we're getting somewhere.
Before i start i need to clear up a couple of things with the remarks that Simon and Richard have made.

According to Erlwin - they are not!...

Erlwin has been away from the tour for the last month. In the US we had to run on seperate supplies because XL's UK PPU was changed out to a US spec PPU and had to operated on 110V, We used isolation on these shows. For the last months shows we have ensured we run the cats on the same supply as the video PPU. It's because of the Neg 240V UPS's in their racks that we have had to use a differing supply, i will seek to find a means around this upon our return in September.

Have you measured the voltages between earth and the SDI ground?

This happens before we connect up our SDi, when we measured it in Seoul, when we were up shit creek, i personally measured it myself and yes there was nothing floating between our grounds, there was however between 1.2 and 1.7 across Neutral Earth. The Decklink HD3 Placed in the same working conditions was fine. I see this not to have been our issue though, read on.

We have been awaiting more detailed information from the tour to ascertain what is causing this issuse. Until very recently, information has been lacking



I refer to my previous comment:
Hundreds of these cards are operating sucessfully in servers around the world - it seems strange that the cards on your show are having a unique problem - does this not point to a problem with your setup?

We have been suffering since early on in this campaign, however, we have never lost a show with these cards. We have done close to 130 shows since October last year without show failure. Having a spare server helped us to keep this so, invariably our faults would arise upon first power up or after a period of dormancy in the afternoon say after soundcheck and before doors open.
I have never said that the Active Silicone support is lacking and am fully aware that you and Neg are trying to get to the bottom of this.
All i have heard is that AS have looked at the boards and seen connection issues between the daughter and mother boards. From my memory of the cards there are pretty hefty pillars that adjoin these two boards and wondering how that connection could failing. They seem very well built.

HD-SDI Isolating transpformers cost £165 each
4way HD-SDI 3G opto isolators are more expensive at £2498 a pair...
(if anyone knows of any cheaper - do get in touch!!!)

I am going to see to it that someone invests in these, if only to irradiate the potential for a bit of PD! Seems like we should get a 4 way unit and wire it before our DA.

When your solution to fixing the problem includes 'wiggling' the card in the PCI express slot - and 'wiggling' the bnc cables connected to it - im not surprised it failed!

I never said that I had been 'wiggling the PCI Card inside the slots' as you pointed out. I have never even turned on a MacPro without all of it's covers on and it being back in the sleeve, the Neg servers are pretty hard to wire up without it been in their sleeves. At least with a Decklink card it's counting frames in catalyst from the second Catalyst comes online, with the AS Cards your relying on a signal being present into the card. When you have a DA delivering 2 machines with 4 SDi Sources and lets say inputs 3 and 4 are not counting on the second Catalyst, but they are on the first then what does that tell you? If then you shut down both servers and switch their SDi inputs, the same fault arrises it's the card correct! In Korea, we had NO connections to our severs.... NO CABLE CONNECTION PRESENT to the Phoenix Card.
Just the vibration coming from sound on stage deck itself was enough to upset the card inside the machine. From functioning to non-functioning in the space of 30 seconds. With a quiet stage and a reboot, recognised again but this time with NO driver installed. Re-installation of the driver and another restart and we'll try again, yes, we cried "it works... let's leave it working with SDi input and go for a cuppa". Upon our return the Sound had started again and both inputs frozen with no frames being counted. We knew at this point it was the cards.
It made me happy to hear that AS have at least found connection issues between there main PCI card and the SDi daughter board that is mounted to it.
Richard, can your driver be made to could frames or at least give card status without the need for an active SDi Signal. I guess it works with Decklink because it goes through at QT encoder right?

Have you actually tried either of these cards in your setup to qualify this statement?

Yes, Actually, My server that delivers my FOH Preview has a Blackmagic and Kona card installed.

Having this level of support is very important to me, i know what Active Silicone are doing to try and put these issues to bed and know that we as users need to do is keep tabs on the issues we have in order to be serviced with support in the way we would like.
I have been using Catalyst now for nearly ten years and still today i keep finding may interesting ways of flaunting it in my designs. Having 4 SDi inputs is a luxury i know and i know that richard has put so much ground work in with Active Silicone to in order to make the product work so well with Catalyst, like i pointed out earlier, i on this Muse tour alone have executed nearly 130 shows with 4 Servers running 2 cards, and have got through every one of them with out major issues.

i started off at the bottom at vari*lite ( as did a lot of people still around ) - fixing broken electronics many years ago - and it broke a lot - probably something like 10% failure rate per show. ( possibly more )

On every show you had to test every piece of equipment together as a system before it went out.

its a tough business - such a harsh environment for electronics - very electrically and physically unstable-
with users are under enormous pressure when things break- equipment on and off trucks every day -

and harsh on software developers too- who arent used to things being in continuous use for days at a time - with one unrepeatable and unmeasurable glitch a day -

its a bit of a miracle it works well enough most of the time-


in most cases the physical failures were electrical - cables - power supplies - connectors - mains - static issues -
electronics blows up really easily if you touch it when there is static electricity around -
and vari*lite built their flight cases with static generating material -
I remember the entire first batch of VL4's blew up when they arrived in london because they hadnt been tested on real 240v power.

then there are the design mistakes and the end user errors - dropping things - vari*lite consoles used to blow up when dropped- the relay which switched between 110 and 240 shorted in a failure mode.
Lamp power supplies had an unsafe failure mode which blew up all the transistors.

equipment that gets connected and disconnected all the time that is constantly shipped around - it breaks.
even the best designed stuff. even the worst designed stuff.

---

in my experience - with equipment failure like this - you always look to the electrical and cable conditions first of all.
and you check and double check what people say - because in show confusion accurate information is not easy to come by.

Thanks Richard, i am aware of your past, and appreciate your look back at the times where kit was unreliable, unstable and susceptible. I began my life in this industry as a service technician fixing dead moving lights, desks and scrollers, i remember it in my nightmares!

In the world of lighting things have got much better, kit still breaks of course but now common faults are mechanical, along with the occasional electrical failings. I have had 100% of my lighting kit hired by me on Muse for 99% of the shows i have used it on, a lot less can be said for the reliability of the LED screen technology i'm using on the other hand! Your right, most of the time faults are traced back to bad cabling and power / data supplies.

I am happy to say that Catalyst has never let me down and i really love using it, i mean that and you won't hear me saying that about many products.
You have developed aspects of your software for me and my ideas and made it more usable with every build.

I still have a MacPro 3Ghz Intel core 2duo that is 3 years old plus, it's been round the world 4 times on Planes, Boats, and Trucks on countless tours and still goes strong to this day. If that is not a testiment to reliability then i don't know what is.

I will ensure from now on that every fault gets logged and sent to Simon and Active Silicone in a means to try and resolve these issues.

Thanks again for everyones time and getting to the bottom of this matter and helping to get it sorted.
A word of advise to all other Video Input users ---- Get some Opto Isolation in order to take good preventative measures and streamline the cause of your faults.

Simon of you need to contact me then Neg has my details.
Thanks a lot
Oli

[samsc]

Oli

my sympathies are always with you - and i hope we can figure it out.

when i found out this week how many failures you have had - i have been checking and rechecking with everyone involved

i will do my best

thanks
rr

[SourceChild]

I had a similar issue with As LFG4 Cards.

Two years ago sent 2 servers to South America with a US Flypack. Those servers had LFG4 cards and I saw the same problems. Specifically the driver working then not. On two machines we had 6 cards for 12 inputs on each machine. On every load-in, I had to fix at least one system. By the 5th show we got a power conditioner for the whole video system and ran fibre to the projectors. When we did the problems dropped off but I still had to replace 2 cards.

[SourceChild]

Last year I had 4 machines out with two Phoenix Cards each for 4 inputs per machine. We were fine in the US but when we sent the system out of the US, we had similar problems. I lost 2 card in 10 shows. On 4 occasions I had to uninstall/reinstall the Phoenix Drivers. On one occasion I saw glitchy capture while we were live. The next day I mounted an 8ch HD-SDI Opto ($£ etc) in the rack and didn't see the problems anymore.

I will note that on 60 shows, the servers were in video village.
On 40 shows they were spread back stage and on stage. It was on stage where (sound giggling) where I noticed more issues.

Little notes I've made with Phoenix experiences:

I have 14 cards. Some are single some are double input.

2007 Machines with x1900 and Phoenix cards: 4 Machines and have never seen a problem averaging 55-60 shows per machine.
2008 Machines with 8800s and Phoenix cards: 8 Machines, replaced 3 Phoenix Cards in 2 years. Seen issues similar to yours on a dozen different occasions. Specifically have seen issues on touring shows and not the one-off/corporate. Average of 140-160 shows per machine.
2009 Machines with 4870 and Phoenix cards: 4 Machines and swopped cards in 2 machines once. Seen issues similar but not as often. Average of 25-30 shows per machine.

I will note that more often on the 2007 and 2008 machines I have seen Buffer errors more often than anything else.

[Mr_P]

L The next day I mounted an 8ch HD-SDI Opto ($£ etc) in the rack and didn't see the problems anymore.

Which opto isolators did you use?


Re Sound Vibration - I have seen the same problem, but not only with AS cards... A 'dual-shock-mounted' rack was next to another hire company 'foamed' rack - 1st beat of song (albeit Prodigy!) the 'foamed' one failed and the 'shock-mounted' one kept working...


Re Cooling: - Some LD's have specified 'pop-up' tents with forced air-conditioning to house the video equipment. Its a fairly simple solution...

S

[SourceChild]

Which opto isolators did you use?

I don't remember. I think it was an Everts chassis with HD-SDI DA cards. I'll look into it.

[SourceChild]

A 'dual-shock-mounted' rack was next to another hire company 'foamed' rack - 1st beat of song (albeit Prodigy!) the 'foamed' one failed and the 'shock-mounted' one kept working...

Simon, I think I've told you several time how much I love your cases.


Alas, if I only had justification to have them shipped across the Ocean. I just have to settle for what I have here.