I don't have this box checked...but in your situation, if your trying to stream DTS audio, maybe it's required. There's a checkbox for "DTS/FLAC to LPCM" near those same settings.
My PS3 and my server are both wired, and I have no issues streaming. The ps3 only has wireless G, this could definitely be a problem with high bitrate HD content. You could try wiring them to rule out network performance being the issue.
Other than that, I'm not sure. If you say it starts to play the video, then I assume it's not a file format compatability issue. If there was an issue with the format of the audio or video stream, the PS3 would pop up that "unsupported file type" error (or however it's worded).
You could try running MediaInfo,
http://mediainfo.sourceforge.net/en, and copy/paste the results from "View > Text" here, maybe someone will spot what's wrong with the video file itself. But since your saying it starts and then stops, it doesn't sound like a format problem.