From the above result, it look like mencoder not able to convert video to FLV format.

What you get when you run the command in terminal ?

If it run on terminal and not in php, it is problem with server cnfiguration.