The command is not working, you are using old version of mencoder.

You need to edit include/settings/video_conversion.php and change mencoder conversion command to the one for older version, then try reuploading the video.