You may need to get latest version of mencoder/mplayer with all codec installed. Check the debug log file (templates_c/debug.txt) after you upload a video, try running the convert command in SSH prompt and see if that works.