The convert command is for old version of mencoder, if you are using latest version of mencoder/mplayer, you need to replace convert command in include/settings/video_conversion.php

http://docs.buyscripts.in/wiki/Mencoder_options