Mostly this is due to faulty mencoder installation and may need reinstallation. Are you using latest version of mencoder ? If you are using SVN version, you can ask in mencoder mailing list.