Your mencoder installation may not support these formats or some codec is missing on the installation. You can ask in mplayer mailing list to find problem with the mplayer installation.