Code:
/usr/bin/mencoder '/home/xxxx/public_html/xxxxx/share/video/bushatbooker.uncut.avi' -o /home/xxxxx/public_html/xxxxxxx/share/flvideo/11938741391727513621.flv -of lavf -oac mp3lame -lameopts abr:br=56\ -ovc lavc -lavcopts vcodec=flv:vbitrate=500:mbd=2:mv0:trell:v4mv:cbp:last_pred=3 -srate 22050 -ofps 24 -vf harddup
If this command works on command line, it should work from php exec also unless some settings on your server block php from executing the command properly.