Edit include/settings/video_conversion.php and try with this code
Code:
$cmd_mencoder = "$config[mencoder] '$video_src' -o $video_flv -of lavf -oac mp3lame -lameopts abr:br=56 -ovc lavc -lavcopts vcodec=flv:vbitrate=500:mbd=2:mv0:trell:v4mv:cbp:last_pred=3 -srate 22050 -lavfopts i_certify_that_my_video_stream_does_not_use_b_frames";
After that error exists, the problem is mencoder is not able to convert the video to FLV format.Contact your webhost to check mencoder installation. Also check http://docs.buyscripts.in/wiki/Mencoder_options