If you are using latest version of mencoder, use following convert command

$cmd_mencoder = "$config[mencoder] '$video_src' -o $video_flv -of lavf -oac mp3lame -lameopts abr:br=56 -ovc lavc -lavcopts vcodec=flv:vbitrate=500:mbd=2:mv0:trell:v4mv:cbp:l ast_pred=3 -srate 22050 -ofps 24 -vf harddup";

If that do not convert video into FLV, try running the conver command in SSH prompt.

If it can't convert video into FLV format, your installation have some problem.