You need to edit file include/settings/video_conversion.php

The command used is

Code:
$cmd_mp4 = "$config[ffmpeg] -i '$video_src' -crf 22.0 -c:v libx264 -c:a libfaac -ar 48000 -b:a 160k -coder 1 -threads 0 $video_flv";

This may need slight changes depending on how ffmpeg compiled and installed.

-crf 22.0
Is what you need to change. You need to experiment with it and see what works good for you. Best thing you can do is install ffmpeg on your local computer, there is even windows version available for download and try different settings and see which one is good for you. Better quality video take more disk space.

You can read more about crf at

https://trac.ffmpeg.org/wiki/Encode/H.264#crf