There are an increasing number of binaural systems embedded with head-related transfer functions (HRTFs), so listeners can experience virtual environments via conventional stereo loudspeakers or head- phones. As HRTFs vary from person to person, it is difficult to select appropriated HRTFs from already existing databases for users. Once the HRTFs in a binaural audio device hardly match the real ones of the users, poor localization happens especially on the cone of confusion. The most accurate way to obtain personalized HRTFs might be doing practical measurements. It is, however, expensive and time consuming. Modifying non-individualized HRTFs may be an effort-saving way, though the modifications are always accompanied by undesired audio distortion. This paper proposes a flexible HRTF adjustment system for users to define their own HRTFs. Also, the system can keep sounds from suffering intolerable distortion based on an objective measurement tool for evaluating the quality of processed audio.