Trellis quantization is an algorithm that can improve
data compression in
DCT-based encoding methods. It is used to optimize residual
DCT coefficients after motion estimation in
lossy video compression encoders such as
Xvid and
x264. Trellis quantization reduces the size of some DCT coefficients while recovering others to take their place. This process can increase quality because coefficients chosen by Trellis have the lowest
rate-distortion ratio. Trellis quantization effectively finds the optimal quantization for each block to maximize the
PSNR relative to
bitrate. It has varying effectiveness depending on the input data and compression method.