A substantial quantity of research on muffler design has been restricted to a low frequency range using the plane wave theory. Based on this theory, which is a one-dimensional wave, no higher order wave has been considered. This has resulted in underestimating acoustical performances at higher frequencies when doing muffler analysis via the plane wave model. To overcome the above drawbacks, researchers have assessed a three-dimensional wave propagating for a simple expansion chamber muffler. Therefore, the acoustic effect of a higher order wave (a high frequency wave) is considered here. Unfortunately, there has been scant research on expansion chamber mufflers equipped with baffle plates that enhance noise elimination using a higher-order-mode analysis. Also, space-constrained conditions of industrial muffler designs have never been properly addressed. So, in order to improve the acoustical performance of an expansion chamber muffler within a constrained space, the optimization of an expansion chamber muffler hybridized with multiple baffle plates will be assessed. In this paper, the acoustical model of the expansion chamber muffler will be established by assuming that it is a rigid rectangular tube driven by a piston along the tube wall. Using an eigenfunction (higher-order-mode analysis), a four-pole system matrix for evaluating acoustic performance (STL) is derived. To improve the acoustic performance of the expansion chamber muffler, three kinds of expansion chamber mufflers (KA-KC) with different acoustic mechanisms are introduced and optimized for a targeted tone using a genetic algorithm (GA). Before the optimization process is performed, the higher-order-mode mathematical models of three expansion chamber mufflers (A-C) with various allocations of inlets/outlets and various chambers are also confirmed for accuracy. Results reveal that the STL of the expansion chamber mufflers at the targeted tone has been largely improved and the acoustic performance of a reverse expansion chamber muffler is more efficient than that of a straight expansion chamber muffler. Moreover, the STL of the expansion chamber mufflers will increase as the number of the chambers that separate with baffles increases.