Matched Filter Mismatch Losses in MPSK and MQAM Using Semi-Analytic BEP Modeling

The focus of this paper is the bit error probability (BEP) performance degradation when the transmit and receive pulse shaping filters are mismatched. The modulation schemes considered are MPSK and MQAM. In the additive white Gaussian noise (AWGN) channel both spectral efficiency and noise mitigation is commonly achieved by using square-root raised cosine (SRC) pulse shaping at both the transmitter and receiver. The novelty of this paper primarily lies in the use semi-analytic BEP simulation for conditional error probability calculations, with transmit and receive filter mismatch, the optional inclusion of a small FIR equalizer. For lower order MPSK and MQAM, i.e., 8PSK and 16QAM Eb/N0 power degradation at BEP = 10−6 is 0.1 dB when the excess bandwidth mismatch tx/rx = 0.25/0.35 or 0.35/0.25, but quickly grows as the modulation order increases and/or the mismatch increases.


Introduction
In the early days of satellite and space communications, digital modulation schemes focused on constant envelope waveforms, in particular phase-shift keying (PSK), with rectangular pulse shaping [Lindsey]. The need for spectral efficiency is ever present in modern communication systems [Ziemer], [Proakis], and [Goldsmith]. The use of pulse shaping makes spectral efficiency possible, at the expense of non-constant envelope waveforms [Ziemer]. Today m-ary PSK (MPSK) and high density mary quadrature amplitude modulation (MQAM), both with pulse shaping, are found in satellite communications as well as terrestrial communications, e.g., WiFi, cable modems, and 4th generation cellular via long term evolution (LTE). The term m-ary refers to the fact that bandwidth efficient signaling is accomplished using an M symbol alphabet of complex signal amplitudes, c k = a k + jb k , to encode the transmission of a serial bit stream. In an m-ary digital modulation scheme we send log 2 M bits per symbol. The objective being to transmit more bits/s/Hz of occupied spectral bandwidth.
In certain applications the precise pulse shape used by the transmitter is not known by the receiver. The use of an equalizer is always an option in this case, but it adds complexity and for burst mode systems, e.g., TDMA, the convergence time of an adaptive equalizer is another issue to deal with.
The focus of this paper is the bit error probability (BEP) performance degradation when the transmit and receive pulse shaping filters are mismatched. The modulation schemes considered are MPSK and MQAM. In the additive white Gaussian noise (AWGN) channel both spectral efficiency and noise mitigation is commonly achieved by using square-root raised cosine (SRC) pulse shaping at both the transmitter and receiver. The system block diagram is shown in Figure 1. The parameters α tx and α rx control the excess bandwidth factors of the transmit and received filters respectively. Notice that this block diagram also shows a symbol-spaced equalizer to allow for the exploration of potential performance improvement, subject to how much complexity can be afforded, and the need for rapid adaptation in the case of burst mode transmission. We take advantage of semi-analytic simulation techniques described in [Tranter] to allow fast and efficient performance evaluation. All of the simulation software is written in open-source Python using classes to encapsulate the computations, waveform parameters, and calculation results. The mathematical foundation is statistical decision theory, which in the machine learning would be multiclass classification with apriori decision boundaries.  Other authors such as, [Harris] and [Xing], have made mention of matched filter mismatch, but not in the same context as we consider in this paper. Harris is primarily driven by sidelobe reduction from the transmitter perspective, while Xing wishes to avoid the added complexity of an equalizer by using a specially designed receiving filter. Here we are concerned with the situation where the receiver does not always know the exact design of the transmit pulse shape filter, in particular the excess bandwidth factor.
The remainder of this paper is organized as follows. We first provide a brief tutorial on digital communications at the waveform level. Next we consider residual errors at the matched filter output when using a simple truncated square-root raised cosine (SRC) finite impulse response (FIR). In particular we consider filter lengths of ±L symbols in duration. We then briefly explain how a symbol-spaced adaptive equalizer can be inserted at the output of the matched filter to compensate for pulse shape mismatch. We then move on to briefly review the concept of semi-analytic (SA)simulation and the develop conditional error probability expressions for MPSK and MQAM. Finally, we move into performance results.

Characterizing Digital Communications at the Waveform Level
To provide more context for the theoretical development of the semi-analytic simulation technique used in this paper and prepare for the system performance characterization of the results section, we now consider three common digital communication characterization techniques: IQ Diagrams, eye diagrams, and bit error probability (BEP) versus received signal energy-to-noise power spectral density (E b /N 0 ) curves.

IQ Diagrams
An IQ diagram is a representation of a signal modulated by a digital modulation scheme such as MQAM or MPSK. It displays the signal as a two-dimensional xy-plane scatter diagram in the complex plane at symbol sampling instants. The angle of a point, measured counterclockwise from the horizontal axis, represents the phase shift of the carrier wave from a reference phase. The distance of a point from the origin represents a measure of the amplitude or power of the signal. The number of IQ points in a diagram gives the size of the alphabet of symbols that can be transmitted by each sample, and so determines the number of bits transmitted per sample. For the purposed of this paper it will be a power of 2. A diagram with four points, for example, represents a modulation scheme that can separately encode all 4 combinations of two bits: 00, 01, 10, and 11 and so can transmit two bits per sample. Figure 2 shows an 8-PSK IQ Diagram.

Eye Diagrams
An eye diagram is a tool for the evaluation of the combined effects of channel noise and inter-symbol interference (ISI) on the performance of a channel. Several system performance measures can be derived by analyzing this display. If the signals are poorly synchronized with the system clock, or filtered improperly, or too noisy, this can be observed from the eye diagram. An open eye pattern corresponds to minimal signal distortion (clear diamond shape in the left plot). Distortion of the signal waveform due to ISI and noise appears as closure of the eye pattern (note partial closure on the right plot). The tight waveform bundles at the maximum opening correspond to tight scatter points of the IQ diagram, i.e., similar to the opposing pair of points on the real axis of Figure 2. Since the waveform is complex there is an eye diagram for both the real part and the imaginary part of the signal. For the purposes of this paper we will be looking at the closure of the eye pattern as the mismatch of the filters increases, similar to moving from the left to right side of Figure 3.

Eye Diagrams
An Eye Diagram is a tool for the evaluation of the combined effects of channel noise and intersymbol interference (ISI) on the performance of a channel. Several system performance measures can be derived by analyzing the display. If the signals are too long, too short, poorly synchronized with the system clock, too high, too low, too noisy, or too slow to change, or have too much undershoot or overshoot, this can be observed from the eye diagram. An open eye pattern corresponds to minimal signal distortion. Distortion of the signal waveform due to ISI and noise appears as closure of the eye pattern. For the purposes of this paper we will be looking at the closure of the eye pattern as the mismatch of the filters increases. The Left Eye Pattern of Figure 8 is an example of a good sampling instant while the right is degraded.

Bit Error Probability (BEP) Curves
In digital transmission, the number of bit errors is the number of received bits over a communication channel that have been altered due to noise, interference, distortion or bit synchronization errors. The bit error probability (BEP) is the number of bit errors per unit time.
The bit error ratio is the number of bit errors divided by the total number of transferred bits during a studied time interval. The BEP curves of a system are usually plotted and describe the performance of a digital communication system. These BEP curves are often compared to theory curves to measure how the channel affected the overall performance of the demodulator. Figure   9 shows the theory curves for BPSK/QPSK, 8PSK, and 16-PSK

Bit Error Probability (BEP) Curves
In digital transmission, the number of bit errors is the number of received bits over a communication channel that have been altered due to noise, interference, distortion (improper filtering), carrier phase tracking errors, and bit synchronization errors. The bit error probability (BEP), P e , in a practical setting is the number of bit errors divided by the total number of transferred bits during a studied time interval. The BEP curves are plotted as log 10 P e versus the received energy per bit-to-noise power spectral density ratio in dB, i.e., 10 log 10 (E b /N 0 ). The shape of the curve is waterfall like with a theoretical BEP curve to the left of curves for real-world systems. A system with impairments in the end-toend transmission path, including the demodulator (think symbol classifier), increase the BEP for a given operating scenario. In a Wireless LAN or cable modem, for example, a low BEP is required to insure reliable information exchange. A large M is used here to indicate a large number of bits per second, per Hz of bandwidth. To see a typical BEP curve jump forward to Figure 10.

Pulse Shaping Filter Considerations
The pulse shape used for this matched filter mismatch study is the discrete-time version of the square-root raised-cosine pulse shape: where T is the symbol period. The name used here is square-root raised cosine (SRC). The transmitted signal bandwidth when using SRC shaping is approximately (1 + α)R s , where R s = R b / log 2 M is the symbol rate and R b is the serial bit rate. Note m-ary signaling and SRC pulse shaping together together serve to increase spectral efficiency in all the applications mentioned in the introductory paragraph.
The upper plot of Figure 4 shows the right half of an SRC pulse shape for α = 0.5 and 0.25. The lower plot shows the result of passing the transmit pulse through a matched and mismatched receiver filter. The point of the SRC-SRC cascade is to provide spectral efficiency and insure that the pulse zero crossing occur at the adjacent symbol periods, i.e. zero ISI. For the mismatched case you can see ISI has crept in.
ISI is non-zero values at integer symbol times 1, 2,… Tx SRC MF SRC Response Impulse Symmetrical SRC pulse decays slower as a approaches 0, offering a more compact frequency spectrum For realizability considerations the discrete-time transmit pulse shaping filter and receiver matched filter are obtained by time shifting and truncating and then sampling by letting t → nT . Residual errors at the matched filter output are present as a result of truncation as noted in both [Harris] and [Xing]. For small M values ±6T is acceptable, but for the higher schemes considered in this paper we found increasing the filter length to ±8T was required to avoid residual errors under matched pulse shape conditions. The residual errors at the zero crossings shown in the bottom half of Figure 4, but now for an ensemble transmit symbols, is shown in Figure 5. Here we see that the errors increase as α decreases.

Fig. 5:
Matched SRC filters at transmit and receiver showing residual error due to FIR filter truncation of the doubly infinite pulse response [Rappaport], for a nominal maximum eye opening of ±1.

Semi-Analytic Bit Error Probability
Semi-analytic BEP (SA-BEP) calculation allows for fast and efficient analysis when a linear channel exists from the AWGN noise injection point to the receiver detector [Tranter]. A block diagram, which applies to the matched filter mismatch scenario of this paper, is shown in Figure 6. The variable z k is the complex baseband detector decision statistic, as the receiver matched filter is sampled at the symbol rate, R s = 1/T , nominally at the maximum eye opening. ISI is present in z k due to pulse shape mismatch and other impairments such as timing error, static phase error, and even phase jitter. This corresponds to an ensemble of conditional Gaussian probabilities. The variance σ 2 w , for each the real/imaginary parts (inphase/quadrature), is calculated using where the variance of the additive white Gaussian noise is denoted N 0 and p r [n] is the matched filter impulse response consisting of N taps . The value of σ w found in the conditional error probability of the following subsections is a function of N 0 , which is set to give the desired average received energy per symbol E s (note the energy per bit E b is E s / log 2 (M)) to noise power spectral density ratio, i.e., E s /N 0 or E b /N 0 . This allows full BEP curves to be generated using just a single ensemble of ISI patterns. The calculation of N 0 , taking into account the fact that the total noise power is split between real/imaginary (or in digital communications theory notation inphase/quadrature) parts is given by To be clear, (E s /N 0 ) dB is the desired receiver operating point.
In the software simulation model we set (E b /N 0 ) dB , convert to (E s /N 0 ) dB , arrive at N 0 for a fixed E s , then finally determine σ w . Note the 2 in the denominator of (3) serves to split the total noise power between the in-phase and quadrature components. The SA-BEP method first calculates the symbol error probability by averaging over the ensemble of conditional Gaussian probabilities where N is the number of symbols simulated to create the ensemble. For the m-ary schemes MPSK and MQAM we further assume that Gray coding (adjacent symbols differ by only one bit) is employed [Ziemer], and the BEP values of interest are small, allowing the bit error probability to be directly obtained from the symbol error probability via BEP = P E,symb log 2 (M) The other impairments noted in (4) refers to the fact that SA-BEP can also be used to model carrier phase error or symbol timing error.
For the SA-BEP analysis model what remains is to find expressions for the conditional error probabilities in (4). A feature in the analysis of both MPSK and MQAM, is that both schemes reside in a two dimensional signal space and we can freely translate and scale signal points to a normalized location to make the error probability equations easier to work with.

M-ary PSK
For MPSK with M > 2 the optimum decision region for symbol detection is a wedge shaped region having interior angle π/M, as shown in the right side of Figure 7.
A simple upper bound, accurate for our purposes, is described in [Ziemer] and [Craig], considers the perpendicular distance between the nominal signal space point following the matched filter and the wedge shaped decision boundary as shown in Figure  7. For unimpaired MPSK (no noise), we consider a normalized MPSK signal point, z k , at angle zero to be the complex value (1, 0). Since z k is actually a complex baseband signal sample, it can be viewed as the point z k = 1 + j0 in the complex plane. The signal point length being one corresponds to setting z k = √ E s = 1, where E s is the symbol energy. The symbol error probability P E,symb is over bounded by the probability of lying above line L a or below line L b , when circularly symmetric Gaussian noise is now added to z k . For the special case of z k = 1 the probabilities of being above and below the lines are equal, hence this upper bound approximation results in where Q(x) is the Gaussian Q function given by Since we have assumed that z k = 1 we use σ w via N 0 to control the operating point, E s /N 0 , and hence also E b /N 0 . The over bound region, shown in light red in Figure 7, is due to double counting the error probability in this region.
With the bound only small differences are noted for the M = 4 case, and then only at very low E b /N 0 values. The bound becomes tighter as M increases and as E b /N 0 increases. We conclude that the bounding expression for P E,symb is adequate for use in semianalytic BEP calculations at P E values below 10 −3 .
When matched filter mismatch is present the complex decision variable z k , obtained by sampling the matched filter output, no longer sits at a normalized value of (1, 0) = 1∠0. The scenario of a perturbed z k is the real intent of Figure 7, where it shows two perpendicular distances, d a and d b , for an arbitrary z k . We now use these distances to form the conditional probability of symbol error, and hence the Gray coded BEP. Using simple geometry to write d a and d b in terms of the angle π/M and z k = |z k |e jθ k we can finally write the conditional symbol error probability as

M-ary Quadrature Amplitude Modulation
For MQAM the noise-free received symbols are scaled and translated to lie nominally at (0, 0) in the complex plane. Here we pattern the development of the SEP expression after [Ziemer]. The decision region for correct symbol detection detection is one of three types: (1) interior square, (2)  Using simplifications similar to the MPSK case, we have the following equations for calculating the conditional SEP for symbol Types 1, 2, and 3. In the semi-analytic simulation software the symbol is known a priori, so in forming the average of (4) we choose the appropriate expression. For type 1 we have: For type 2 we have: Finally for type 3 we have: In all three conditional probability of bit error expressions, (9), (10), and (11), the variable a is defined is defined in terms of the energy per symbol, E s and modulation order M using a = 3E s 2(M − 1) . (12)

Software Tools and Reproducible Science
All of the analysis and simulation software developed for this study is written in Python. It makes use of the scipy-stack and the authors GitHub project scikit-dsp-comm [Wickert1]. Two classes MPSK_sa_BEP and MQAM_sa_BEP do all of the heavy lifting. The code base specifics for this paper can be found on GitHub at [Wickert2]. The contents include Jupyter notebooks and code modules. All of this is open-source and freely available.

Results
In this section we consider the impact of filter mismatch in MPSK and MQAM. Equalization is not included in these first two studies.
Next we consider how a short length equalizer can be employed to mitigate the mismatch performance losses, at increased system complexity.

Effects of Mismatch Filtering on MPSK
To limit the amount of data presented to the reader the figures shown for MPSK have a constant α tx = .25 while varying α rx = .3, .4, and .5. Later we provide heatmaps of E b /N 0 degradation results over a range of α tx and α rx scenarios. Figure 9 shows IQ diagrams across orders of M while varying α rx . The IQ diagrams plot the received symbols of the ideal matched filter system overlaid with the received symbols of a mismatched filter system. The left column shows that a small mismatch results in minimal error with every symbol being clearly defined, even at 32PSK. However, on the far right we see a more extreme case of mismatch filtering resulting in more ISI. With less separation between symbols it is expected that higher orders of M are more affected by mismatch filtering. Figure 10 shows a row of BEP curves for M = 16 while varying α rx . The BEP Curves show how mismatch filtering affects P E across E b /N 0 while comparing it to a theory curve. Each curve plots the theory curve for the modulation type, a SA-BEP curve with a perfect matched filter, and a SA-BEP Curve that varies α rx with a constant α tx . These results correspond to the first row of IQ diagrams presented in Figure 9. On the left we see a small mismatch results in minimal error with all three curves tightly together. On the right we a large degradation, denoted as the increase in E b /N 0 to achieve the same P E with perfect matched filter. Figure 11 shows one row of eye diagrams across for M = 8 while varying α rx . The eye diagrams show the effects of the added ISI introduced by mismatched filtering at the maximum eye opening sampling instant of the symbols. The same pattern of Figures 9 and 10 are seen here in terms of eye diagrams: a wide eye on the left side at the sampling instance meaning less ISI and noise. While on the right side the ISI begins to close the eye. Not shown here, higher orders of M are more perturbed by the introduction of mismatch filtering. Figure 12 shows the degradation over various BEP threshold values of {10 −5 , 10 −6 , 10 −7 , 10 −8 , 10 −9 }, M = 4, 8, 16, and 32, and many combinations of α tx /α rx ∈ [1/2, 2]. The degradation is the measured shift in E b /N 0 in dB between ideal theory and a system with filter mismatch at a particular BEP threshold. As M increases and α tx /α rx moves above or below 1 the degradation gets worse. With the worse degradation happening at M = 32 and α tx /α rx reaching the extremes of 1.2 and 2. Note degradation values of less than 0.01 dB are considered insignificant and are entered in the heatmap as zero values.

Effects of Mismatch Filtering on MQAM
Here we show only IQ diagrams for α tx = .25 while varying α rx = .3, .4, and .5. As in the MPSK case later we provide E b /N 0 degradation results over a range of α tx and α rx values. Figure 9 shows two rows of IQ diagrams for M = 16, 256 while varying α rx . The IQ diagrams plot the received symbols of the ideal matched filter system overlaid with the received symbols of a mismatched filter system. The left column shows that a small mismatch results in minimal error with every symbol being clearly defined, even at 256QAM. However, on the far right we see a more extreme case of mismatch filtering resulting in serious ISI, particularly for 256QAM. With less separation between symbols we expected large E b /N 0 degradation will occur in the BEP plots. Figure 14 repeats Figure 12 for MQAM. Results are similar for low modulation M, but the degradation for 256QAM is more serious than 32MPSK. This is not surprising when one considers the IQ diagrams, i.e., signal points are closer in MQAM than MPSK.

With Constrained Use of Equalization
The above results for MPSK and MQAM show that the ISI introduced from mismatch filtering is the greatest at highest modulation orders of, i.e., M, i.e., 32PSK and 256QAM, and when α tx = .25 and α rx = .5. In this subsection we briefly show how even a very simple adaptive equalizer can mitigate filter mismatch. An 11-tap equalizer is chosen to jointly minimize mismatch ISI yet balance noise enhancement. The short tap design can adapt quickly and minimize system complexity. To fit the SA-BEP analysis framework the equalizer is designed for fixed operation at E b /N 0 = 20 dB, while the SA-BEP simulation is run for 20 dB ≤ (E b /N 0 ) dB ≤ 25dB. In general an equalizer for digital communications is made adaptive using the least mean-square (LMS) adaptation algorithm [Ziemer] to minimize the mean-square error (MMSE) between the filter output and hard decision symbol estimates. Figure 15 shows the effects of mismatch filtering when paired with a short length equalizer on 256QAM and α tx /α rx = .25/.5. The E b /N 0 degradation is brought to about 1 dB at P E = 10 −6 . As you can see from Figure 15 the equalizer drastically reduces the ISI introduced by the filter mismatch. Even though the equalizer is designed for an operating point of 20dB it performs well across the entire range of E b /N 0 .

Concluding Discussion and Future Work
The effects of mismatch filtering on lower orders of M in both MPSK and MQAM, in particular M = 4, are almost negligible. With greater than .1dB E b /N 0 degradation when the α tx /α rx ratio reaching the extremes of 1/2 and 2. The effects of mismatch  system. Subfigures (a)(d)(g)(j) show that a small mismatch results in minimal error with every symbol being clearly defined all the way to 32PSK. However, subfigures (c)(f)(i)(l) show a more extreme case of mismatch filtering resulting in more noise and ISI. With less separation between symbols it is expected that higher orders of M are more affected by mismatch filtering.

Discussion
As you can see from the above diagrams the equalizer drastically reduces the ISI introduced by the matched filter in both cases. Even though the equalizer is designed for an operating point of 20dB it performs well across the entire range of = > /? @ . Assuming the user had enough data for the equalizer to get into steady state as well as knew the operating point the signal is being received at you could use an equalizer and a non-ideal RRC filter and have an improved BEP. filtering grow drastically as M increases and the BEP threshold point increases.
The IQ Diagrams show that the symbol clusters with mismatch are not circularly symmetric about the ideal symbol points. In general these cluster clouds, which we know result from ISI, appear biased toward the center of the IQ diagram. Characterizing the cluster cloud probability density function could serve as an alternative to SA-BEP technique presented in this paper.
Also observe is that the degradation values in the heatmaps are essentially symmetric for both MPSK and MQAM, with regard to the α tx /α rx ratio and its inverse. What this means is that the α tx /α rx ratio and its inverse give essentially the same E b /N 0 dB degradation values. Does this make sense? The signal path is identical since the same two filters are connected in series (see Figure 6) in either case. Linear processing means the filter order can be reversed without changing the mismatch. What is different is that the white noise enters at the second filter, which is the receiver input. If the α tx /α rx ratio is less than one more WGN arrives at the receiver decision stage, but more signal energy also enters the receiver, in spite of being mismatched. If the α tx /α rx ratio is greater than one less AWGN arrives at the receiver decision stage, but less signal energy also enters the receiver, again in spite of being mismatched. Although a conjecture at the start of this research, the SA-BEP simulation results in Figures 12 and 14 support the above argument.
The use of SA-BEP modeling allowed this data to be quickly compiled and be easily repeatable. The code could quickly be modified to run any combination of MPSK, α tx /α rx and present the data in any of the above formats. A purpose of this paper was reproducible science, for not only the author to be able to run the code but for any user to use the created code for their purposes and produce the same results. The use of SA-BEP modeling paired with the power and flexibility of object-oriented Python running in Jupyter notebooks accomplishes this goal.