Corrections for Windowing functions in FFT calculations

Having problems ? let us know the details here
Post Reply
dnessett
Active User
Active User
Posts: 39
Joined: Tue Oct 16, 2018 10:30 pm

Corrections for Windowing functions in FFT calculations

Post by dnessett » Sun Apr 21, 2019 5:28 am

Hi,

This is really not a "getting started" sort of question, but I could find no other board to use in asking it.

When the PicoScope FFT spectrum analyzer is applied to an input signal, there is the option of selecting a window function to reduce spectral leakage from one bin to another. A general discussion of this problem is provided in this paper, which concentrates on power spectra, but the whole idea of spectral leakage applies to any FFT calculation, including that in which the spectral values are voltages or log voltages (i.e., dBV). I have had extensive discussions on this topic with Gerry in this thread and many of my questions were answered by him.

However, there is one question I failed to ask; specifically, when a windowing function is selected, are the data in the bins of the resulting spectrum adjusted by the effective noise bandwidth associated with that function. This is normally required when the spectrum comprises values associated with white noise. For power spectra, this adjustment comprises dividing the bin values by a constant associated with the windowing function. For example, if Blackman-Harris windowing is selected, the constant is 1.7.

The question is: does the PicoScope 6 software perform this correction, or is it the responsibility of the user to do so. This is important because if the PicoScope 6 software performs the correction, the user should not, because that would give an incorrect result.

This problem is illustrated by attempting to compute the noise floor of the PicoScope. If one puts a 50 ohm terminator on an input and then collects data from that input, the result will represent the noise floor of the instrument. However, it isn't clear whether that is the case, since if the PicoScope 6 software does not correct for spectral leakage due to the windowing function, then the results would have to be divided by the constant associated with it.

I have asked similar questions before and Gerry provided very detailed and informative answers. However, I failed to ask this specific question. I am computing the noise floor of the PicoScope 4262, so I need to know whether the PicoScope 6 software performs the correction or if I need to do so.

Gerry
PICO STAFF
PICO STAFF
Posts: 440
Joined: Mon Aug 11, 2014 11:14 am

Re: Corrections for Windowing functions in FFT calculations

Post by Gerry » Mon Apr 29, 2019 3:00 pm

Hi Dan,

Apologies for the delay in responding.

The Windowing functions are necessary to prevent spectral leakage of the signal where the signal contains frequency components that have not been sampled coherently. Incoherent sampling is where, for a specific frequency, the captured data does not result in an integer number cycles, and therefore results in transients being created by the Fourier transform process of repeating the the captured data. The transients in the Time Domain are transformed into wideband Magnitude components in the frequency domain which means that they are leaking across many bins (in fact all bins to some extent). The Windowing functions change the start and end of the data in a window so that when the captured data is repeated there is minimal disjoint, and therefore minimal transients and minimal wideband spectral leakage. Because the windowing functions alter the the actual data, it must be compensated, according to whether the spectrum is an Energy (Power) Spectrum, or Amplitude (Voltage) spectrum (because the correction values are different). PicoScope 6 applies coherent gain to the Window Function so that the Amplitude level is matched before and after the FFT is applied, because we plot an Amplitude (Voltage) spectrum.

If you need any further explanation of windowing then I fully recommend these 2 articles (2 of the best articles for simple explanations of the phenomenon that I have found): https://community.plm.automation.siemen ... a-p/432760, and https://community.plm.automation.siemen ... a-p/431775

The answer to your question regarding the noise floor is that you need to be very careful about measuring noise in a Spectrum plot created from a DFT or FFT. First of all when analysing noise in a Spectrum plot, most people are interested in noise power Spectrums as these tend to be how noise is analyzed, and the calculations of standardized measurements are straightforward to perform (as, for instance RMS is based on the squares of the magnitudes, which is what you get in a power spectrum, for which the appropriate windowing function correction will have been applied, i.e. for energy/power). Also, as I've mentioned in previous posts, you need to take into account Process Gain which reduces the noise floor as a proportional logarithm of the number of bins used for the plot. So, As PicoScope 6 computes the Voltage and applies the windowing correction for that, you would need to undo the windowing correction for voltage, square the component for power, and then apply the correction for power, if you want the power spectrum equivalent of each plotted value.

Regards,

Gerry
Gerry
Technical Specialist

dnessett
Active User
Active User
Posts: 39
Joined: Tue Oct 16, 2018 10:30 pm

Re: Corrections for Windowing functions in FFT calculations

Post by dnessett » Sun May 19, 2019 10:52 pm

Hi Gerry,

Previously, I decided to ignore the corrections for ENBW associated with a particular window function, since they were small. Consequently, while I read and understood your response to my original question, I did not pursue it further. I have changed my mind and now wish to ensure these corrections are applied.

When I read your last post on this topic, you indicated that the PicoScope 6 software applies the window corrections to the amplitude data in order to compensate for spectral leakage. Just be clear, I am displaying the spectrum in dBV units and capturing that data to a csv file (I have moved to a Windows based version of PicoScope 6, since the Linux version doesn't seem to correctly capture csv data, as we are discussing in this topic). However, you say in order to properly compute power spectrum data, I need to back out the voltage corrections, square the voltages and then apply the window corrections associated with power spectra.

In order to do that, I need to know the correction factor (for Blackman-Harris) PicoScope 6 applied to the voltage data. I also need to know what variation of Blackman-Harris you are using (e.g., 3-term, 4-term, etc.), so I can apply the correct power correction factor.

In addition, there is another issue concerning the PicoScope 6 software that I need to understand. According to this this paper (see section entitled "Computing Power Level and Spectral Noise Density"), the correction that should be applied is division by the the Noise Power Bandwidth of the window times the bin width. So, if the bin width is, say 19 mHz, the denominator correcting the power value would be .019*1.7 (assuming a 3-term Blackman-Harris window). You did not mention whether the PicoScope 6 software uses the bin width in the correction computation. I need to know this, since if the bin width is used, I need to back out the correction by multiplying by the Noise Power Bandwidth of the window times the bin width of the window.

Regards,

Dan

Gerry
PICO STAFF
PICO STAFF
Posts: 440
Joined: Mon Aug 11, 2014 11:14 am

Re: Corrections for Windowing functions in FFT calculations

Post by Gerry » Wed May 22, 2019 5:36 pm

Hi Dan,

To answer your question, the correction factor is correcting the amount that a windowed data element represents so, for example, for a Hanning window you would need to multiply the data element by 2 in order to convert the amplitude peak represented in a power spectrum to what it should be in an amplitude spectrum. So, as you apply a windowing function to a bin, by definition, the correction is applied according to the number of bins that there are, which is obviously inversely proportional to the bin width. So, the bin width must be (indirectly) taken into consideration in computing a correction (for plotting or conversion) and, as mentioned previously, we apply the amplitude correction to the bins as we ensure that the level is matched before and after computation of the plotted values.

Regards,

Gerry
Gerry
Technical Specialist

dnessett
Active User
Active User
Posts: 39
Joined: Tue Oct 16, 2018 10:30 pm

Re: Corrections for Windowing functions in FFT calculations

Post by dnessett » Wed May 22, 2019 9:04 pm

Hi Gerry,

Thanks for the response. Just to make sure I understand correctly, the adjustment PicoScope 6 makes to the data is to divide by the factor associated with the window, not by this factor times the bin width. So, to back it out, I multiply by that factor. Correct?

However, I still lack the necessary information to convert the amplitude based measurement (with window correction) to a power based measurement. In particular, I am using Blackman-Harris windowing. The links you gave in a previous message (from siemans) do not specify an amplitude correction factor for Blackman-Harris, so I don't know what value to use as a multiplier to back out the correction PicoScope 6 has applied. Also, there are variations of Blackman-Harris (see table 1 in this paper by Harris). It identifies two variants: 1) minimum 3-sample Blackman-Harris, and 2) minimum 4-sample Blackman-Harris. These variants have different power correction factors. I need to know which of these (or perhaps another) variant PicoScope 6 is using in order to apply the appropriate correction.

Regards,

Dan

Gerry
PICO STAFF
PICO STAFF
Posts: 440
Joined: Mon Aug 11, 2014 11:14 am

Re: Corrections for Windowing functions in FFT calculations

Post by Gerry » Fri May 24, 2019 11:11 am

Hi Dan,

We actually use a library function that takes configuration data, one element of which is the number of bins and another element being the sample rate, so as the bandwidth is up to the nyquist frequency the library can calculate the bin width as bandwidth/number of bins. In my last reply I was telling you what information we make use of (to show you that we do take the bin width into account), the calculation will still be "times the bin width", or "times the bandwidth divided by the number of bins".

We use the most popular form of Blackman-Harris which is the min 4-term variant.

Regards,

Gerry
Gerry
Technical Specialist

dnessett
Active User
Active User
Posts: 39
Joined: Tue Oct 16, 2018 10:30 pm

Re: Corrections for Windowing functions in FFT calculations

Post by dnessett » Sat May 25, 2019 1:13 am

Thanks Gerry,

I am now missing only one thing I need to convert windowed data with amplitude correction to windowed data with power correction. The amplitude correction factor for 4-term Blackman-Harris. As this is probably hidden in the library function PicoScope 6 calls, I figured you might not be able to tell me what this factor is. So, I began a long search for this value with no luck.

Fortunately, during my search I ran across a short blog note that solves the problem. At the end of this note is a discussion how to convert amplitude corrected spectra to power corrected spectra. If you look at the equation in the section "Converting Amplitude to Energy Correction Scaling" and assume you are summing over a single bin, then the RMS voltage value equals amplitude_RMS divided by the square root of the ENBW, where ENBW = window effective bandwidth * bin width. So, there is no reason to back out the amplitude correction. Just divide the amplitude corrected bin value by this conversion factor and the result is the uncorrected voltage value.

So, for 4-term Blackman-Harris (effective bandwidth= 2, see Table 1 in this paper by Harris), take the result returned in the CSV file by PicoScope 6 operating in amplitude mode, divide by the square-root of (2*bin_width)) and square the result. Or more simply, square the result and then divide by 2*bin_width.

Added after submitting post: I don't know why the forum software is not properly formatting the URLs. The second one is a copy/paste of the source I used in a previous message that worked correctly.

Added 5-25-2019: I reread the blog post this morning and I believe the analysis above is wrong. Using the character "_" to represent subscripting, Total power = sum-over-all-bins(Power_i/ENBW). If the sum is over exactly one bin, then the correct Power value will be: Power_i/ENBW, where "i" identifies the bin. The values returned by amplitude correction are RMS_amplitude/sqrt(ENBW), again this is apparent by using the formula in the last section and summing over one bin. So, if these values are squared, the result is: RMS_amplitude^2/ENBW, which is Power_i/ENBW. If this analysis is correct, I need do no further ENBW correction to the amplitude corrected values in order to obtain the equivalent power value.

However, there is another issue that isn't so easy to analyze. The amplitude values returned by PicoScope 6 are averages. In other words, for n samples, a_i, the value returned would be: (a_1+a_2+...+a_n)/(n*sqrt(ENBW)) The power values returned by PicoScope 6 are also averages. For n samples, p_1, the value returned would be: (p_1+p_2+...+p_n)/(n*ENBW). But, p_i = a_1^2, so the power value is: (a_1^+a_2^+...+a_n^)/(n*ENBW). So, just squaring the summed amplitude values doesn't give the correct power value. The ENBW correction is fine, but the square of the amplitude sum is not the sum of the amplitude squares. I am not sure how to get around this.

dnessett
Active User
Active User
Posts: 39
Joined: Tue Oct 16, 2018 10:30 pm

Re: Corrections for Windowing functions in FFT calculations

Post by dnessett » Tue May 28, 2019 12:59 am

I am sad to say that I have given up on using the Linux version of PicoScope 6 and now use the Windows version. I have switched to dBm@50 ohms measurements and given up trying to convert dBV values into dBm values. The problem is the Linux version is just too crippled (no dBm@50 ohms, problems with last buffer processing). It is a shame, since unless people use the Linux version, it will never gain traction. I don't think it will be used extensively until its features are brought into line with the Windows version.

On a more positive note, I have published some material on the EEVblog that describes how I make measurements of phase noise using a PicoScope 4262. In case anyone is interested, this material begins with this post and continues with the next 3.

dnessett
Active User
Active User
Posts: 39
Joined: Tue Oct 16, 2018 10:30 pm

Re: Corrections for Windowing functions in FFT calculations

Post by dnessett » Fri May 31, 2019 5:30 am

Hi Gerry,

I am getting some unexpected results when analyzing the phase noise of an oscillator. After re-examining the corrections I apply (and they seem correct), it occurred to me that when gathering data at dBm@50 ohms, the PicoScope 6 software may already correct the output for processing gain. Might this be the case?

Cheers,

Dan

dnessett
Active User
Active User
Posts: 39
Joined: Tue Oct 16, 2018 10:30 pm

Re: Corrections for Windowing functions in FFT calculations

Post by dnessett » Fri May 31, 2019 11:02 pm

Hi Gerry,

I am getting unexpected results, but I don't think they are due to PicoScope 6 correcting for processing gain when using dBm@50 ohm display. In fact, I don't think they have anything to do with the PicoScope, so no need to reply to my previous message.

Sorry for the bother.

Regards,

Dan

Gerry
PICO STAFF
PICO STAFF
Posts: 440
Joined: Mon Aug 11, 2014 11:14 am

Re: Corrections for Windowing functions in FFT calculations

Post by Gerry » Mon Jun 03, 2019 5:46 pm

Hi Dan,

No problem. I kept meaning to get back to you regarding your earlier question of the values returned by PicoScope 6 when using the Effective Noise Bandwidth so, in all fairness, I haven't been able to help you move forward.

I've had a look at this and I will have some time tomorrow to give you a definitive answer.

Regards,

Gerry
Gerry
Technical Specialist

Gerry
PICO STAFF
PICO STAFF
Posts: 440
Joined: Mon Aug 11, 2014 11:14 am

Re: Corrections for Windowing functions in FFT calculations

Post by Gerry » Fri Jun 07, 2019 3:44 pm

Sorry Dan,

I didn't manage to get a clear answer completed, so I will definitely deal with it tomorrow.

Regards,

Gerry
Gerry
Technical Specialist

Gerry
PICO STAFF
PICO STAFF
Posts: 440
Joined: Mon Aug 11, 2014 11:14 am

Re: Corrections for Windowing functions in FFT calculations

Post by Gerry » Wed Jun 12, 2019 6:00 pm

Hi Dan,

I apologise for the delay in answering. The reason for the delay is that I had to find time to research, in order to clear up some ambiguities regarding the question you posed.

Fundamental FFT maths, simply applied, in general is well documented, but because the FFT is so widely used, with many freely accessible libraries, the available documentation tends to focus on the transformation rather than the preparation and corrections that you would need for measurement accuracy. When you do happen to find what you're looking for it often contains a heavy emphasis on mathematical proof with very short or unclear explanations, or examples of applying the principles, and somewhat ambiguous references for terms that just adds confusion.

So, to try and provide a little clarity, and make things less ambiguous, let's consider that the standard way of referring to the points plotted in any Spectrum as a result of applying an FFT to time domain data is 'Spectrum Amplitudes' (there is a discussion of why this is the best terminology to use here: https://www.nist.gov/sites/default/files/documents/calibrations/tn699.pdf). We can then consider how we prepare the data captured for transformation into Spectrum Amplitudes, what the transformation does to the data, and what these spectrum amplitudes will represent as a result.
So, for instance, a Linear Spectrum would be created by directly using Potential Difference data with units of Volts. In general, the FFT algorithm averages the time data over the period of the capture to plot a Spectrum of the data, so Potential Difference data would be transformed into Spectrum Amplitudes having units of Volts[mean]/Hertz.
For a Power spectrum we would use the Potential Difference data and assume that it has been measured across an impedance of 1 ohm, so that we can just square the data to represent Power before transforming it. The squared values would also be averaged by the FFT algorithm creating Spectrum Amplitudes in units of (Volts^2)[mean]/Hertz. Note that if we were to take the square root of the Power Spectra Amplitudes, we would have √((Volts^2)[mean])/Hertz which, put another way, is Volts[rms]/Hertz, which is a hint of how you convert from Power Spectra Amplitudes to Linear Spectra Amplitudes. So, for general conversion between the Spectra for the different modes of measurement, the process is quite simple, and actually what you would intuitively expect (I won't repeat the Mathematical proof here as there are plenty of sources on the Internet for that).

Converting between a Linear Spectrum and a Power Spectrum
  • Power Spectrum Amplitude = (Linear Spectrum Amplitude)^2
    Linear Spectrum Amplitude = √(Power Spectrum Amplitude)
You will notice that in the previous paragraph I was referring to the units of Potential difference as Volts. Again for clarity, the potential difference refers to the instantaneous voltage measured, i.e. Volts[AC]. The FFT plot of this will give you 2 double sided Spectra (a positive frequency side and negative frequency side), one double sided plot being Real and the other Imaginary. To convert the 2 sets of double sided plots to single sided plots is simple. You then have 3 choices of what to plot as the Spectra, (a) the Real and Imaginary data (which is useful for certain analysis processing applications, e.g. reducing the level of the noise floor relative to signals through data record averaging), (b) the Magnitude and Phase of the Spectra, by performing rectangular to polar conversion, or just the Magnitude, or (c) using the Magnitude (which will be the peak amplitude of the Time Domain signal) and converting it to Volts[rms] (i.e. divide by √2, bearing in mind that the FFT plots Amplitude Spectra are representing pure sinewaves). An FFT library could produce any of these, or any number of them according to parameters that are set. So, the conversions are as follows:

Converting Double sided (+ve & -ve frequency) Spectrum to Single Sided (+ve)
  • Discard the -ve Sided Spectrum (except the plotted point at 0Hz), and multiply all Spectral Amplitudes (except the point at 0Hz, and the one at the Nyquist Frequency) by 2
Converting Single Sided, Real and Imaginary Amplitude to Polar form
  • Peak Amplitude = √ ( (Real Amplitude)^2 + (Imaginary Amplitude)^2 )
    Phase = ArcTan (Imaginary value / Real value)
Converting Peak Amplitude to RMS Amplitude
  • RMS Amplitude = Peak Amplitude/√2
In PicoScope 6, we plot the single sided V[rms] Linear Spectrum of the Time domain data (later converted to a logarithmic scale). To validate this I have created a Time domain plot and Spectrum plot of the same data, along with the data file, which you see below.
Time Domain plot of 1V peak generated signal with peak measurement.png
Spectrum plot of 1V peak signal with peak measurement.png
As you can see from the image of the Time Domain plot, the Signal generator is generating a 996mV signal with a 4mV offset to maximize the measurement range, which when captured is a 997.9mV peak, and in the image of the Spectrum plot, the peak of the fundamantal frequency of the signal is at -3.223dBv (or -3.223dB relative to 1V), which is 0.690081V. There is some very low level harmonic distortion at -96.17dBV which is 0.000016V giving a grand total of 0.690097V. The true RMS value of the 997.9mV peak should be 0.9979/√2= 0.705622. So, the measured value shows that the plotted value is the rms value with a 0.005922V or 0.8% error (which is close enough).

Regarding using windowing and the effects that this has on the results of a Linear Spectrum or Power Spectrum, as the Window correction for Amplitude or Energy is a linear scaling of the post transformed data, there is no need to roll back the transformation as I previously mentioned (you can just apply all scaling after the FFT). The document that you referred to in your post here: https://www.picotech.com/support/topic39393.html?&p=140473#p140349, goes through the step by step process of Windowing, performing the the FFT, and then applying the corrections very well (as summarised on page 21). However, the author emphasises that squaring of the Spectra amplitude is required before averaging the collected records but, as we have discussed previously, we don't do that in PicoScope 6, and there are valid reasons for not doing that (one being mentioned in this document under 'Averaging Techniques' on page 10: https://static1.squarespace.com/static/5230e9f8e4b06ab69d1d8068/t/56ce6c37f699bb91e5aa778b/1456368704362/Dynamic+Signal+Analysis+Basics.pdf).

So, once windowing has been applied and the FFT has been performed, corrections need to be applied according to how they have changed the data. If you are plotting a Linear Spectrum of peak Magnitude then you should compensate for the reduction of the Spectrum Amplitudes due to applying the window function (i.e. dividing by Coherent Power Gain, which, contrary to its name, is actually a loss factor). If using a Power Spectrum, then you should correct for the increase in energy due to the noise that is also integrated under the window envelope, by dividing by the Equivalent Noise BandWidth (ENBW). (Note that there are tables for the windowing corrections in the reference that you gave for "On the Use of Windows for Harmonic Analysis with the Discrete Fourier Transform" by Harris). However, the corrections will need to be further adjusted, not only according to the window type used, but also depending upon how much Window Overlap there is. There may also be Scalloping Loss (also window type dependent).Note that if you want to avoid having to compensate for all of these 3 parameters you can use a Flat Top window (although the 4-term Blackman-Harris window is a good all round performing window). This document discusses these losses, but doesn't mention Window Overlap: https://kluedo.ub.uni-kl.de/frontdoor/index/index/docId/4293.

The Document that prompted your question (i.e.: http://blog.prosig.com/2009/09/01/amplitude-and-energy-correction-a-brief-summary/) explains the correction for a Linear Spectrum of RMS Magnitude.

Note that for highest accuracy of Signal Amplitude measurements in Spectral Analysis you should also consider the accuracy of the calculations or tables used for the Sine/Cosine functions used in the analysis (which can be the largest contributor of errors). We use a library of highly optimized, adjustable accuracy functions, which will have been set to high enough accuracy level to not be significant.

I hope this helps,

Regards,

Gerry
Gerry
Technical Specialist

dnessett
Active User
Active User
Posts: 39
Joined: Tue Oct 16, 2018 10:30 pm

Re: Corrections for Windowing functions in FFT calculations

Post by dnessett » Thu Jun 13, 2019 10:34 pm

Hi Gerry,

Thanks for the well-researched and highly readable analysis of the problems I raised. Also thanks for the references you provided, especially the one on Dynamic Signal Analysis Basics. I have added these to my library of tutorials on digital signal analysis and will, no doubt, refer back to them in the future.

I don't know if you are interested in applications that use PicoScope products, but I have documented my use in a recent set of posts on EEVblog. The topic is characterizing hobbyist oscillators and the first example is the Morion MV89A (which I purchased used on ebay and which have not performed well). If interested, you can find the topic here: https://www.eevblog.com/forum/metrology/analysis-of-some-hobbiest-oscillators/

Regards,

Dan

Gerry
PICO STAFF
PICO STAFF
Posts: 440
Joined: Mon Aug 11, 2014 11:14 am

Re: Corrections for Windowing functions in FFT calculations

Post by Gerry » Tue Jul 16, 2019 9:12 am

Hi Dan,

I meant to come back to this post but forgot, so thanks for the links - I will take a look when I have some time.

Regards,

Gerry
Gerry
Technical Specialist

Post Reply