Strange behavior when saving data in csv format

Discussion forum for the new Picoscope Linux software
Post Reply
dnessett
Active User
Active User
Posts: 39
Joined: Tue Oct 16, 2018 10:30 pm

Strange behavior when saving data in csv format

Post by dnessett » Mon May 13, 2019 11:34 pm

Hi,

I am getting some strange behavior when I save spectrum results in csv format. In particular, sometimes the file entries do not correspond to expected bin widths selected when the spectrum is created. For example, I created a spectrum that should have bin widths of 19.07 mHz. However, when the csv file is saved, the bin width is 76.29, (roughly) 4 times larger. I have attached an image that shows the properties and below copy the first rows of the csv file. I can also supply the full file, but it is 275 KB compressed, so I didn't attach it in case it wasn't desired.

Frequency,Channel A
(kHz),(dBV)

0.00000000,-84.97343000
0.00007629,-88.48713000
0.00015259,-95.54079000
0.00022888,-108.80030000
0.00030518,-130.93380000
0.00038147,-128.97480000
0.00045776,-127.82370000
0.00053406,-127.41810000
0.00061035,-128.22430000
0.00068665,-134.52730000
0.00076294,-124.82200000
0.00083923,-116.49030000
0.00091553,-114.14130000
0.00099182,-116.34770000
0.00106812,-123.40880000
0.00114441,-138.59680000
0.00122070,-142.08900000
0.00129700,-140.85600000
0.00137329,-138.98800000
0.00144958,-138.66550000
0.00152588,-140.42930000
0.00160217,-139.42490000
0.00167847,-137.32300000
0.00175476,-139.24670000
0.00183105,-141.46530000
0.00190735,-136.31380000
0.00198364,-130.75950000
0.00205994,-126.10370000
0.00213623,-123.78400000
0.00221252,-125.11980000
0.00228882,-131.02320000

I have run into this before, but thought it might be operator error (which is always a possibility). Can anyone help me with this? I am running PicoScope 6.13.7.707 on Centos 7.

Regards,

Dan
Attachments
Spectrum Properties Anomalous CSV file.png

dnessett
Active User
Active User
Posts: 39
Joined: Tue Oct 16, 2018 10:30 pm

Re: Strange behavior when saving data in csv format

Post by dnessett » Tue May 14, 2019 1:05 am

I know it is unusual to reply to my first post without intervening comment. But, I thought I would run the experiment again in order to save the psdata file (which I failed to do the first time). Now the csv file has bins 2 times the 19.07 bin width. That is:

Frequency,Channel A
(kHz),(dBV)

0.00000000,-83.25687000
0.00003815,-87.83801000
0.00007629,-96.00891000
0.00011444,-110.61150000
0.00015259,-130.21470000
0.00019073,-144.05780000
0.00022888,-138.04730000
0.00026703,-131.21130000
0.00030518,-128.86520000
0.00034332,-130.67900000
0.00038147,-135.09820000
0.00041962,-134.30140000
0.00045776,-130.29360000
0.00049591,-128.82960000
0.00053406,-129.47630000
0.00057220,-131.33260000
0.00061035,-135.96640000
0.00064850,-140.12620000
0.00068665,-142.87640000
0.00072479,-143.02880000
0.00076294,-141.86100000
0.00080109,-140.86340000
0.00083923,-138.50570000
0.00087738,-135.96010000
0.00091553,-136.33530000
0.00095367,-143.48860000
0.00099182,-147.99460000
0.00102997,-141.35930000
0.00106812,-141.66780000
0.00110626,-140.30010000

There seems to be some random factor that is causing the bin widths in csv files to be multiples of the bin width shown in the spectrum properties.

Dan

Gerry
PICO STAFF
PICO STAFF
Posts: 441
Joined: Mon Aug 11, 2014 11:14 am

Re: Strange behavior when saving data in csv format

Post by Gerry » Tue May 14, 2019 1:59 pm

Hi Dan,

Could you post your data file that you saved - thanks.

Just for completeness, on windows and ubuntu there is no such issue as you can see from the screenshot of the PS4262 below:
Spectrum plot and exported bin widths.png
Regards,

Gerry
Gerry
Technical Specialist

dnessett
Active User
Active User
Posts: 39
Joined: Tue Oct 16, 2018 10:30 pm

Re: Strange behavior when saving data in csv format

Post by dnessett » Tue May 14, 2019 8:24 pm

Hi Gerry,

I assume you meant post the csv file. It is ~500 KB compressed. The psdata file is 11 MB (compressed or uncompressed). I have attached the former to this email. I you want the psdata file, let me know. However, it may be too large to attach to a forum message, in which case I can upload it to my Google account and make a link available.

Just to be crystal clear, sometimes saving in csv format works as expected. Other times bin width doubling or quadrupling occurs.

Regards,

Dan
Attachments
TMJ Faraday Run 2_01.csv.tar.gz
(522.53 KiB) Downloaded 42 times

Gerry
PICO STAFF
PICO STAFF
Posts: 441
Joined: Mon Aug 11, 2014 11:14 am

Re: Strange behavior when saving data in csv format

Post by Gerry » Wed May 15, 2019 9:18 am

Hi Dan,

Actually, I meant the psdata file (specifically where you got the doubling or quadrupling when you exported to csv) so that we could see if we can recreate what you are seeing.

When you say sometimes it works as expected, is that consistent, i.e. when it fails with a capture does it always fail with that specific capture or can you sometimes export correctly from the same capture?

Regards,

Gerry
Gerry
Technical Specialist

dnessett
Active User
Active User
Posts: 39
Joined: Tue Oct 16, 2018 10:30 pm

Re: Strange behavior when saving data in csv format

Post by dnessett » Wed May 15, 2019 4:05 pm

Hi Gerry,

I tried attaching the psdata file to a message, but when I previewed the message, the 11 MB file didn't show up as being attached. So I am attaching the csv file (TMJ Faraday Run 2.csv) to this message and have uploaded the file to my Google Drive account. You should be able to download it at:

https://drive.google.com/file/d/1bTqVuM ... sp=sharing

I have also captured a spectrum that reduced the bin width by 32. The csv data is attached (as test 1 - save 1.csv). The psdata file is on my Google Drive account here:

https://drive.google.com/file/d/10DXjfv ... sp=sharing

The spectrum properties are shown in the screenshot attached (test1.png).

To answer your question, I saved the spectrum for test 1 twice. In both cases the bin width was 32 times 19.02 mHz.

Regards,

Dan
Attachments
test 1 - save 1.csv.zip
(33.32 KiB) Downloaded 13 times
TMJ Faraday Run 2.csv.zip
(522.53 KiB) Downloaded 13 times
test 1.png

dnessett
Active User
Active User
Posts: 39
Joined: Tue Oct 16, 2018 10:30 pm

Re: Strange behavior when saving data in csv format

Post by dnessett » Wed May 15, 2019 4:16 pm

Hi Gerry,

Without changing anything, I ran a spectrum capture for the same device connected to my PicoScope 4262. This time the bin width was 4 times 19.02. I have attached the csv file to this message (test2.csv) and uploaded the psdata file to my Google Drive account:

https://drive.google.com/file/d/1gIwRyt ... sp=sharing

Regards,

Dan
Attachments
test2.csv.zip
(268.69 KiB) Downloaded 16 times

Gerry
PICO STAFF
PICO STAFF
Posts: 441
Joined: Mon Aug 11, 2014 11:14 am

Re: Strange behavior when saving data in csv format

Post by Gerry » Wed May 22, 2019 11:08 am

Hi Dan,

Sorry for the delay in responding.

Looking at your psdata file, the problem is only in the last of the 32 buffers that you captured. The last capture somehow managed to store less than 1/4 of a buffer of data so, with less than a 1/4 of the data points to work with, the FFT conversion has to compensate for that by widening the bins in the exported file. But what I don't understand is how you managed to capture less than a 1/4 of a buffer of data in block capture mode (the PS4262 has more than enough buffer memory to capture the data).


I can't recreate what you are seeing in windows, so could you try capturing 32 buffers of data on a much shorter Timebase (say, 100ms/div) with 250,000 samples (or as close as possible to it) and see if the last buffer is complete. If you can recreate the problem at that sample rate then could you try different numbers of total captures (by changing the number of Waveform Buffers in Tools->Preferences->General->Waveform Buffer) to see if there is any kind of pattern to this.

Regards,

Gerry
Gerry
Technical Specialist

dnessett
Active User
Active User
Posts: 39
Joined: Tue Oct 16, 2018 10:30 pm

Re: Strange behavior when saving data in csv format

Post by dnessett » Wed May 22, 2019 10:44 pm

Hi Gerry,

I did what you requested, doing runs with 32 samples and 100 ms/div. I did 5 runs and each showed no problem. The bin size changed, of course. It was 953.7 mHz (I have attached a screenshot of the properties). By the way, I don't know how to tell if the last buffer was complete other than by seeing if the bin width on the csv file is the same as that displayed in the properties window.

However, one thing occurred to me about the previous problematic runs. I had configured (under tools->preferences->general tab) the collection time units as "Total Collection Time" (which I set to 50 seconds - you can see this from the previous properties screenshot I provided). I had to change this to "Times Per Division" to execute the runs you requested. I also noticed that when I ran PicoScope 6 on Windows, I could select the collection time units as "Total Collection Time", but when I actually specified the collection time units, it wouldn't let me provide a total collection time. The only choice I had was times per division. I wonder if the problem has something to do with using "Total Collection Time"?

Cheers,

Dan
Attachments
32 samples 100 ms per div.png

Gerry
PICO STAFF
PICO STAFF
Posts: 441
Joined: Mon Aug 11, 2014 11:14 am

Re: Strange behavior when saving data in csv format

Post by Gerry » Tue May 28, 2019 11:16 am

Hi Dan,

Sorry for the delay in responding.

I noticed your comment on the other post about giving up on the Linux implementation of our software. You're correct in that we don't have feature parity with the Windows version of the software, so you do have to decide if the missing features are what you need.

Regarding this specific issue, it shouldn't really be a problem, because the simple workaround would be to always export all buffers (which should give you a folder of all of the CSV files) and delete all except the first buffer (which will always be complete). You can even set the number of waveforms to 2 in the Preferences (instead of 32) so that you just delete the last one.

When capturing multiple waveforms the last one will always be apparent by the waveform counter (which in your image is showing 32 of 32 i.e. the last one) and will typically be the one being displayed after a multi-capture.

The issue Shouldn't be anything to do with Displaying the units as Total 'Collection Time' (instead of 'Collection Time Per Division') as all it is doing is changing the units. You should be able to select the Total Collection Time in Windows the same way you select the Collection Time Per Division, by just using the left right arrows or drop down list (next to the 'Home' icon).

Regards,

Gerry
Gerry
Technical Specialist

Post Reply