Longtime streaming mode via USB problem

Post general discussions on using our drivers to write your own software here
Post Reply
KTH
Newbie
Posts: 0
Joined: Mon Mar 09, 2020 10:27 am

Longtime streaming mode via USB problem

Post by KTH »

Before I go deeper I would like to define my aim: Longtime data recording via USB, at least with a sample rate of 1024ns.

To do that, i use the PicosCope 2206B with python 3.8 and the SDK. From the datasheet [1] I got the following facts:
  • Max. Sampling rate: 500 MS/s (1Ch) | 250 MS/s (2Ch)
  • Maximum sampling rate (USB stream): 9,7MS/s (31 MS/s with SDK)
  • Buffer Memory for USB Stream mode: 100MS (1Ch) | 50MS (2Ch)
  • Vertical resolution: 8 Bit (max. 12 Bit with downsampling)
Well, just for a better understanding I did the following rough calculations. Each sample is stored in the memory represented by 2Bytes. The communication overhead of USB-Streaming is maybe around 30-40%. This will lead to 10MS/s via USB per channel. That would be around 40MByte/s. The specification of USB2 [2] say, this is the limit without overhead.
So, focus on the aim with 1024ns sampling rate will give around just 1MS/s per channel and a data transfer rate via USB of 4MByte/s for both channels. That should be fine for the device and USB.

To test a long time streaming I used a function generator with 10Hz and 50Hz and start streaming by ps2000aStreamingExample.py [3]. I run this function with different sample rates, no downsampling and try to store more than 100MS (50MS per Ch) via USB at disc. For 16ns the maximum usb buffer is reached after 0,8 seconds and for 1024ns after 51,2 seconds.
The results are shown in the following pictures for a sample rate of 16ns and 1024ns. The x-axis is the time in second.
SamplingRFF.jpg
After around the maximum buffer memory for USB Stream mode (100MS) somthing get wrong.
Is more clear at FFT. Left picture shows the fft for less that 100MS and the right picture for more than 100MS. After the max of USB memory the sampling rate changed or/and some not nice internal stuff is happened.
Sampling.jpg
Anyway - with that I’m not able to stream over a long period of time. Is that a problem of streaming mode, firmware, dll, ...? Should or could i use Block mode instead? Thanks for help.

There is one more thing. I would also like to use the downsampling (256) in a next step. This would bring addition 4Bits of resolution. 4ns sample rate (safety factor 2) and 256 downsamling. For that I should get only 4MB of USB transfer per second and 10 internal Blocks for 25ms (32MS) to store data. Read one block vis USB, one Block get new data and 8 Blocks buffer for reading. Maybe i have to adapt the block size to reduce the USB overhead.

Is something like that really possible with the SDK?
Even, just 10Bits (8Bits) resolution and 1024ns sampling rate would be fine.

Documents
[1] https://www.picotech.com/download/datas ... eet-de.pdf
[2] https://de.wikipedia.org/wiki/Universal_Serial_Bus
[3] https://github.com/picotech/picosdk-pyt ... Example.py

Gerry
PICO STAFF
PICO STAFF
Posts: 1034
Joined: Mon Aug 11, 2014 11:14 am

Re: Longtime streaming mode via USB problem

Post by Gerry »

Hi KTH,

First of all, your goal of a sample period of 1024ns should be achievable with the PicoScope 2206B, but it depends upon the speed of your data handling.

Just to clear up a few misconceptions. the vertical resolution of the 2206B is 8-bit only. Downsampling does not gain you any higher resolution. Our PicoScope 6 software has a feature called Enhanced Resolution Mode, which applies processing to increase the effective resolution to a maximum of 12-bits (which is where you may be getting confused). So each sample is stored in only 1 byte of the Buffer memory in the Hardware PicoScope.

The maximum rate that you can reliably stream samples from the PicoScope 2206B to the computer would be 31MS/s as stated in our Data Sheet (which would be shared among channels). So, for instance, if you are using 2 channels, you should be able to stream the samples at the closest 2206B sample rate to 15.5MS/s. However, this is assuming that you are just collecting the samples and storing them. If you are performing tasks that take significant enough time for processing on the computer, then you will not be able to handle the incoming streamed samples quickly enough to avoid not-yet-transferred data in the hardware PicoScope buffers from being overwritten by new incoming data.

Looking at your examples, a sample rate of 62.5MS/s (1024ms sample period) and even a sample rate of 9.7MS/s (1024ns sample period) may well be too fast for collecting a batch of the data and sending it to a plotting function to then be displayed on the screen, before getting back to the overview buffer to transfer the next batch of data to computer memory before displaying it. This will be because the rate at which you are emptying the overview buffer will be slower than the rate at which it is being filled by the Driver. Eventually the Driver will reach a point where it doesn't have anymore room in the overview buffer, so it can't accept any more transferred data. The Hardware PicoScope will then not be able to transfer anymore data, and will have to, instead, start overwriting old data that hasn't yet been transferred to the computer.

So this is what you are seeing, in your second images. There is missing data, because some of your data has been overwritten by newer data, before being transferred. So you need to do one or more of the following:
(1) To stream the data at faster rates just use a minimal, efficient transfer routine, to just transfer the data from the overview buffer, and leave any display/processing/saving of the data until it has all been collected.
(2) If you are downsampling the data then you will be able to receive the same information (at a lower frequency resolution) with a slower sample rate so you will have more time to collect and process the data (note that, as already mentioned the data will still only be 8-bits when downsampled).
(3) If you must display the data without downsampling, you should reduce the sample rate to a more manageable rate, based upon the amount of overhead required for your displaying of the data.
(4) If you use a computer with more powerful resources (processing/memory, etc) and/or less overhead (stripped down background tasks), you will be able to use faster streaming rates without data loss.

Regards,

Gerry
Gerry
Technical Specialist

bennog
Advanced User
Advanced User
Posts: 102
Joined: Mon Nov 26, 2012 9:16 am
Location: Netherlands

Re: Longtime streaming mode via USB problem

Post by bennog »

I would use C++ for this and use multiple threads.
1 for reading the data from the picoscope.
1 for processing the read data (without locking the 1st thread)
1 for displaying the result or send the result to a python display app.

Benno

Gerry
PICO STAFF
PICO STAFF
Posts: 1034
Joined: Mon Aug 11, 2014 11:14 am

Re: Longtime streaming mode via USB problem

Post by Gerry »

Hi Kth,

If you can implement it then Benno's solution could be a better solution (if it you are still able to empty the overview buffer faster than it is being filled).

Regards,

Gerry
Gerry
Technical Specialist

KTH
Newbie
Posts: 0
Joined: Mon Mar 09, 2020 10:27 am

Re: Longtime streaming mode via USB problem

Post by KTH »

Hallo Gerry,

really thanks for your detail explination. After testing, I tought it runs that way. I'm a bit ... .

Well, you "sell" this as fast streaming mode devices. But in fact, it's a oscilloscope and no device to store high speed data over a long time via USB.
Anyway, I leared and have an oscilloscope. For sampling data, i'll go an other way.

At the end, the support is great, thank you for that

KTH
Newbie
Posts: 0
Joined: Mon Mar 09, 2020 10:27 am

Re: Longtime streaming mode via USB problem

Post by KTH »

Hello,

to digging deeper:
I just read out directly the samples from Picoscop into the RAM. No drawing or anything, just read into the RAM (PC). In a post processing I draw some figures.
All works fine to the point the USB-databuffer is full. (around 0,78s - as shown in the fig., see above).

So i reduced the sampling rate to 1024, 2048, 4096, and so on.
At the end, I always hit the same problem. I believe there is no way to get a real streaming with the Picoscop.

Do someone really stream data over a long period? If so, I will sit-down again and also find a solution and post it.

bennog
Advanced User
Advanced User
Posts: 102
Joined: Mon Nov 26, 2012 9:16 am
Location: Netherlands

Re: Longtime streaming mode via USB problem

Post by bennog »

I have done streaming 10Ms/sec for about 4 weeks 24/7 so it is possible.

Also done 4 ch 5Ms/sec (so a total 20Ms/sec) for about 8 hours.

O and both of the above was done on a USB2 type scope the 4424.
I have also a USB3 version 4444 scope but not tested, it should stream even faster.

Benno

Gerry
PICO STAFF
PICO STAFF
Posts: 1034
Joined: Mon Aug 11, 2014 11:14 am

Re: Longtime streaming mode via USB problem

Post by Gerry »

Hi KTH,

We 'sell' the devices as PC oscilloscopes (look under 'Oscilloscopes' here: https://www.picotech.com/products) which are Oscilloscopes that use the PC for display, control and long term storage. What you are paying for is just the part of the Oscilloscope that captures the data, as you already have the display, processing, and storage in your computer. This means that as a company, we can do what we do best, i.e. put the extra time and money into making the data capture better for less money than you would pay for a Desktop oscilloscope.

Streaming Mode is just one feature of the device, and we just refer to Streaming as USB streaming. In fact, in our PicoScope 6 software, we use the term Slow Sampling Mode, which essentially uses streaming, and Fast Sampling Mode which captures the data in blocks faster than you can stream it. So we certainly don't "sell" the PicoScope 2206B as a fast streaming mode device.

In my last response I explained the factors that will affect how fast you can stream the data, and with an application written to implement streaming mode efficiently enough, and a computer with enough resources and fast enough processing, you can stream the data at the rates that we quote (how do you think we were able to establish what the fastest rates are).

I wrote an application for the older 5000 series PicoScopes that could stream the data for long periods, at a rate that would be comparatively fast for streaming, while checking for a break in the waveform (as a sign of overwritten data) but didn't do anything else other than storing the data. So, yes, you can stream the data over long periods.

I'm about to go on holiday, but when I'm back for the new year I will see if I can find that streaming code (which incidentally, wasn't even written as Benno suggested).

Regards,

Gerry
Gerry
Technical Specialist

KTH
Newbie
Posts: 0
Joined: Mon Mar 09, 2020 10:27 am

Re: Longtime streaming mode via USB problem

Post by KTH »

Hello,
that sounds great. I'll have to take a closer look on what is happened in my case. Now, that i know it will run - really thanks to @bennog - i'll find a solution

rzingg
Newbie
Posts: 0
Joined: Tue Sep 28, 2021 2:21 pm

Re: Longtime streaming mode via USB problem

Post by rzingg »

@Gerry

We try to log 1h of data and the most important thing is to not lose any samples. We want to store the data into .csv or .txt files and for this we plan to use the streaming mode and the python-api.

Is this the correct approach?
Could you provide your example application because this sounds similar to our application!
Thanks alot

Raphael

bennog
Advanced User
Advanced User
Posts: 102
Joined: Mon Nov 26, 2012 9:16 am
Location: Netherlands

Re: Longtime streaming mode via USB problem

Post by bennog »

Depending on the sample rate and if your HD can keep up with the writing.
Also the conversion from integer data to ASCII takes up some (a lot) of time if you want to write 1000000 values per second.

Benno

Post Reply