Adding capabilities to my ELS system

[rabler]

I can't number the variables. I barely know what they are when written out. I won't remember what they are when I code this, and that makes the likelihood of a garbage log near certainty.

Can you create a data dump structure, write/copy what you want to that structure? Then dump the whole structure to the serial port via something very similar to Dan's FTDISerial.write routine. That way on the receive end you just use the same structure definition to parse the data (assuming you don't go between big endian/little endian machines). That may mean sending some irrelevant data to the current snapshot, so isn't optimal, but it saves a lot of parsing headaches.

 

[WobblyHand]

Can you create a data dump structure, write/copy what you want to that structure? Then dump the whole structure to the serial port via something very similar to Dan's FTDISerial.write routine. That way on the receive end you just use the same structure definition to parse the data (assuming you don't go between big endian/little endian machines). That may mean sending some irrelevant data to the current snapshot, so isn't optimal, but it saves a lot of parsing headaches.

Yes I can. It does mean a lot of irrelevant stuff gets pushed over the wire, but at least the structure has a known format. I will make up a dump list and compare how many bytes it is, vs, just doing text. I want to know the payload, binary vs text. Or if I can pre-format anything. Trying to figure out what makes sense from both a time and payload perspective.

For mass data export, of course binary makes sense, since the format is known. But sending stuff that's irrelevant is a large loss in efficiency. And a bit of work for me, so I'm checking to see if the juice is worth the squeeze. I have about 210 variables in my main header file. (I left some out.) Clearly, I'm not going to send that amount of data every time! They are a mixture of booleans, ints of all sorts, signed and unsigned, floats and a rare double. So at any one time, maybe 10% of the data payload could be relevant. This is what is my concern - moving that 90% of irrelevant data in binary may negate any savings in transport time for far smaller payload in ascii. For binary there's no slowdown due to formatting, however. I don't know how to evaluate that yet.

This is a big program, I'm no longer testing a small routine, I'm searching out a bug hidden somewhere in a complex routine, that has multiple interrupt levels, and a pretty tricky state machine complete with some error checking. All my logging to date is showing reasonable behavior at program level, but a loss of repeatable phase start. The thread pitches remain as originally designed, they are created with "zero" error.

Definitely been interesting. I'll be glad once I figure it out. Probably give myself a dope slap for overlooking the painfully obvious cause. At the moment, it's there, in plain view but hanging in my personal blind spot. I've been living with this code base for about 3 years now... But the parts that are kicking my butt are within the past couple of months. (Feed to stop, and thread to stop.)

 

[WobblyHand]

I can laugh now, but I was very unhappy last night and this morning. Made a lot of edits yesterday and something was preventing compilation, breaking the project. The compiler complained about not finding an initializer and pointed to the first line in a file. There was nothing wrong that I could see with the file. (Usually the issue is a dropped semicolon, or non-matching parentheses...)

I did a git restore of that file and it still complained. All that I had done was some formatting and code compaction, but no logic changes. The error was in the preceding file! I just found that out, by chance, a few minutes ago.

I was thinking of making a logging function and made a stub in main. Unfortunately, I got distracted and didn't finish it. So the function said

void myfunc

No parentheses, no semicolon, just as written, on the last line of main before the DRO file that had a compilation error.. Dunno, maybe I did that just before dinner... I'm lucky that the order of compilation was such that it was the file to the left, rather than some other file (there are 8 source files and 3 header files in the project).

No, it hasn't solved my logging problem, but at least I can move forward again. I thought I was losing my marbles last night... I lost my edits in the DRO file. They were only to make a more compact printout, they weren't functional. All the other pending edits were saved, especially my adding in the FTDI initialization stuff. I could have stashed the changes, (in git) but I was too excited/agitated to do that. I needed the project to compile again! Interesting what irrational things one does under pressure. I don't know git that well, so when something unexpected happens, I need to look it up, and (as usual) there's a lot of garbage advice out there. It takes time to separate the authoritative information from the rubbish.

If I recall, I was evaluating what data to log... Back to my regularly scheduled programming. First up, do a commit, so I have a place to restore to. Git has saved me quite a few times, definitely recommended.

 

[dkemppai]

So your function was void! LOL

I can laugh, because I struggle with C sometimes. For me it's nested If else if stuff. I always end up with a } missing somewhere!
In those situations, all I can do is remind myself to take a breath, and start with the first error in the list. One step at a time. I do a lot of programming, but not a lot in C. So even the noob stuff still bites me.

 

[WobblyHand]

Spent a bit of time reformatting, simply to get to 80 characters wide, if I need to print stuff. Print formatting seems to be a lost art these days, and it seems the old tricks I did no longer seem to work.

But going through everything just for formatting sort of gives you a fresh eye, and you see things. Like stale unused variables, deprecated methods, and some occasional errors. So I fixed (most!) of them one file at a time. Did the fixes, then recompiled, then did a commit, a file at a time. Went through all 12 files, one after the next. Boring, but unfortunately necessary.

Then added some logging, mostly a short descriptor of where I was and a timestamp. I want to make sure that part passes sanity. This shouldn't be a big burden. I'll run that to get a baseline. I kept the existing variable logging active. At the moment, there is no logging during interrupts, so this is all "main loop" plus a dive into threadingtostop(). Basically just telling me what time it is when I get to a part of the code. If it seems reasonable, then move on to the next step. Not expecting a surprise, but logging will tell me the non-interrupt execution of the code.

Then I intend to get "more sophisticated" with collecting data. At that point, I need to pickup some HW. I had thought the tariff had doubled the cost of components, but that's not what DigiKey is stating (as of yesterday). (Still at 10%) So I'll order a real FTDI cable, as it is $22 + ~$8 shipping. Then I will have one. Might buy one of the cheapo cables too, just as a test. It would only be another $10, and could be a lifesaver.

 

[WobblyHand]

So your function was void! LOL

I can laugh, because I struggle with C sometimes. For me it's nested If else if stuff. I always end up with a } missing somewhere!
In those situations, all I can do is remind myself to take a breath, and start with the first error in the list. One step at a time. I do a lot of programming, but not a lot in C. So even the noob stuff still bites me.

I am such a poor C programmer. It's astounding to me, that this stuff is running at all.

Done tons of programming in Python, Scipy & Numpy, and before that MatLab. Still program a lot in Python. Did full up radar simulations with it, using multiple cores for acceleration, was able to do some neat stuff with it. Is it as fast as C/C++, no, I've gotten 1.1x, vs C++, but it's over 100x faster to develop in, and that matters a lot in the real world...

If you need speed, prototype in a high level language to prove the concept works, then port to C++. I did that, 2 days to understand client server architecture, 2 days to write client server code in python using TCP/IP over sockets and test it. One day to port to C++. (I still don't know C++, but I programmed both the client and server in Python using OOP, so the conversion was pretty easy.) Done in a week. Was doing 1000 1M point double precision FFT's on the server. It took longer to transport the data (on 1GbEnet) than to compute. (2009). Was network bound, even on a private network. Then bonded two 1 Gb networks together. Still, the data transport was far longer than the computation time. (The server was running 32 cores in parallel for computation, plus 4 cores to shuttle the data to the computation cores.). All that junk, just to synthetically test the susceptibility of "incompatible radars".

 

[WobblyHand]

Laughing out loud. Logging every event is a sure fire method to generate massive files. Or run out of buffer memory in TyCommander. Filled up 200K lines of the buffer in not all that long, like less than a couple minutes. (In less time than to set up the logging event!) Haven't yet figured out how to get TyCommander to write to file... It opens the file, but I don't know how to tell it to WRITE to the file. So nothing got written.

Well, nothing got written for an infinite length file. A 1MB file is written in 1 second! So I have to try a 100MB file to have a chance. That gives me a minute and a half to try to get an error. That's probably not going to work, I've had the errors show up 5 or more minutes later.

The main loop is actually very fast, if I don't write to the display, I get flooded. I go around in 5ms or so, with a dozen time points noted. If I use the display it can take 50ms. So I don't have a decent log yet... I'm guessing this will be 100's of MB, just to get it to the point where it might fail? Oh this is gobs of fun

Maybe, I will have a flag that is set and no logging happens before entering thread to stop? Touching threadtostop button starts logging? Exiting threadtostop mode ends logging? Just thinking about this, I bet I enter and exit this routine a lot, even while the thread to stop display is active. I was hoping to capture that. More cogitation needs to happen.

 

[WobblyHand]

Oh, I was way off the mark, no wonder it's so much data. In the "tight loop" when doing nothing (most of the time) it's 75.26us for the main loop. That's going to fill up a buffer! Roughly 1.17 MB/sec logging just idling, haven't even gotten to the fun part. Yeah, that logging flag will be necessary. Even so, this won't be easy.

 

[ivel03]

Does the help show anything?:

tycmd help monitor

It's also possible that you can redirect the command to a file such as:

tycmd monitor > file.txt

I think having the serial TTL (FTDI) cable will be of great help as you can use any serial console app and log it with that.

Logging can be such a double edged sword - need to log everything to find it but once you log everything how do you find it?!?

 

[WobblyHand]

Does the help show anything?:


It's also possible that you can redirect the command to a file such as:


I think having the serial TTL (FTDI) cable will be of great help as you can use any serial console app and log it with that.

Logging can be such a double edged sword - need to log everything to find it but once you log everything how do you find it?!?

Haven't tried the command line version. The graphics version has no help. It links to MacOS help, which isn't useful! I went to the author's github, there wasn't much there either. (Haven't installed tycmd, it seems, just TyCommander.). At the moment, I can simply log 99.9MB files using TyCommander. But that's less than 100 seconds! Got to get gating working.

Logging is usually fun. As you said, a double edged sword. At work we had to build up infrastructure (huge server farms, disk farms, and extremely fast networks) to deal with the data, and to transport it. We had typical data logs of 4TB, which were being generated 100 times per week, so 400TB/week, and it all had to be analyzed. There was a whole analysis team, and there was a group of 8 people who worked on the programs to pre-parse the data to interesting events. Those 8 people were highly paid, as they saved the company 1000's of analysts. They were wizards in big data. Oh, the disk farm ended up being 1ExaByte, that's 1024 PetaBytes. It likely got expanded, but I had left by then (2014). In the disk farm it was all Infiniband stuff at 40 Gbps (at the time).

Fortunately I don't have to deal with such large stuff. Nonetheless, logs get big VERY fast unless managed. In the beginning, you don't know what you are looking for, so you tend to log everything - then you get beaten back by the reality of huge files.

I'm still struggling with whether FTDI will help or not. 3 Mbps is a low data rate. 1.17MB/sec is 11.7 Mbps at 10 bits/byte. So my idle data rate is greater than the entire bandwidth of FTDI. Yeah, I can send binary, and save 60% of that. But that's still ~ 7 Mbps, or more than twice the rate of my idle data. It's hard getting the data rate down just to fit in the data pipe. Another way of saying the problem gets big fast.

I guess it's all about the time of interrupt blanking... I need to know more about that. Seems like USB is a big greedy. Wonder if Ethernet is ok.

I wrote a python program that delivered 786Mbps on 1Gb Ethernet, which is the full data payload capacity of a 1 Gbps link using TCP/IP. I'd guess that it's possible to achieve 78.6 Mbps on a 100 Mbps Ethernet link. (Teensy has 100 Mbps Ethernet) I have that code somewhere, and the C++ version of it. I don't want to do this, if I don't have to...