VEMS Community Forum

VEMS => Calibration & Mapping => Topic started by: dnb on July 05, 2007, 05:12:47 PM

Title: Data analysis software for mapping based on log files
Post by: dnb on July 05, 2007, 05:12:47 PM
I have a few ideas I'd like to kick around here, and given we're all trying to do basically the same thing then sooner or later we'll all have the same ideas so if these ideas are already coded, then please point me in the right direction!

Alternatively, you can tell me I'm daft and it'll never work. ;) 

I have thought of ways for most of this to be automated, requiring little or no user intervention.  (Details are deliberately a little sketchy here, otherwise I may lead you down certain ways of thinking and I prove myself right by circular argument... ;) )

I'd like to take a log file and bin it according to the maps in the appropriate MSQ file.

This should allow estimates to be made on VE accuracy given certain constraints, application of certain filters and a statistically valid number of samples per bin.  We can then derive and apply to the MSQ a set of map corrections that should be of a much higher quality than autotune could ever manage.

It would also allow an estimate of drift WRT intake temperature (This was a serious problem for me on the TVR last time) and allow the value of the correction parameter to be calculated.

We could look at knock sensor noise data in a similar way.  The first thing is to estimate the divide between MBT based and knock based operation, then apply appropriate corrections to the knock based areas.  Again, there are filters that must be applied to the data.

I can't quite see my way round the next ideas, but I'm sure there is a way :)

If we use the 2ndary trigger to log vehicle road speed (is there a way to do this?) then we can construct a much better lambda map based the position of the main cruise speeds (say 30, 40, 60 and 70) in the map.  We can also look where in the map deltaTPS is appearing.  This can then be correlated to differences between lambda target and actual lambda, given that we know what the EGO correction is up to on average in these cells.  This leads to real-world estimates of corrections for acceleration enrichment.

The addition of vehicle speed in the logs adds even more potential - we can then do meaningful fuel use analysis etc.

Most of this can be done in excel, but it takes many hours work in setting it all up.  It would be much nicer if this could take minutes.
Title: Re: Data analysis software for mapping based on log files
Post by: LT401Vette on December 06, 2007, 02:15:30 AM
I hear you :)
MegaLogViewer does a great job with the VE tables, in the MegaSquirt community MSTweak3000 has long since been put to rest. But there is a whole lot more opportunity. Ultimately it is possible to build an application that can tune most aspects. Ideally with full connectivity even things like true measurement of injector open time is not much work to implement.
This got me going a while back and I set out to write it in Oct 2006, this has eaten into the time that may have otherwise gotten MegaLogViewer even further by now, but this new core platform is going to allow all these things to be realized. I am near an Alpha release, by years end. The initial release will be little more than the next generation MegaTune, MegaTune functionality and compatibility with a richer UI, custom dashboards, wizards, and a plug-in interface for extensions. But once that is stable, the core allows for so much flexibility to build all the automated tuning pieces.

Phil
http://www.efianalytics.com/MegaLogViewer (http://www.efianalytics.com/MegaLogViewer)
Title: Re: Data analysis software for mapping based on log files
Post by: [email protected] on December 06, 2007, 08:39:52 AM
Awesome stuff Phil,  I have to say thanks to you for MegaLog viewer, its helped me out more times than I can count. And I hear nothing but praise for it from the people I advise to use it.

I see that your page has a Buy MegaLogViewer Now! link, is this for the new software? https://www.efianalytics.com/register/register.jsp?appName=MegaLogViewer

What will the plug-ins need to be written in?
Title: Re: Data analysis software for mapping based on log files
Post by: GintsK on December 06, 2007, 10:36:19 AM
I much say big thanks for Phil too! I use this piece of software every day!
with a some hacks in vems3.ini i got VEanalyser working for VEMS too. Not so good as for MS, but for usable 16x14 Alpha-N table 1 hour was enough.
What we must do for full adapting MLV for VEMS?
Title: Re: Data analysis software for mapping based on log files
Post by: dnb on December 06, 2007, 10:55:35 AM
Hi Phil!

I've made quite a bit of progress since I posted the thread.  All of the fuel side is now automated to a reasonable extent, including estimating injector parameters (although I'm not too happy with the answers yet - I need more resolution in the logs for certain areas...)

Next stop is the obvious one - an interface with a dyno to do ignition timing optimisation :)
Title: Re: Data analysis software for mapping based on log files
Post by: LT401Vette on December 06, 2007, 03:30:52 PM
Compatibility should be easy. I just need to see some logs & msq's.

The latest version of MLV has all fields references pulled out into an external properties file, so you are no longer required to use the MegaSquirt field naming convention.

The msq is the other piece, there are still a few references in there to specific bins from MS1 and MS1.

The "Buy MLV" is for MLV, it is shareware now. There is no disabled functionality though, just a popup nag to register. To buy will get a registration to get rid of the nag. The nag is only meant to be a reminder, not much of an annoyance I hope.
Title: Re: Data analysis software for mapping based on log files
Post by: [email protected] on December 10, 2007, 08:55:01 AM
Quote from: LT401Vette on December 06, 2007, 03:30:52 PM
Compatibility should be easy. I just need to see some logs & msq's.

The latest version of MLV has all fields references pulled out into an external properties file, so you are no longer required to use the MegaSquirt field naming convention.

The msq is the other piece, there are still a few references in there to specific bins from MS1 and MS1.

The "Buy MLV" is for MLV, it is shareware now. There is no disabled functionality though, just a popup nag to register. To buy will get a registration to get rid of the nag. The nag is only meant to be a reminder, not much of an annoyance I hope.

Here's a log and MSQ for your perusal, let me know if you need more:
http://koabi.lazyslacker.com/VEMS/logs/20071201megasquirt.msq
http://koabi.lazyslacker.com/VEMS/logs/20071201datalog.xls

Shareware nags are fine by me, I was wodnering if you were planning on making the license mandatory at any point?
Title: Re: Data analysis software for mapping based on log files
Post by: LT401Vette on December 10, 2007, 03:23:08 PM
Thanks, those will help. Do you know if there is much variance with different firmwares? Actually is it common to have various firmwares for VEMS, or is a single strain that just moves forward?

I don't have any plans to make the registration mandatory. I would guess that would narrow adoption anyway, many people would probably stay on the earlier and it would sort of be hanging out there with no support. The people paying are typically just doing it to show their appreciating and return the favor more than to get rid of the nag. That is why I mean for the nag to be more of a reminder than a pita.

However with that said, I do see some new larger features in the future that may not be enabled without a registration.
Title: Re: Data analysis software for mapping based on log files
Post by: [email protected] on December 10, 2007, 04:20:20 PM
The firmwares tend to build on each other - the output from the datalog stays pretty consitent.  The .ini file for MegaTune changes quite a bit inbetween firmware releases - mainly so that we can put more fields in, and remove the redundant ones.  I imagine that

I see no reason why you couldnt charge for the "pro" features.  The bigest complaint I get from tuners is the poor keyboard support - the ability to select a group and either enter a number directly from the keyboard or use the arrow keys.  I did some work on a table editor:

http://www.vems.co.uk/VEMSTech/table.zip

Cheers Rob
Title: Re: Data analysis software for mapping based on log files
Post by: LT401Vette on December 10, 2007, 04:29:05 PM
That works pretty good.

Thanks, that is good feed back. I was actually putting better keyboard in, but more for menu functions. The table editors would really benefit too. Expect that to be in an upcoming version.
Title: Re: Data analysis software for mapping based on log files
Post by: [email protected] on December 10, 2007, 04:44:07 PM
Here's the keys:
    *  Ctrl-L allows editing of the load table.
    * Ctrl-R allows RPM table editing.
    * Ctrl-C copies the content of the selected cell.
    * Ctrl-V pastes the copy into the selected cell or group of cells.
    * F9 changes to VE table.
    * 10 changes to Ignition table.
    * 11 changes to Lambda table.
    * Select a group of cells in the main table:
    * Page Up to increase the cells' value.
    * Page Dn to decrease the cells' value.
    * Type a numeric value to set the selected cell's
    * Bounds checking for keyed entry (0-255 for VE, xx.(25|50|75) for ignition, 0.7->1.x for lambda) Lambda does not scale correctly - please advise
    * Small increment with q|+|Pg Up
    * Small decrement with w|-|Pg Dn
    * Large increment with e|Ctrl +|Ctrl Pg Up
    * Large decrement with r|Ctrl -|Ctrl Pg Dn
    * Paint value - Ctrl Up, Down, Left, Right
Title: Re: Data analysis software for mapping based on log files
Post by: cliffb75 on December 10, 2007, 07:10:25 PM
Something I found useful in another aftermarket system was the ability to lock cells that you have mapped and then interpolate between them. It makes getting the intial shape of a maps quite quick and easy, and avoids accidentally adjusting a 'real' value

Also, for group selection, typing a fixed value is useful, and so is adding/subtracting an offset(ofter useful for ignition or boost related tables) and multiplying/dividing (for fuel tables).

Title: Re: Data analysis software for mapping based on log files
Post by: gunni on September 27, 2008, 10:34:52 PM
To update on this.

assuming your running normal petrol I found that using in Megalogviewer to use VE_analyzer with Vems.
Wideband O2-AFR should be "Dynojet linear"

that will result in Lambda 1 = AFR(WBO2) to be 14.7
and will thus allow the VE_analyzer to be used,

When I tested I had it disregard MAP below 35kpa which is sort of overrun and Vems can show some very high numbers in the lambda field  that will mess with the results. Running it over a very recent log it showed exactly the results I expected.

Tomorrow IÃ,´m changing
Inj open time @ 13.2v (us) to 0 from the 1008 it is now, which is wrong since I have a diode in place.
So using the VE Analyzer IÃ,´m expecting fuel to be sorted in less then an hour.

Title: Re: Data analysis software for mapping based on log files
Post by: LT401Vette on September 30, 2008, 10:55:12 PM
Gunni,
Thanks for bringing this thread back to life. Somehow it completely fell off my radar!
I think I have just been too heads down with TunerStudio.

So is the WB sensor consistent with VEMS? It is always the same and matches the Dynojet?
Title: Re: Data analysis software for mapping based on log files
Post by: gunni on September 30, 2008, 11:10:28 PM
Log files have Lambda values directly in them and the wideband is integrated in the VEMS no outside voltage stuff.

It would be best if the VE analyzer could just use the "lambda" logged value as well as reading the msq lambda table.
As well, some vems log it as "lambda" while other version "Lambda" .

Title: Re: Data analysis software for mapping based on log files
Post by: GintsK on September 30, 2008, 11:21:03 PM
I use MLV for a while. Including lambda tables and VE analyzer. My hack was rename lambda table to afrtable inside vemsv3.ini.
Awesome software for VE building!

Gints
Title: Re: Data analysis software for mapping based on log files
Post by: LT401Vette on October 01, 2008, 02:05:14 AM
GintsK,
Good hacking..
Sorry I'm not sure how this Thread fell off my radar. I'll get VEMS support into MLV next build.

Title: Re: Data analysis software for mapping based on log files
Post by: LT401Vette on October 01, 2008, 06:58:03 AM
Try a Check for updates from the help menu.
0.952 should make VE Analyze work with the VEMS logs and msq's.

I also put in a filter to take out those crazy high lambda log records. So VE Analyze won't ever see Lambda as over 2.0

Your lambda tables are still going to be converted to AFR. In MLV 3.0 I am going to change that, because I really think it should always be working with Lambda, but if a user prefers to view in afr, well that should be the view. Inside Lambda makes more sense, but right now it is using AFR inside.
Title: Re: Data analysis software for mapping based on log files
Post by: GintsK on October 01, 2008, 09:48:48 AM
Phil, thanks for being here!!!

I have some questions about MLV:
When tuning n/a engines with race camshafts it is usual VE table becomes with waves. Sometimes VE changes a lot in narrow rpm range. It looks to me that MLV does some averaging and do not allow beside values be with big difference. Even when RPM bins is chosen very close to real breakpoints and engine works exact at this points during logging. How to solve this?

VEMS can use 16x14 tables. MLV for some reason can't export this tables. It give message:
QuoteError saving VEX file:
Unable to write C:\....\yyyy.vex
15
Why so?
Instead it allow save whole msq. Looks without errors!

16x14 VEX looks so:
EVEME 1.0
UserRev: 1.00
UserComment:
Date: 10-01-2008
Time: 11:47
Page 0
VE Table RPM Range              [16]
   [  0] =  10
   [  1] =  13
   [  2] =  17
   [  3] =  21
   [  4] =  26
   [  5] =  31
   [  6] =  36
   [  7] =  40
   [  8] =  43
   [  9] =  45
   [ 10] =  48
   [ 11] =  53
   [ 12] =  62
   [ 13] =  69
   [ 14] =  76
   [ 15] =  84
VE Table Load Range (MAP)       [14]
   [  0] =   0
   [  1] =  10
   [  2] =  20
   [  3] =  30
   [  4] =  45
   [  5] =  65
   [  6] =  90
   [  7] = 115
   [  8] = 140
   [  9] = 165
   [ 10] = 185
   [ 11] = 210
   [ 12] = 230
   [ 13] = 255
VE Table                        [ 16][ 14]
           [  0] [  1] [  2] [  3] [  4] [  5] [  6] [  7] [  8] [  9] [ 10] [ 11] [ 12] [ 13] [ 14] [ 15]
   [  0] =   33    26    25    24    23    42    34    31    28    32    35    26    13     5     1     1
   [  1] =   37    30    27    29    32    36    32    31    32    31    28    23    16    13     9     7
   [  2] =   47    40    33    32    31    26    26    27    24    23    23    20    15    14    13    13
   [  3] =   55    46    39    40    36    30    28    26    25    23    23    23    22    19    18    17
   [  4] =   57    54    47    47    42    39    36    38    34    35    33    33    31    27    27    24
   [  5] =   65    62    59    58    51    48    45    49    44    43    43    42    38    31    29    31
   [  6] =   84    72    71    67    60    55    50    52    57    54    55    61    55    34    33    38
   [  7] =   92    84    78    82    69    65    62    63    67    71    71    74    71    48    45    44
   [  8] =   92    88    83    86    85    77    80    85    89    88    91    89    82    74    69    62
   [  9] =   95    98   102    98    99   106   101   105   121   128   144   138   115   111    96    80
   [ 10] =   95   101   107   108   109   107   104   115   143   153   173   164   142   127   118   100
   [ 11] =   95   100   105   111   111   103   104   112   154   172   188   181   171   158   140   119
   [ 12] =   97   103   111   114   117   106   104   130   161   179   207   201   179   173   142   123
   [ 13] =   94   103   114   114   113   103   106   133   167   183   208   203   179   173   144   126


Regards
Gints

Title: Re: Data analysis software for mapping based on log files
Post by: LT401Vette on October 01, 2008, 03:12:38 PM
The msq saves and Vex saves are completely different and I the msq's are much more adaptable by nature being xml. The Vex format is terrible, it doesn't have a well formed structure. I was just finally implementing Vex import for compatibility, but I really am looking to have MLV and TunerStudio default table export to a new .table format that is xml based.

But in short, the msq saves should be good. The 16x14 Vex export should also work though. I'm guessing the only problem is that I've never tested it :). Send me an msq with a 16x14 table and i will fix it.

The way VE analyze works it does pull values closer together if it the RPM & MAP is hitting anywhere in between. If you are dead on a cell, then only that cell will get changed, but typically the input VE value is interpolated from 4 cells. The when the correction is applied, it is applied to all the cells that provided input with a weighting that corresponds to the strength of that cells voice.

There are actually 2 ways each record produces a recommended
1) The interpolated value is corrected and the corrected value is applied to all for input cells with weighting.
2) The correction percentage is calculated and that percentage is applied to the value of all 4 input cells with a weighting.

So, #1 helps to smooth the table, #2 prevents edge cells from getting way off base. An Edge cell being one that only has input data on one side. Like the top row may be 100, but it always ends up getting input from 95 kPa, so it doesn't know that at 98kPa more fuel is needed.

It sounds like #1 is what is getting you. But I think I may have fixed your problem in this last release. I noticed that the VEMS logs let the Lambda go wild. There is Lambda of 60 in there. That is an AFR of 882:1, so naturally even if any input from that record is of very small weight, it is REALLY lean :). So it was quickly sending some of the cells to crazy high numbers.

In the new version 2.952, VE Analyze takes any Lambda greater than 2 as 2.0. I even considered making that threshold lower, like 1.6, but see how it works here.

I have not put together an installer for 2.952 yet, first i figure we'll see how this one works. I'm expecting a few more tweaks. :)
Title: Re: Data analysis software for mapping based on log files
Post by: GintsK on October 01, 2008, 03:48:48 PM
Hi!

I upload project files here:
http://rapidshare.com/files/149979529/BMW1800.rar.html

There is datalog and related msq too. 16x14. alpha-N.

I hope it is downloadable.

Gints
Title: Re: Data analysis software for mapping based on log files
Post by: LT401Vette on October 01, 2008, 04:28:04 PM
That msq appears to be 12x12
Title: Re: Data analysis software for mapping based on log files
Post by: GintsK on October 01, 2008, 08:28:17 PM
Sorry!
At start I tune with 12x12 later migrate to 16x14.
It must 16x14:
http://rapidshare.com/files/150053490/BMW1800.rar.html

Gints
Title: Re: Data analysis software for mapping based on log files
Post by: LT401Vette on October 01, 2008, 10:31:06 PM
Gints,
That was pretty simple, the array was being sized backwards 14x16 vs 16x14.
I guess most tables are square so it never broke it before.

It is fixed. I'll put another update out tonight.
Title: Re: Data analysis software for mapping based on log files
Post by: jrussell on October 02, 2008, 02:40:59 AM
Thanks Phil. I updated MLV tonight and it so much nicer not having to open up Excel and mess with the log file first.
Title: Re: Data analysis software for mapping based on log files
Post by: Karl Buchka on October 25, 2008, 03:51:47 AM
Phil,

Many thanks for adding native VEMS support. Coming from a Megasquirt background it literally cuts the software learning curve to zero.