What's the largest historical data file you've been able to import into AmiBroker?

[theorist]

Smaller historical tick data (HTD) files of ~1 GB aren't a problem for my AB setup, but I'm unable to import a large HTD file, and I'm wondering if it's a limitation of my setup or of AB x64 (which advertises itself as being able to use up to 1 TB of RAM):


I tried importing a 9.2 GB HTD file (369 million lines of data) into AB x64 in Windows 7 Enterprise (running natively on a Mac in Bootcamp). The computer has 16 GB of 1600 MHz DDR3 RAM. Processor is a 2.4/3.5 (turbo) GHz quad-core Core i7 (late 2011). During import I continuously monitored Windows Task Manager; total system RAM use peaked at 13.6 GB, well below the 16 GB capacity. Yet I got an "out of memory" exception handling error from AmiBroker, forcing me to close the program.


So I'm wondering if anyone else has been able to successfully import files of this size and, if so, what's your computer setup?

Thanks!

[theorist]

update: I believe I have a solution. I've found AB's RAM requirements are about 10x the size of the historical tick data file you are using. Importing the file requires ~3x the file size of RAM, and working with the file, inserting indicators, etc., ups that to at least 7x the file size (for instance, with a 2.3 GB file, I found that importing causes the program to use 7.3 GB RAM, and inserting an indicator ups that to ~15 GB). So, to ensure you have enough headroom, I'd estimate you'd want to have available RAM of ~10x the file size. In my case, that limits me to files of ~1.5 GB (my system uses 1 GB, leaving 15 GB available for AB).

[trash]

I'm not sure but there could be a difference between importing large sized files and running large sized data bases.

So you could probably run a multiple GB database but can only import a specific file size depending on your CPU cache(?). So in your case you can only import multiple 1.5 GB CSV files to run a multiple GB data base. Have you asked support for guidance?

Have you tried to set in-memory cache to 11 in preferences-data?

[theorist]

Hi trash -- nice to see you here as well!

Yes, I did contact support, and they're looking into it. [Actually, I first posted here, but then heard something from support, at which time I thought I would take this thread down and wait to hear what support said, but once a thread is up here it can't be deleted.]

My on-chip cache is only 6 MB, which is tiny relative to both a, say, 0.5 GB and a 10 GB import file. So I would be surprised if the cache size is relevant here (though I haven't tried setting in-memory cache to 11 -- wonder if that's a Spinal Tap reference ). And with respect to import vs. creating database, I don't see that distinction, since by import I mean the process of creating the database and displaying the chart. I.e., what actually happens is that, when I import a file of size L GB, AmiBroker draws upon increasing amounts of RAM, finally reaching 3.2 L when that process is complete, at which time it displays a tick-based price chart, and stays at 3.2 L RAM use unless I change the display. Any indicator use adds RAM requirements on top of this baseline. Specifically, here's what I've found (MD = "memory demand" from RAM, for AmiBroker alone):

small file (L = 2.3 GB):
MD for import (results in tick-based chart): 7.3 GB (3.2 L)
peak MD when dragging DEMA indicator into new pane: ~ 14 GB (6 L)
MD after switching from tick to minute/daily/weekly display: 3.6 GB (1.6 L)
MD during scan and backtest of "Formula 1" from http://www.amibroker.com/docs/MTFIndicators.html, modified with Buy/Sell signals: no change (scan and backtest were both very quick -- maybe 10 seconds)
MD when tried to insert plot from "Formula 2" of above: on the first try it caused the program to crash, probably b/c memory was exceeded; on the second try it worked, but MD was 15.3 GB (6.6 L), causing total system RAM use to max out at 16 GB. [Based on this, I suspect you'd want ~ 10 L of available RAM to ensure you have enough headroom.]


medium file (L = 4.5 GB):
MD for import (results in tick-based chart): 14.5 GB (3.2 L)
MD when switched from tick to minute/daily/weekly display: 7.3 GB (1.6 L)
MD when tried to create a new chart: program crashed because memory exceeded

So you can see it's not that I'm limited to importing only a 1.5 GB file -- since 5 GB * 3.2 = 15.6 GB, I could probably get away with a 5 GB file -- but I wouldn't be able to run any indicators on it, because that would require a minimum additional ~15 GB of RAM.

[trash]

Theo, the largest contiguous memory block on 64-bit Windows systems is 2GB. That's probably the reason of your import issues.
As for your other calculations let's wait what Tomasz will reply to you. I guess it's because you have more than one opened pane. So each pane needs same amount of memory. It's all math.

And as for your 1TB ... well, you can't expect to run a 1TB data base on a system having just 16Gb RAM as yours. How do you wanna do that?

[theorist]

Regarding the 1 TB -- I think you've misunderstood me. What I meant (in my first post) was: Given that AB can address 1 TB, why isn't it addressing all of my 16 GB of RAM (when I tried to import the 9.2 GB file, the program crashed before memory maxed out)? But, as I explained in my second post, I found AB could in fact address all the available RAM (when I tried importing the 4.5 and 2.3 GB files), so that wasn't the issue. Probably what was happening was that it got to 13.6 GB, and then asked for the next block. If block requests for are 2 GB, as you mentioned, then that would have exceeded the memory (13.6+2+1 >16; 1 is for system), and thus explained why it crashed before maxing out memory.

I'll let you know what Tomasz says.

[trash]

Here is link regarding performance http://www.amibroker.com/guide/x_performance.html

[theorist]

Thanks for the link. I suspect that, when they said to reduce max symbols to 11 for large (> 2GB) data files, what they had in mind were files that are large because the had, say, minute data on thousands of symbols; in such cases, the files could be easily divided to deal with just 11 symbols at a time.

However, in my case, the file is large because I have a few years of tick data on a single symbol. And, at most, I would create one or two additional artificial symbols using AddToComposite. So, while there certainly could be something about the internals of the program I'm misunderstanding, it seems that reducing max symbols to 11 should not make a difference.

Nevertheless, I tried reducing max symbols to 11, and importing the same 2.3 GB file examined above. Memory use on import was the same (7.3 GB). I did see some different behavior, in the sense that some things were better, and others worse (I managed to exceed memory and crash the program). But my best guess is that this difference from last time is probably more likely because I didn't exactly replicate what I did before, than that changing max symbols to 11 had any effect.