Dataset size limitations?
Home › Forums › PATN and Pattern Analysis › PATN Discussion › Dataset size limitations?
- This topic is empty.
- August 12, 2004 at 3:07 am #387Andrew KMember
Gday, I’m wondering if there’s a size limit to the table you can import for version 3.01? I guess if you’re importing from Excel then there’s the 65 536 row limit, but what about from a .csv file exported from another program?
I can’t seem to see anything about maximum row/column numbers on the website or in the help file.
I’ve left an import of 1.4 million rows x 5 columns running at full processor capacity (P1.8Ghz, 1000 MB RAM) for about 20hrs with no result. I know someone’s probably going to say reduce the dataset or subset it, but I’m interested in the max limits regardless !!
thanksAugust 16, 2004 at 1:58 am #438Andrew KMember
Ok ok !! I got those 1.4 mil records imported and it took 35 hrs (2Ghz CPU). Just wasn’t being patient enough I guess.
The only small issue is that you can’t use the slider in the scroll bar to jump further down the table – you get an error message saying invalid calculation.
Will be interested to see how long a non-hierachical cluster takes now !!August 17, 2004 at 3:19 am #440leeKeymaster
Sorry for the delay in getting back to you. I’ve been camping for three weeks. Mia culpa but I seriously needed a break after 3 months of 18 hour days.
Your comments have been noted and we have addressed speedier data import for huge datasets in soon to be released v3.02 (along with fixing the row label sizing that leon has noted in other BB entry).
God, 35 hours is certainly not acceptable, but 1.4 million records is definitely biggish. The DOS version of PATN was (by comparison with Windows code) was minimal and enabled maximal use of memory. I’m not sure what the limits are on v3+ but I would hope that a few million records could be handled. Once the physical memory is exhausted, disk I/O will certainly slow it down. So if you plan to do this regularly, a few GIG of memory will be a must.
The Excel limit doesn’t relate at all to PATN – there are no fixed limits in PATN.
csv should be the easiest and fastest import method.
We will also address the slider bar issue for 3.02 – which will be released within days I hope. I’ll keep all posted on this given that our auto-update via eSellerate is not as we would have wished. Stay posted on that one.
- You must be logged in to reply to this topic.