java - How to load JTable with 2 million rows -

i writing application uses jtable display lines of log file. have data parsed, when try add rows abstracttablemodel receive either "gc overhead limit exceeded" or "java.lang.outofmemoryerror: java heap space"error. there configure garbage collector or change abstracttablemodel allow me load needed rows?

package gui;  import java.util.arraylist; import java.util.arrays; import java.util.date; import java.util.list; import javax.swing.table.abstracttablemodel; import saxxmlparse.logevent;  /**  *  * @author david.crosser  */ public class mytablemodel extends abstracttablemodel {      private string[] columnnames = new string[]{"type", "time", "tid", "lid", "user", "message", "query", "protocol", "port", "ip", "error"};     private list<logevent> data;      public mytablemodel() {         data = new arraylist<>(25);     }      @override     public class<?> getcolumnclass(int columnindex) {         if (columnindex == 1) {             //return date.class;                         return string.class;          } else {             return string.class;         }     }      @override     public string getcolumnname(int col) {         return columnnames[col];     }      @override     public int getcolumncount() {         return columnnames.length;     }      @override     public int getrowcount() {         return data.size();     }      @override     public object getvalueat(int row, int col) {         logevent value = data.get(row);         object retobj=null;         switch (col) {             case 0:                 retobj = value.gettype();                 break;             case 1:                 retobj = value.gettime();                 break;             case 2:                 retobj = value.gettid();                 break;                 case 3:                 retobj = value.getlid();                 break;                 case 4:                 retobj = value.getuser();                 break;                  case 5:                 retobj = value.getmsg();                 break;                  case 6:                 retobj = value.getquery();                 break;                   case 7:                 retobj = value.getprotocol();                 break;                  case 8:                 retobj = value.getport();                 break;                  case 9:                 retobj = value.getip();                 break;                  case 10:                 retobj = "n";                 break;         }         return retobj;     }      public void addrow(logevent value) {         int rowcount = getrowcount();         data.add(value);         firetablerowsinserted(rowcount, rowcount);     }      public void addrows(logevent... value) {         addrows(arrays.aslist(value));     }      public void addrows(list<logevent> rows) {         int rowcount = getrowcount();         data.addall(rows);         firetablerowsinserted(rowcount, getrowcount() - 1);     } }      package gui;  import java.sql.resultset; import java.util.list; import javax.swing.swingworker; import saxxmlparse.logevent;  /**  *  * @author david.crosser  */ public class tableswingworker extends swingworker<mytablemodel, logevent> {      private final mytablemodel tablemodel;     string query;     database.database db;     int totalrows=0;      public tableswingworker(database.database db, mytablemodel tablemodel, string query) {          this.tablemodel = tablemodel;         this.query = query;         this.db = db;     }      @override     protected mytablemodel doinbackground() throws exception {          // deliberate pause allow ui time render         thread.sleep(2000);          resultset rs = db.querytable(query);          system.out.println("start polulating");          while (rs.next()) {              logevent data = new logevent();              (int = 0; <= tablemodel.getcolumncount(); i++) {                 switch (i) {                     case 0:                         data.settype((string)rs.getobject(i+1));                         break;                     case 1:                         data.settime((string)rs.getobject(i+1));                         break;                     case 2:                         data.settid((string)rs.getobject(i+1));                         break;                     case 3:                         data.setlid((string)rs.getobject(i+1));                         break;                     case 4:                         data.setuser((string)rs.getobject(i+1));                         break;                     case 5:                         data.setmsg((string)rs.getobject(i+1));                         break;                     case 6:                         data.setquery((string)rs.getobject(i+1));                         break;                     case 7:                         data.setprotocol((string)rs.getobject(i+1));                         break;                     case 8:                         data.setport((string)rs.getobject(i+1));                         break;                     case 9:                         data.setip((string)rs.getobject(i+1));                         break;                     case 10:                         data.seterror((string)rs.getobject(i+1));                         break;                 }             }             publish(data);              thread.yield();         }         return tablemodel;     }      @override     protected void process(list<logevent> chunks) {         totalrows += chunks.size();         system.out.println("adding " + chunks.size() + " rows --- total rows:" + totalrows);         tablemodel.addrows(chunks);     } } 

my answer applicable general type of problem need work on large data set, not specific "2 million rows in table" problem.

when have problem data need operate on larger container (in case, more memory system physically has, can apply data larger container - physical, virtual, logical, or otherwise), need create mechanism streaming data need @ given time, , possibly more if want buffer.

for example, if want able show 10 rows in table, , data set way large, need create table model knows 10 rows being displayed, , have swap data out needs when view changes. create table model holds 10 records, not 2 million. or optional buffering mentioned, make model hold 30 records; 10 records in view, , 10 before , after can change data user scrolls table small scrollbar increments highly responsive - problem of streaming data on fly apparent when user scrolls far fast (ie: click scrollbar "thumb" , drag top bottom).

this same way compression algorithms compress/decompress 100gb of data; that's not in memory @ once. or how software backed tape drives work (they have no choice since it's not random-access). or, example familiar with: how online video streaming works. think of youtube , loading bar @ bottom of video grey buffer zone; if "fast-forward" time in buffer zone switches immediately, if change time past video might stop second while loads next frame (and buffers more). how enormous table works too, except "streaming" data model either memory or disk, stream source , destination being in same process. otherwise, same idea.