It looks like the issue is related to the heap size which is automatically set since version 6.0.
Since NetBeans version 6.0, the default limit for heap size (-J-Xmx) is determined automatically, with respect to the amount of memory available on the system. There is no -J-Xmx option specified in netbeans.conf. However, if you specify the heap size limit (i.e. you add the -J-Xmx... option to netbeans.conf), then the limit given by you will be respected.
So I ended up increasing the heap size in netbeans.conf by adding the "-J-Xmx2048m" option. Here are my current settings:
netbeans_default_options="-J-client -J-Xss2m -J-Xms32m -J-Xmx2048m -J-XX:PermSize=32m -J-XX:MaxPermSize=1024m -J-Dapple.laf.useScreenMenuBar=true -J-Dsun.java2d.noddraw=true"
Although code completion is sometimes slow, my IDE never hangs and completes initial scanning within 2-3 hours.