This tutorial was written back in the Windows XP era, but the instructions are still applicable. http://www.duckware.com/pmvr/howtoincreaseappletmemory.html
Make sure you modify the “Java Application Runtime Settings,” not the Applet settings shown in the picture.
Add ‘-Xmx300m’ to the “Runtime Parameters” box. Replace 300 with whatever amount of RAM you wish to allocate; but make sure you don’t go overboard (eg >1024) or else your performance will seriously suffer.