Memory Usage Optimizations for CPU rendering – Chaos Help Center

This article provides detailed information on how to handle projects with Excessive Memory (RAM) Consumption which might increase render times or force Host Application to crash/freeze.

Additional 3ds Max-specific information on the subject can be found here.
More information regarding GPU Memory(VRAM) Optimizations can be found here.

Overview

Depending on the scene setup, V-Ray might reach high RAM consumption limit due to scene complexity and render settings, which might end up with Unhandled Exception, stalling of the machine or huge render times. Adding more RAM to the machine is always the best option to resolve the issue, but if that is not an option for you, you can use the following techniques to reduce the amount of RAM used.

First thing is to check the Memory Usage from the OS native monitoring tool (Task Manager for PC’s or Activity Monitor for MAC). If the Memory is soaring up to the limit during rendering it's a clear sign there isn't enough memory on the machine. In such a case, you may apply some of the following techniques to optimize Memory Usage.

Memory Tracking functionality of V-Ray is very helpful to determine how much memory is used 
by different parts of the project like textures and geometries and can give insights into 
areas that can be optimized. Currently available for V-Ray for 3ds Max and V-Ray for Maya 
only.

Using Chaos Cloud service

Chaos Cloud is a brilliant option to render projects which can't fit into a local machine's memory. It's a one-click solution that will help you render the scene without investing in additional hardware or losing time to optimize the scene to use less memory.

Using Bucket Rendering

In V-Ray for Sketchup | Rhino | Revit, to enable Bucket mode you need to switch off 
Progressive mode.

V-Ray is equipped with Bucket and Progressive Rendering modes for both Image Sampler and Global Illumination. While Progressive mode has many advantages compared to Bucket, it needs to load all assets into the memory. The Bucket mode on the other hand is much more Memory Efficient as it needs to load only the information required to render the current bucket.

Use Bucket Image Sampler instead of Progressive one. Keep in mind that bucket size is also something you have to consider. A very small size will cause less geometry to be loaded in the Memory. However, that will increase the Render Time due to filtering of the pixels on the Bucket's Edges. Keep in mind that the default size of the buckets is optimized for most cases.
Use Brute Force GI engine instead of Light Cache. Please note that disabling Light Cache will also disable other V-Ray very powerful features like Adaptive Lights and Adaptive Dome Light but it would definitely help optimize Memory Usage geometry being loaded in

Using V-Ray Proxy

V-Ray Proxy is a very powerful tool that is especially useful for optimizing projects with a lot of geometry. They are constructed in a way that small parts of it (as well as the whole proxy) can be loaded and unloaded from the memory on demand without the need to fill the memory with the whole proxy geometry. Proxy geometry will also take less memory than the static original geometry.

As described in the Using Bucket Rendering paragraph above, V-Ray Proxies will not be memory efficient if a scene is being rendered with Progressive Image Sampler and/or Light Cache GI.

Optimizing project geometry

Optimize dynamic geometry. Hair/Fur, Displacement and Subdivision objects tend to consume a lot of memory. Displacement and Subdivisions are also dependable on the image resolution, the higher the resolution is the higher the memory consumption will be. Make sure the geometry generated by these features is with reasonable polygon count otherwise it could fairly easily eat the whole memory.
Remove unnecessary/out-of-camera view objects (If they are not specifically needed). It's not uncommon to have a project populated with lots of unneeded or not-in-camera view objects. Removing those will certainly help reduce the memory usage
Optimize regular geometry. Very often objects with unnecessary dense geometry end up in the project, those are usually assets gathered from external libraries. Optimizing those will definitely help reduce memory usage

Setting Dynamic Memory Limit

N/A for V-Ray for Sketchup | Rhino | Revit.

Dynamic Memory Limit parameter sets RAM limitation for the Dynamic Geometry (generated during Render Time) like Displacement, VRayProxies, VRayFur, etc.
The default value is set to 0, allowing V-Ray to work with as much memory as needed which may be a little dangerous in case there isn't enough RAM available. Setting this parameter to 80-90% of the total RAM might help to render projects with lots of Dynamic Geometry.

Using Tiled Textures

Converting regular textures to Tiled Textures can be done with Image to tiled multiresolution 
EXR converter utility tool which comes with V-Ray installation.

Using the Tiled Textures (.tx or .exr) brings many advantages among which is the great memory savings. Their main characteristics are that they contain smaller versions of the original image for use in geometries far away from the camera. This way, the Full Sized texture might never be loaded in its entirety depending on the proximity of the geometry to the camera, which saves memory. They are also written and read in Buckets so they work well with the Bucket Rendering.

Tiled Textures has a separate option to control maximum cache size or maximum resolution. More information on the subject could be found here: 3ds Max, Maya

Keep in mind that textures undergo the same load/unloading process as proxies as the RAM fills up. With Tiled Textures, this amount gets optimized since V-Ray will load much smaller resolution of the original image for the geometry far from the camera.

In 3ds Max you can choose the mip-map resolutions for the tiled textures using the Blur 
parameter in the coordinates section of the map. We will not suggest using smaller values 
than 1, since they will force higher resolutions to be loaded and possibly not perform any 
filtering. If you decide to change that Blur parameter, please make sure to test it in 
advance, in order to find the best look.

Render Elements might consume a lot of memory

Keep in mind that having many render elements and rendering huge resolutions will affect the memory as well. Keep it tidy and make sure you have only the Render Elements that you really need.

Disable Memory Frame Buffer

N/A for V-Ray for Sketchup | Rhino | Revit.

This technique is very helpful when rendering very large images. Those can be Memory Intensive as very large resolutions require more memory to be added additionally to the scene's own memory needs. By writing the rendering image file directly to disk, the Memory Buffer is disabled, saving on overall memory use. This is very advantageous for machines with limited memory. Check the following few links for more information on how to apply this technique in 3ds Max, Maya and Modo.

Increase Virtual Memory (System Page file) size

Increasing Virtual Memory might also help to complete the rendering in case the required memory is just a little bit above the available one. Be prepared for longer Render Times when using this approach since the Virtual Memory is a lot slower than the RAM Memory.

Export to .vrscene and render with V-Ray Standalone

Host Applications just like V-Ray need some RAM to keep the scene open and to operate with it. Depending on scene complexity Host Application may occupy a few gigabytes of RAM which could be very valuable for V-Ray in case there isn't enough Memory for the Rendering. Exporting the scene to a .vrscene file and Rendering it with V-Ray Standalone will minimize RAM consumption by excluding the Host Application.

Render in separate passes (foreground/background)

This technique is very useful for extremely complex projects where all other Memory Optimization techniques fail to help. Rendering in passes means splitting the scene into separate sub-projects. The most common approach is to render the Foreground and Background separately and then merge both render outputs in a Compositing/Image Editing software.

Close other Memory Intensive applications

It's not unusual to have multiple applications open when working on a project. Although it's very handy to have all the needed apps up and running, this will also reduce the amount of memory available for the rendering process.