Stop PS3 Media Server .MKV Stuttering

Problem

PMS is exceeding the available Java heap memory allocated when PMS starts. Newer versions of PMS handle this error by half-ing the transcode buffer (sometimes leading to skips or crashes depending on the renderer) while older versions just let the video crash.
The heap is the sum of the transcode buffer memory plus all of the overhead memory (the actual PMS application, media library, iTunes support, Radio plugin, etc), and the overhead memory can grow quite a bit if the “extra” features (iTunes, Radio plugin, etc) are enabled.

Solution

Install the latest version of Java and PS3 Media Server. Goto General Configuration and Install as a Windows Service.
Edit file C:\Program Files (x86)\PS3 Media Server\win32\service\wrapper.conf , search the file for the Heap Size and change it from 768 to 1536.

# Maximum Java Heap Size (in MB)
 wrapper.java.maxmemory=768  (change this to 1536)

• 32-bit JVM on a 64-bit machine with 4+ GB of RAM can safely use a heapsize value of 1536
• 32-bit JVM on a 32-bit machine with 4 GB of RAM should at most use a heapsize value of 1200

Goto Transcoding Settings and change these settings:

• Transcode Buffer: 600
• Number of cores: 4
• Video Quality Settings: Keyint=5:vqscale=2:vqmin=3

�

Restart the PS3 Media Server service. The javaw.exe*32 process will now use more memory and your 1080p video should run smoothly.