Since version 5.1 the deduplication system is using new database technology and this new database is making use of RAM and swap space much more agressive and this can potentially lead to the operating system to take protective measures.
We recommend you upgrade to 5.1.11 or later
If not already done you should set the db.block_index_cache_max_size value to 50% of the available RAM. Values below 8Gb will not yield acceptable performance.
Setting or modifying the value can be done through the admin GUI under server preferences or via ade_admin.
get db.block_index_cache_max_size to see current setting
set db.block_index_cache_max_size=32G to set it to 32GB
set httpd.keepalive = 1
set httpd.nb_thread =256
Make sure the diskspace available for swap (pagefile) is equal to 1,5 or 2 x the size of RAM