Sign in to follow this  
Followers 0
Xjph

Option to enable NTFS compression

11 posts in this topic

This should be fairly straightforward.  It would allow people to cram a little bit more into any given RAM disk at the cost of some CPU overhead.

 

Access times might be slightly slower, though still far far better than a physical drive, and the gains can be enough in some cases to recover a respectable amount of space, as shown here:

DDCompress.png

 

Note the "size on disk" being notably smaller than the "size" of the data.

Share this post


Link to post
Share on other sites

Isn't there a some specific instructions supporting compression and decompression in AMD and Intel CPUs? Might there be an API out there that would facilitate the implementation of such a feature? Instead of using the NTFS.

Share this post


Link to post
Share on other sites

There actually is, but it is more about how Windows handles the compression.  It is pretty easy to test!  Do this:

 

  1. Create a Dimmdrive, say  5GB
  2. Enable compression on that by clicking on the drive in Windows Explorer
  3. Copy about 4 GB of random files to it
  4. Open up a command prompt and do a "dir"  every 10 seconds.  Notice how the drive size shrinks, a little at a time?

 

If you do that "dir" every 10 seconds, you'll end up doing it about 10 minutes before it stops compressing.  It's very, very slow.

 

I'm definitely going to look into this more in the future, as the concept is pretty cool of taking 8GB ramdrive and making it, say, 10GB.

Share this post


Link to post
Share on other sites

Compression tends to be a performance killer.  Deduplication may be a better way of improving capacity utilization without killing performance, but that would have to be implemented separately from NTFS.  At the moment, NTFS deduplication is only natively supported on Server 2012 platforms, and I'm guessing there aren't a heck of a lot of people gaming on Server 2012.

 

Deduplication runs are generally scheduled and don't operate in real time due to the memory constraints (and, thus, performance penalty) imposed by the deduplication table.  That would likely not be feasible with a product of this nature since data isn't going to be sitting on a RAM drive very long.  Deduplication would thus have to be real-time (or near real-time) to provide a benefit.  Since deduplication tends to be RAM-intensive, this might cause an issue with a RAM drive that is already borrowing a good chunk of RAM from the OS.

 

That said, if you're ever feeling experimental, I'd love to play with a RAM drive with deduplication options.  It could be a killer product in the right system.

Share this post


Link to post
Share on other sites

i hadn't heard of deduplication before, been doing some light reading, here's a quote that seems to sum up my biggest concern with it:

 

 

Deduplication

 

While deduplication takes potentially the same amount of processing power, it by comparison can be more memory intensive than compression because the comparable range is now “across all the files†(in fact segments of those files) in the environment. Just like compression, it is ideal to do these comparisons in memory so the latency of hard disk based storage does not slow down the process of confirming a unique or redundant segment of data. 

 

Many deduplication technologies report needing 16GB of storage for every 1TB of data being managed. That means a 10TB storage system could require 160GB of RAM! 

 

also I'm reading it's even more CPU intensive than compression, but this may be on systems with less ram, unsure.

 

it's an interesting idea, but I would imagine it taking a long time to go through each block of data and cross check for duplication, index and rewrite it all.

Share this post


Link to post
Share on other sites

Either way, I do think this should be looked into.  It would be a very interesting feature, especially on systems with some spare CPU cycles.  I won't look into it right away, but I definitely want to in the near future.

 

For example, if sacrificing 5% cpu and 1500 MB/s to get a ramdisk that is 30% more in size, heck, I think that'd be worth it!

Share this post


Link to post
Share on other sites

To deduplicate you could create a hash of the file in a hashtable, for each file being copied check if the hash exists and compare the files if identical deduplicate by symlink ;-)

For large files you could use more heuristic scanning, but that would require much more effort and processing.

Share this post


Link to post
Share on other sites

this would have to be done when the ram drive is being created but he is correct, it might give you more space but it would depend on how much duplicate data the game actually uses, more likely the compression will give greater benifit if you have a lot of extra cpu cycles. (such as mine since I have 8 cores and most games don't touch beyond 4 most only use 2)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0