Sign in to follow this  
Followers 0
cinoaz

DIMMDRIVE to SSD workaround

12 posts in this topic

First,

 

Pay for this program and support the author, he's doing great work !!

 

No, seriously, pay for the program!!

 

Second,

 

This isn't some "Great Find" nor am I saying this is the best thing since sliced bread.  It's a workaround until DimmDrive supports it natively.  I also was curious on learning about symlinks and wanted to try my hand at it.  It's not a slick GUI implementation but if you wish to learn, like I did, keep reading.

 

I'm using Batch files to accomplish this, it has nothing to do with DIMMDRIVE except it's doing the exact same thing DD is doing, just without the slick interface, the easy to use sliders, the great support (you get the picture).

 

Why do this?

 

DIMMDRIVE we all know creates a psuedo hard drive from unused system memory, copies the game files to that newly created drive and then creates the symlink to it.  This allows games to run from RAM for an increase in performance.  Cool, we all get it.

 

However, many games are very large and when presented with economy of scale it becomes extremely cost prohibitive to increase system memory to the necessary space. DD solution is to uniquely identify certain files only.  That get's pretty tedious once you've done a game or two.   SSDs, however, are flash (not as fast as System Memory) but still much faster than spindle drives.  The problem is, finding a 4TB SSD to install all your Steam games to is again cost inefficient to say the least.  Trying to manage install/unintalls across different drives is a pain as well. 

 

Now, 120GB SSDs are pretty cheap ($60 if you keep your eyes open for deals).  So, the thought is, install all your games to your 4TB 7200RPM drive.  When you want to play a game, copy the files from the 4TB drove to a blank 120GB Flash Drive, create the symlinks and have the game run off of that.  You see a tremendous increase in performance and you don't have to worry about moving installations around.  Hence my project.

 

It provides an increase performance in games using hardware that is extremely cost friendly.  In other words, it leverages the cheapest, fastest solution for everyday use.

 

How it works:

 

I used DimmDrives process as the template:

 

1. Copy the files from the STEAM installation folder(4TB drive) to your SSD drive(120GB)

2. Rename the original game folder on the 4TB drive

3. Create the symlinks to point to the SSD Drive

4. Launch the Game via STEAM.  Steam is pointing to the 4TB drive, which then redirects the requests to the SSD drive...bingo.

5. Play the game, rage quit...

6. Compare the files on the 120GB drive to the original source files on the 4TB directory.  Certain files change while playing (Saved games and such) that need to be captured.  Copy changed files from the SSD back to the 4TB drive.

7. Delete the symlinks

8. Rename the 4TB directory back to the original name

9. Delete contents from SSD Drive.

 

All this is done in a BATCH file.  Code example for DOTA 2 below:

rmdir "C:\DIMMDRIVE_SSD" /S /Q

mkdir "C:\DIMMDRIVE_SSD"

xcopy "D:\Program Files (X86)\Steam\steamapps\common\dota 2 beta" "C:\DIMMDRIVE_SSD" /E

rename "D:\Program Files (X86)\Steam\steamapps\common\dota 2 beta" "dota 2 beta prelink"

mklink /D "D:\Program Files (X86)\Steam\steamapps\common\dota 2 beta" C:\DIMMDRIVE_SSD

pause

"D:\Program Files (X86)\Steam\steamapps\common\dota 2 beta\dota.exe"

xcopy "C:\DIMMDRIVE_SSD" "D:\Program Files (X86)\Steam\steamapps\common\dota 2 beta prelink" /E /d /q

rmdir /q "D:\Program Files (X86)\Steam\steamapps\common\dota 2 beta"

rename "D:\Program Files (X86)\Steam\steamapps\common\dota 2 beta prelink" "dota 2 beta"

rmdir "C:\DIMMDRIVE_SSD" /S /Q

Code explained:

--Clear the SSD Drive, Quiet mode (No prompts to verify delete function)
rmdir "C:\DIMMDRIVE_SSD" /S /Q

--Create temporary directory
mkdir "C:\DIMMDRIVE_SSD"

--Copy source files to temporary SSD Drive
xcopy "D:\Program Files (X86)\Steam\steamapps\common\dota 2 beta" "C:\DIMMDRIVE_SSD" /E

--Rename original source directory
rename "D:\Program Files (X86)\Steam\steamapps\common\dota 2 beta" "dota 2 beta prelink"

--Create the symlink to point to the SSD drive
mklink /D "D:\Program Files (X86)\Steam\steamapps\common\dota 2 beta" C:\DIMMDRIVE_SSD

--Take a drink of soda (actually gives the system time to generate all the necessary symlinks)
pause

--Launch the game
"D:\Program Files (X86)\Steam\steamapps\common\dota 2 beta\dota.exe"

--Upon quiting the game (i.e. Rage quit) copy any files that changed back to the orginal source directory
xcopy "C:\DIMMDRIVE_SSD" "D:\Program Files (X86)\Steam\steamapps\common\dota 2 beta prelink" /E /d /q

--Remove the Symlink
rmdir /q "D:\Program Files (X86)\Steam\steamapps\common\dota 2 beta"

--Rename the original source directory back to it's original name
rename "D:\Program Files (X86)\Steam\steamapps\common\dota 2 beta prelink" "dota 2 beta"

--Wipe out temporary directory on SSD drive
rmdir "C:\DIMMDRIVE_SSD" /S /Q

Eventually I'm sure DD will have this feature available, it's just a matter of time.  Until that time, here is "A" method (Not the only method) to make use of that cheap SSD you picked up during Black Friday or Cyber Monday sale.

 

Oh, I verified using PROCMON that reads/writes were hitting the SSD and not the 4TB Spindle.  It wouldn't be much work to build a simple parser to autogenerate batch files for all installed games but not sure if there truly is a demand.  Better would have this option in DD.  Patience is a virtue....but BATCH files are still fun.

 

Enjoy..

Napoleon, jibjabst and Tim like this

Share this post


Link to post
Share on other sites

Hey there, thanks for all the work you've put into this extensive post!  Very cool, I really appreciate this actually!  It just goes to show that many people are asking for some of the same types of features to get added to Dimmdrive.   Speaking of which...

 

... The next version of Dimmdrive will have all of this built-in  :)    

 

I've been working on an "auto learn mode" as well as replacing the "USB3 Turbo" with something more intelligent that allows for specifically what you're talking about.

 

Plus more secret new stuff :)

Share this post


Link to post
Share on other sites

That would be wicked, when I heard of dimmdrive I was thinking it would be a good idea to "dimmdrive" to an ssd instead of ram... especially for games that are really big.  But ssd isnt as fast as ram... so it would be even better if we could use BOTH!  :)

Personally I keep my games on 7200rpm drives, cause I got alot of them... I would consider buying a ssd if it was useful with dimmdrive.  But I dont own one now cause they dont have enough space for me... 

Share this post


Link to post
Share on other sites

Well,

 

Took a bit of a hiatus from DIMMDRIVE, I was all excited to come back and see that I can cache to my blank SSD drive, loaded up DimmDrive and...................ugh....

 

Still not implemented !?!?!?!?!?!

 

Tim, whats going on man, this really isn't that big of a deal.  I'm going back to Batch files...oh the humanity !!!!!

 

Tim, show the love, show the care, show how much paying customers like me are worthwhile of having ...

 

Let's get this in, a quick release, you and me, what do ya say?

Share this post


Link to post
Share on other sites

Yo, if anyone is still using this method in Windows 10.  Even tho u turn off UAC, you still need to run BATCH files and such as an Administrator otherwise the SymLinks won't get created.  To eliminate the need to right click on batch file and say run as admin, there is a workaround.

 

Create the batch file.  Now create a shortcut to that batch file.  Right click on shortcut, click ADVANCED, put a check mark in RUN AS ADMIN. Now, just double click the shortcut and it's ran as admin and bingo, all good.  You skip having to right click the file and say run as admin.

 

As an option, you can skip launching the executable via the batch file altogether.  Just delete everything under the "pause" command in my example.  Once the files are copied and symlinks are created, you can launch the game via your Steam Library just as always.  What is pretty cool, you can leave all your games on the SSD drive as long as you want.  The only hitch, if you need to make room on the SSD for another game, you need to resync the files from the SSD DIMMDRIVE back to the source folder so any changes (game saves, profile updates, etc) are copied from the temp SSD file back to the original source.

 

Another cool option.  Nothing is lost if your computer locks up or restarts.  Since all the files are on the SSD, not the RAM, nothing is lost during reboots and such.  So, if you reboot, no need to resync or redo your symlinks, they are all still there.  So, once you create the symlinks for your game, you're pretty much done.  Leave them on the SSD as long as you want.  You don't need to wait for a RAMDRIVE to be created, you don't need to copy files over and over.  Do it once, now just launch via Steam.

 

Hopefully that's clear, if not, send me a pm, no biggie.

Share this post


Link to post
Share on other sites

Sorry, it was meant to be in jest, sorta.  I was really hoping you would implement the option to cache to a spare drive instead of just SysMemory.  You indicated this was to be made available in the next rev but wasn't part of the 1.25 release.

 

Sooo, to keep things lighthearted, it was more of hyperbole final statement.  No, I don't expect to work with you on this.  This is your thing, I'm just a customer.  It was more of "please....please....add it in natively.."

 

Thanks

 

For example, CrystalMark I get ~

 

150MB/s reads from my spindle drive

~490MB/s reads from my SSD drives

~4500MB/s reads from my System Memory

 

So, yes, System Memory is the "Best", but, I'm currently running two 120GB SSD drives, Raid 0.  I get 240GB for temporary storage of my games with ~1000MB/s read performance, of course, using my symlink method defined above.

 

The best scenario is to idenify the top hitters to SystemMemory, the rest to SSD and your SPINDLE DRIVE for the mass quantities of storage.  Thus, you limit your System Memory cache but still get really good performance on everything else.

 

To give an example.  From left to right:

 

7200RPM Spindle Drive

240GB SSD 6gb/s

2 120GB SSD 6gb/s in RAID 0

DIMMDRIVE RamDrive

 

Granted, nothing comes close to the Ramdrive, but, two 120GB SSDs cost me ~$100 bucks, so I'm getting basically maxed Serial ATA 3.0 (6gb/s) throughputs (Realistic, not theoretical calculations).  The cost of System Memory is a major expense in order to have enough spare to push today's games to.  No denying it's performance, just cost of getting there, however, look, for $100 your trashing spindle drives, you're twice as fast as a single SSD and when DimmDrive supports caching to SSDs natively, for the price and performance, yeah, you are sitting pretty.

 

Now for the next challenge, do a double symlink.  Push the entire game to the SSD RAID array, create the symlink, then push the MOST USED files from a game to the RAMDRIVE and create a symlink from the symlink....hhhmmmm

 

 

Disk.JPG

Share this post


Link to post
Share on other sites

I'm not a tech. but everyone will be switching over to SSD -- and what you guys are talking about is a revolution in gaming speeds / a revolution in computer storage  -- Tim, you play your cards RIGHT, CINOAZ idea is a genius plan 4 Dimmdrive and could make you millionaire, however, you do need to move fast b4 an elite discover what you're doing and decide to beat you to the PUNCH. I hate to say it, but  torrent sites are the best advertisement on the net.  Put a dumb, dumb, Dimmdrive on the Torrent sites and some fool like me will install it and say, " damn! This is a good PROGRAM! I gotta buy THIS! "  LOL :)    

Share this post


Link to post
Share on other sites

Okay so let me see if I understand this correctly :)

 

You're saying this:

 

a )  All your games go on your platter-based drives (the slow, traditional ones)

b )  You have some SSD storage available on your SSDs, both of them. 

c )  You also have (example) 6GB of Dimmdrive storage.

 

 

You want to be able to:

 

-- First, auto detect the most used files when you play a game --

Put the most used files on Dimmdrive (primary)

Put the secondary used files on your SSDs scattered (secondary)

and leave the thirdly, least used files on your platter drives

 

Then when you start up the game, it pulls the game off the drives in those 3 orders.

 

That sound right? 

 

 

Because Dimmdrive does all those things now EXCEPT the using of the SSD in addition to the platter (as well as the auto detected most used files).   So you think the REAL BIG HELP would be the using of the third type of drive as a sort of secondary cache (The SSD), or the auto detecting of a game most used files?

Share this post


Link to post
Share on other sites

Actually, remove the platter from the equation totally.

 

Right now, Steam points to my platter drive for all my games (D: drive) I move the game from my platter drive to the SSD drive(X: Drive) and create the symlinks via Batch file.  I then turn on DD.  DD goes to my platter drive (D:) and is redirected to my SSD drive (X: drive) via the symlinks. I configure DD to grab the most used files (using ProcMon) and push them to the RAMDRIVE.  When DD launches, it moves files to the RAMDRIVE but it's actually creating the symlinks on the SSD Drive.  It's a double redirection via the O/S.  So, ALL of the game files run off of the combined SSD and RAMDRIVE, the platter sits idle.

 

When I reboot my machine, my top played games are already on the SSD drive.  Thus, when I turn on my RAMDRIVE, on average, there's less files that need to be copied and the game is ready sooner.

 

Now, do I need to move files to RAMDRIVE in this scenario.  No.  I could be very happy with just the performance gain I get for moving files from the platter to the SSD raid.  Being able to redirect once again to the RAMDRIVE is just that much cooler and better.

 

I envision games having two sliders in DD.  One moves the game from the platter to the SSD.  Then there's a second button called "BOOST", that create the RAMDRIVE and moves files onto it, as configured.

 

Not everyone has oodles of system memory, but, pretty much everyone can afford a couple of SSDs and see their performance increase almost 10 fold, which, then will want them to increase it even more, then there you go.

 

Regarding the "Auto-File feature".  I believe this has been said before, but, allow a user community to submit configurations for games.  I mean, I did all this work to identify the files for CoH2 and I'm enjoying the results.  The XML structure is pretty straight forward, actually, kudos for your work I would've gone with xml or yaml myself.  Should be a simple "search and replace" in the xml file for the >linked< tag.  No need to share the whole file because who knows directory names and structures, but then again, that's why you designed it the way you did :-)

Share this post


Link to post
Share on other sites

Tim; Cinoaz had some really genius idea here...if |his| suggestion for implementation into Dimmdrive natively is possible whatsoever, you seriously should consider it. Dimmdrive is p h a n t a s t i c(!) Period.

-

When I build up my new machine last year, I've gone for an micro setup; albeit max. performance possible (Asus Maximus Impact VII) Unfortunally, this MB only supports 16GB RAM and JUST 2 weeks later, I decided

to purchase Dimmdrive. Arrgh. Would I've only known better before, I would've gone for an complete different machine setup; supporting 64GB RAM. So now - for nowadays heavyhitters like GTA V, Just Cause 3, Fallout 4 and MGS 5... there is waaay to less RAM avaiable here and its (sorry) a pain in the ass to select most used files via procmon - only to lose setup every time one needs to reinstall an new version of Dimmdrive and do it all over again. So Tim;..pretty please? No really; please open up an special Donation link, if necesarry - but please make it happen, like Cinoaz suggested - this would be really (really!) cool. Thank you for listening...  :)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0