Tips for Datastore / DataStore.Save();

[.phase]

I just read Snakes post under the Tech Support thread and it made my eyes go wide . I wanted to reply there right away to warn him, but it was locked. (It felt like I was jumping and screaming to warn someone that's about to walk in front of a moving bus, but had headphones on. xD)

Removing all instances of Datastore.Save(); may get rid of the problem at the moment with write permissions, but it may be bad in the end.

I had a conversation with Sleepy when he was active on Magma. He mentioned that he doesn't use ANY Datastore.Save(); because it just caused him problems. I saw his logic and agreed that a Datastore.Add(); is already writing data to the file.

Now, I don't know if this is a difference between running a server from a GSP vs private, but for me on a GSP, at least, if I don't perform a Save(); the Datastore.ds file never updates, so I can never back it up. If my server crashes or reboots, all the Datastore info is lost!

I then realized that I had to find a middle ground. Some people were adding DataStore.Save(); just about everytime they were writing to it, which is probably why Sleepy found it to cause problems, too many Save(); bumping into Add(); Remove(); Get(); Flush(); etc, and I still agree with that. IMO--the key is to put them in only for critical stuff that isn't accessed often.

For instance, for me, everytime someone uses a teleport, I update their cooldown timestamp in Datastore. I don't save here because that's completely unnecessary. If it crashes and cooldowns are reset, fuggit!

Where I do save is for thing's that are pretty much static. Like when someone sets their home location. I want that to save as soon as they set it, so it's never lost. (Unless, the server crashes during a write and it corrupts the ds file, and then you're screwed unless you backup religiously.)

Just wanted to share my experience with everyone. I use Datastore for just about everything, because posts on Magma convinced me that it's faster than reading/writing to multiple, separate INIs. And you can easily access the same data set across multiple functions/plugins. That's why it's vital for me to save and backup, and why I wanted to give you guys a heads up if you were unaware.

Cheers!

 

[balu92]

You shouldn't call .Save() manually, ever. Why? Because of serialization and other errors. (some dev will store non-serializable data in the ds in runtime, you can't do anything about that, unless you do everything for yourself).
DataStore should be saved more often and not just when at shutdown, tho.

 

[DreTaX]

You shouldn't call .Save() manually, ever. Why? Because of serialization and other errors. (some dev will store non-serializable data in the ds in runtime, you can't do anything about that, unless you do everything for yourself).
DataStore should be saved more often and not just when at shutdown, tho.

+1

 

[.phase]

Absolutely agree that it should be automatically saved more often.

In the meantime, if I don't place a couple manual save instances, I can never back up my ds file locally unless I interrupt the server with a shutdown.

I haven't experienced any issues with Datastore in the few months that I've had things set up this way. What I have seen is when the server crashes and I lose my entire ds file (gets overwritten with a new file). So, I am inclined to keep it this way at the moment.

If someone can elaborate, though. What was the intention of letting us call it manually to begin with? Was there ever an intended "correct" use case?
Just curious.

Many thanks for shedding some light on the topic. Will be very good for everyone to understand it better.

 

[balu92]

Absolutely agree that it should be automatically saved more often.

In the meantime, if I don't place a couple manual save instances, I can never back up my ds file locally unless I interrupt the server with a shutdown.

I haven't experienced any issues with Datastore in the few months that I've had things set up this way. What I have seen is when the server crashes and I lose my entire ds file (gets overwritten with a new file). So, I am inclined to keep it this way at the moment.

If someone can elaborate, though. What was the intention of letting us call it manually to begin with? Was there ever an intended "correct" use case?
Just curious.

Many thanks for shedding some light on the topic. Will be very good for everyone to understand it better.

It was meant to use it as you will, I guess! Its just not safe to call, when ppl store players and vector3s and who knows what in the ds.

 

[xEnt]

hmm i think i was the one who wrote this class awhile back for magma i know it was fairly basic and can hardly even remember the code, but if thread synchronization is becoming a problem, is it worth adding some lock/release code on the mutators, or a thread safe collection?

also if i can remember correctly, i did want volatile memory to not be serialized but didn't want to have to create another collection and another set of mutators with a different name, or extra arg to distinguish. so i kept it simple for scripting and let everything save/load. this could be a good thing, and a bad thing.

I wonder if its worth moving even to SQL/SQLite to allow external access to the data down the track?

 

[DreTaX]

hmm i think i was the one who wrote this class awhile back for magma i know it was fairly basic and can hardly even remember the code, but if thread synchronization is becoming a problem, is it worth adding some lock/release code on the mutators, or a thread safe collection?

also if i can remember correctly, i did want volatile memory to not be serialized but didn't want to have to create another collection and another set of mutators with a different name, or extra arg to distinguish. so i kept it simple for scripting and let everything save/load. this could be a good thing, and a bad thing.

I wonder if its worth moving even to SQL/SQLite to allow external access to the data down the track?

SQLite sounds good.

 

[mikec]

This problem is fixed in master branch and will be in my next MC release. Anticheat stores Vector3 objects temporarily in the DataStore. They don't actually need to be saved when the server shuts down. But Vector3 does not implement or inherit a serializer. So the Hashtable is corrupted when saved to disk. The solution was to convert Vector3 to a string going in, and instance a Vector3 coming out. I've been running Anticheat since Sunday, and I uncommented DataStore.Save/Load from Magma plugins on my test server, and I have had no DataStore exceptions.

You can still throw an exception if you try to use a null value as a key. But that's Hashtable. Should I convert null keys to string "null"?

private object StringifyIfVector3(object keyorval)
{     
    if (keyorval != null) {
        try { 
            if (typeof(Vector3).Equals(keyorval.GetType())) {
                return "Vector3," +
                ((Vector3)keyorval).x.ToString() + "," +
                ((Vector3)keyorval).y.ToString() + "," +
                ((Vector3)keyorval).z.ToString();
            }     
        } catch (Exception ex) {
            Logger.LogException(ex);
        }     
    }     
    return keyorval;
}     

private object ParseIfVector3String(object keyorval)
{     
    if (keyorval != null) {
        try { 
            if ((keyorval as string).StartsWith("Vector3,")) {
                string[] v3array = (keyorval as string).Split(new char[] { ',' });
                Vector3 parse = new Vector3(Single.Parse(v3array[1] as string),
                                    Single.Parse(v3array[2] as string),
                                    Single.Parse(v3array[3] as string));
                return parse;
            }     
        } catch (Exception ex) {
            Logger.LogException(ex);
        }     
    }     
    return keyorval;
}

 

[mikec]

I should probably also generalize this for Vector2 and Vector4. And maybe Bounds (which are a pair of Vector3).

 

[DreTaX]

This problem is fixed in master branch and will be in my next MC release. Anticheat stores Vector3 objects temporarily in the DataStore. They don't actually need to be saved when the server shuts down. But Vector3 does not implement or inherit a serializer. So the Hashtable is corrupted when saved to disk. The solution was to convert Vector3 to a string going in, and instance a Vector3 coming out. I've been running Anticheat since Sunday, and I uncommented DataStore.Save/Load from Magma plugins on my test server, and I have had no DataStore exceptions.

You can still throw an exception if you try to use a null value as a key. But that's Hashtable. Should I convert null keys to string "null"?

private object StringifyIfVector3(object keyorval)
{    
    if (keyorval != null) {
        try {
            if (typeof(Vector3).Equals(keyorval.GetType())) {
                return "Vector3," +
                ((Vector3)keyorval).x.ToString() + "," +
                ((Vector3)keyorval).y.ToString() + "," +
                ((Vector3)keyorval).z.ToString();
            }    
        } catch (Exception ex) {
            Logger.LogException(ex);
        }    
    }    
    return keyorval;
}    

private object ParseIfVector3String(object keyorval)
{    
    if (keyorval != null) {
        try {
            if ((keyorval as string).StartsWith("Vector3,")) {
                string[] v3array = (keyorval as string).Split(new char[] { ',' });
                Vector3 parse = new Vector3(Single.Parse(v3array[1] as string),
                                    Single.Parse(v3array[2] as string),
                                    Single.Parse(v3array[3] as string));
                return parse;
            }    
        } catch (Exception ex) {
            Logger.LogException(ex);
        }    
    }    
    return keyorval;
}

C# WEBSITE SYNTAX DO YOU SPEAK IT ​

 

[mikec]

MODERATOR POWERS DO YOU USE THEM​

 

[balu92]

I should probably also generalize this for Vector2 and Vector4. And maybe Bounds (which are a pair of Vector3).

Quaternion maybe?

 

[DreTaX]

MODERATOR POWERS DO YOU USE THEM​

NO I HAVE ADMIN POWERS, WHY SHOULD I

 

[mikec]

Quaternion maybe?

Why not - same structure as Vector4.

 

[mikec]

Datastore.Add(); is already writing data to the file.

Nope. It only adds it to the in-memory Hashtable.

For instance, for me, everytime someone uses a teleport, I update their cooldown timestamp in Datastore. I don't save here because that's completely unnecessary. If it crashes and cooldowns are reset, fuggit!

100% correct. Anticheat took this approach. Unfortunately, Anticheat could not predict when some other plugin would invoke DataStore.Save() or when the server might be shutdown. Both events would write the Hashtable to a file, and the temporary Vector3 stored in it would cause corruption.

I use Datastore for just about everything, because posts on Magma convinced me that it's faster than reading/writing to multiple, separate INIs. And you can easily access the same data set across multiple functions/plugins.

All correct, and that was why I figured the best approach is to make DataStore safe for the things people are using it for, rather than tell people not to use it for certain things, or not to do this or that too much.

 

[balu92]

On_DataStoreSave():void - :evilface:

 

[mikec]

On_BaluConnected() :rightbackatcha

 

[Snake]

LOL I just saw the post now... well thanks anyways I discovered that the .Save() calls were destroying my plugin so I deleted them all (I figured this out some days after the post, about 6/09 but whatever, thanks for all the info

BTW I never locked the post, that was a mod/admin not me :S

 

[mikec]

        public static void savealldata()
        {
            try
            {
                AvatarSaveProc.SaveAll();
                ServerSaveManager.AutoSave();
                Helper.CreateSaves();
                DataStore.GetInstance().Save();
            }
            catch (Exception ex)
            {
                Logger.LogException(ex);
                Logger.Log("Error while auto-saving!");
            }
        }

        public static void shutdown()
        {
            time = int.Parse(Core.config.GetSetting("Settings", "shutdown_countdown"));
            System.Timers.Timer timer = new System.Timers.Timer();
            timer.Interval = 10000.0;
            timer.AutoReset = true;
            timer.Elapsed += delegate(object x, ElapsedEventArgs y)
            {
                shutdown_tick();
            };
            timer.Start();
            shutdown_tick();
        }

        public static void shutdown_tick()
        {
            if (time == 0)
            {
                Util.sayAll(Core.Name, "Server Shutdown NOW!");
                try
                {
                    AvatarSaveProc.SaveAll();
                    ServerSaveManager.AutoSave();
                    Helper.CreateSaves();
                    DataStore.GetInstance().Save();
                }
                catch (Exception ex)
                {
                    Logger.LogException(ex);
                }
                Process.GetCurrentProcess().Kill();
            }
            else
            {
                Util.sayAll(Core.Name, "Server Shutting down in " + time + " seconds");
            }
            time -= 10;
        }