service.write_file() delayed?
I've been working on a variant of neuro-evolution as a strategy, and part of that requires writing out the neural net config to a gene pool file so it can be read back, mixed with other winning variants and mutated.
All of that it working, but I've noticed that whatever I write to my file, the changes aren't available to read back immediately. There seems to be a delay of several minutes, and I'm worried that my evolution process might be losing potentially good strategies in the process. Even if there's no risk of data loss, it's slowing down the process of training my system considerably. Is there a way to work around this delay, or remove it?
Comments
JD
I suspect the restriction you are hitting is inherent in the design of our system.
When you run a job it, each day is a simulation all to itself.
It pulls all the files and data it needs and runs as a self contain capsule of a single day run.
Once that simulation has completed, any file creation/modificaiton that was done within the confines of the backtest is written back to the "User Data" section.
Hence we suggest, if you are running a multiday parallelised backtest, that you add the date of the backtest to the file name to prevent file overwrite issues. In a "normal live programming environment" you would have file contention issues but you can see that with a backtesting system that would be very difficult to handle. If you wish to use a single file, modified day to day, you can turn off the parallelisation on your simulation checking "RUN AS A SINGLE MULTIDAY JOB" which will force the simulator to run each day individually and wait for it to complete before running the next day.
Obviously this negates all the advantages of parallelisation and your backtest will take a lot longer to run.
If you have already taken all of this into account and are seeing some other type of delay (ie the job completes but the file is not written to user data for a couple of minutes) then that may be another issue. If so let me know, either here or direct at customer_success@cloudquant.com
I use the "RUN AS A SINGLE MULTIDAY JOB" option all the time, since the nature of what I'm trying to do sort of requires it.
What's weird is that the file contents don't update until the whole job completes.
It's not all that important now, though. I figured out a slightly ugly way to cache the file contents in my strategy so I don't lose any data.
That makes sense, if you imagine it as spinning up a machine environment for each job.. so what defines a "job"...
When you run a parallel backtest, each day gets an emulated machine (job) spun up and at the end of each day any file creation/amendments are copied from the emulated machine to your USER DATA area.
When you "Run as a single multi day job" that is effectively one emulated machine (job) that stays up the whole time, only writing the file out when the emulation completes.
As long as you are backtesting and running in that way you are fine to keep your data in memory, but if you ever go live you will want do it the way you originally attempted...to write to a file then read in the next day
You can detect if you are running live or backtesting with ktgfunc.backtesting() which returns True if it is a backtest siimulation or False if it is Live/in Production.