"Backtest Unable To Complete. See Console Tab For More Details"
When I open one of the backtests (a specific one, does not happen for others), I see the main graph briefly flashed, then disappearing and giving way to the above message. I scrolled through all the dates -- there are no failed jobs -- they have all completed. And it even looks like the report is being generated. But a summary and a graph on the web would still be nice.
Please, advise.
Tagged:
Comments
KC This is a known bug, it is causing frustration for our internal users as well. The devs have multiple examples and are trying to get to the bottom of the issue. For now you have two options... a) Just re-run the back-test b) If you do not care about the reports and you do not carry trades from one day to the next, just re-run the failed dates.
We are working hard to identify the cause of this issue.
Thanks!.
Interestingly, in this particular case there are actually no failed dates But yes, I definitely at this point don't care about the picture on the web -- was hoping to download the report, and look at the data, but it is stuck in "report generating" state.
Will try to re-run...
I just got another one, but this time I have saved the submission hash (which does not appear on the webpage because scorecard is not completed) -- could you please look at it and generate the report, if possible (i think it is promising
:
dfcac6b0782b36f449a72c316d6369b5
KC: I cannot tell which of three issues you are referring to here : scorecard did not generate, individual days failed within a long backtest or the report generation got stuck?
Those are three different issues. For the first two the only solution is to re-run.
"Individual days failing" is a random, rarely occurring intermittent issue, those are the hardest to find.
It tends to happen more (or be more painful for) longer backtests. Yet, run the same backtest again and there is a good chance it will complete without issues.
The failures seem to show up randomly in the middle of long backtests. That makes it very difficult for the developers to find. Developers always prefer short code that runs fast, can replicated the issue and can be re-run quickly after a fix is put in place to demonstrate that the issue has been resolved. That is not possible in this case.
We do believe we have located a couple of points of random failure, these were not software but were on small subsections of our backtesting infrastructure.
All hashes for failures are useful and are passed to the devs, but from a point of view of your own personal resolution, 9 times out of 10 I would say if an immediate re-run may resolve, just put it straight back in.
Sorry should have been more clear on that: the backtest has completed. When I click on the link in the "Results" page the backtest page loads, the scorecard briefly flashes (for about 1 sec) , but then immediately disappears, and is replaced with a text message "Backtest unable to complete, please see console for details". However, none of the days in the backtest shows up as "Failed", so not sure where or what in the console output should I look for.
Understood on other issues above. Perhaps a way out might be to run backtests on year-per year basis.
If you can run on a year by year basis then consolidate the results and that works for you then that is certainly one way out. Our devs were working on the system over the weekend and "drained the queue" a couple of times, which should not have caused a problem but it does seem to - someone internally was running some large backtests over the same period and had the same issues - so the devs are aware and the extra Hash can only help them to try to pin it down.
ok, thanks. i will try year-by-year, good idea.
Hi! I'm running into the same problem "Backtest unable to complete, please see console for details". All days complete fully with no errors.
I'm porting over about 40 factor model strategies from another platform that should all pass the "funding criteria". I was able to complete the first one. In some cases I can download the "detailed porfolio report" others just stall in the queue. Obviously I need to run over 2 years to meet the criteria so the year by year won't work for me. Here's the first one run that completed successfully on 3/15. These 40 are uncorrelated and can be combined naively into a >4.5 sharpe strategy or regressed over a risk model for greater sharpe. I would greatly appreciate a fix. Thanks!
AS
I feel your pain, we are fully aware of the situation and are trying to resolve it. The problem is that it only happens on long backtests and it is random. Those two things combined create a real difficulty in tracking it down. What I would say is that most of the time, if you just re-run you will get a successful backtest. Believe me, it is frustrating for us that large complicated backtests fail because of a couple or a few random failed days and the who run has to be re-submitted, consuming yet more CPU time. But until this is resolved I would go so far as to suggest, if you are experiencing it regularly.... submit the same backtest twice at the same time. You will have more chance of a "complete" backtest then.