Weekend Maintenance and Tick Data: Just how much can we really get?

Recently, I have been receiving a lot of customization/development requests for tools that use tick data, and I think that’s great. It is great that people realize that tick data can be useful, and try to integrate it into their trading. However, sadly, tick data is very difficult to work, often crashing Trading Station if we have too much of it, but that is not that common because it is hard to get a hold of in the first place. Sadly, despite my best efforts, I cannot produce what the client wants.

Why is hard to get a hold of FXCM tick data? After all, the Strategy Backtester has tick data support right? …

Unfortunately, the Strategy Backtester does not actually support real ticks – surprised? While it is true you can select the tick timeframe for your back-testing, the tick data used in back-testing is simply a simulation of the one-minute bar. The details about how this works can be found on this FxCodeBase Price Simulation page, so I do not need to go into them, but you should know that this simulation can severely impact the results when back testing strategies that rely on tick data (more on this when I complete by Backtesting vs Live Trading post…)

Okay, so we cannot get the tick data from the back-tester, let’s just request it live from the FXCM servers. Again, yes and no.

You may have noticed that if you open a Tick Chart on Market Scope, and scroll back as far as you can, there is a limit. You may even have noticed that the limit appears to be different for different instruments. Well, that is both true and false at the same time. The limit enforced is actually a 5000 tick limit, which depending on the volatility of that instrument, may mean longer or shorter histories when measured in time.

So just how long is 5000 ticks? Well, at the moment of writing this post at 10:40am Japan Time,
– EURUSD.T shows me about one hour of data,
– About the same for USDJPY,
– XAGUSD however gives me about 12 hours of data, and finally
– USDOLLAR shows me 26 hours of data.

Luckily, the framework allows us to programmatically request more history. I suspect that this was probably not designed for tick data originally but rather to be used for candle histories. You see, while there is a 5000 limit for tick data, the default limit for candle/bar data is only 300 bars. If you want more than 300 bars in your indicator or strategy, then it needs some special logic to request this.

Utilizing this same special logic, we can actually request more tick data. So, if we do this, just how much tick data can we actually get?

To test this, I produced a tool called the Tick History Checker. It simply allows me to select the instrument and off it goes requesting as much tick data as possible. The results, perhaps as expected, were different depending on the instrument. However, perhaps not as expected, the more volatile the instrument, the more tick data we got. Why is this?

Symbol Results
USDOLLAR Loaded 5043 ticks for period 2015/05/18 07:00:51 to 2015/05/19 11:10:51′.
EUR/USD Loaded 242619 ticks for period 2015/05/18 07:00:00 to 2015/05/19 11:14:30′.
XAU/USD Loaded 100652 ticks for period 2015/05/18 08:00:04 to 2015/05/19 11:20:00′.
GBP/USD Loaded 208394 ticks for period 2015/05/18 07:00:01 to 2015/05/19 11:22:55′.
AUD/USD Loaded 163575 ticks for period 2015/05/18 07:01:20 to 2015/05/19 11:22:26′.
USD/JPY Loaded 142929 ticks for period 2015/05/18 07:00:00 to 2015/05/19 11:21:29′.
USOil Loaded 25837 ticks for period 2015/05/18 08:02:52 to 2015/05/19 11:24:17′.
NZD/USD Loaded 139707 ticks for period 2015/05/18 07:00:01 to 2015/05/19 11:27:15′.

Well, looking at the results above, it is pretty obvious that we are not restricted by a tick count, but rather by date. While it may be tempting to think that we are restricted to about one day of tick data, I believe this limit is actually the start of the week.

This ties in nicely with a theory that I have. You see, I have made some interesting observations in the FXCM system over the years… they are:
1) The price servers are unavailable on the weekend.
2) To back-test the current week data, you need to login.
3) Sometimes price data goes missing if you leave Trading Station open over the weekend: http://www.fxcodebase.com/code/viewtopic.php?f=25&t=40395&start=20#p94409
4) Back-testing the current week data takes a lot longer to start/load the data than last week’s data. Additionally, this problem appears to be a resource-contention issue. Sometimes it works, sometimes it does, and there is a higher chance of success when the markets are quiet indicating it has something to do with how busy the servers are.

These observations lead me to believe that the price data for the current week is stored differently than the previous week’s data (which makes sense from a technical architecture point of view). Additionally, over the weekend, I believe FXCM perform some post-processing and maybe filtering or compacting of the results. I have not been able to confirm this with FXCM, but I am not the first person to notice such inconsistencies (they did back in 2011!). Unfortunately for them, they never got confirmation either either: http://www.dailyfx.com/forex_forum/fxcm-trading-station-ii-marketscope-charts/451955-why-fxcm-downgrading-tick-volume-values-each-weekend.html

So there you have it … now we know how much tick data we can get, and why we cannot get more!

Posted in Backtesting, FXCM, Indicators, MarketScope, Research, Strategies, Trading Station and tagged , , , , , , , .

One Comment

  1. Yes, they cleanup tick data. They remove bad ticks. The admitted this publicly. But very ticks are deleted.

    Also, using their APIs you can get tick data as far back as 2007 (up to when that instrument started trading)

Leave a Reply

Your email address will not be published. Required fields are marked *