Author Topic: AI - Artificial Intelligence  (Read 14690 times)

Jurgis

  • Hero Member
  • *****
  • Posts: 4312
    • Porfolio
Re: AI - Artificial Intelligence
« Reply #10 on: June 14, 2018, 10:34:28 AM »
Wow, so good luck with this? This should be a really, really hard thing to do without a good dataset. I imagine most quant funds (e.g. Renaissance or Bridgewater) have access to large, high quality datasets. I seriously doubt you will do well running LSTMs on widely available datasets (also, donít use LSTMs - use GRUs)

Anyway, if you are serious about this, a good place to start for tools is probably Quantopian. I know one of the principals there and I donít think I can vouch for their financial market chops but there toolsets are probably pretty good (i.e. their python interfaces)

I think you are right that the competition in this is growing exponentially (lol, I just love when someone says "exponentially" especially in bull case writeup  ;D I need to do this exponentially more often  ;D ). And that's definitely an issue, because backtesting will mostly test on data where there were way fewer competitors affecting the market. So you may get good historical results, but the future performance will be crap. Of course what really matters is if the real-money algos are saturating and killing the prediction edge or not. This is tough to measure. I'm sure the hardcore funds have some kind of metrics of noticing when algo gets "exhausted" to shut it off or whatever. But this is where basic theory is not sufficient I'd say.

Anyway, thanks for input.
"Before you can be rich, you must be poor." - Nef Anyo
--------------------------------------------------------------------
"American History X", "Milk", "The Insider", "Dirty Money", "LBJ"


Liberty

  • Lifetime Member
  • Hero Member
  • *****
  • Posts: 10916
  • twitter.com/libertyRPF
    • twitter.com/libertyRPF
Re: AI - Artificial Intelligence
« Reply #11 on: June 26, 2018, 06:45:14 PM »
"Most haystacks don't even have a needle." |  I'm on Twitter  | The importance of saying 'oops'

Jurgis

  • Hero Member
  • *****
  • Posts: 4312
    • Porfolio
Re: AI thread
« Reply #12 on: June 30, 2018, 08:36:58 PM »
Looks like the name of the thread was not great. Can't find it using search function.  :( Should have named AI - Artificial Intelligence.  ::) I don't think I can rename right now.

Anyway, today I've read pretty impressive paper on predicting index X using deep neural nets using 20XX-20XX dataset. Great results. Not sure if this is one of these situations where back testing works and currently the approach stopped working or not. I would have thought that their approach should not have worked on historical data either, since this seems should have been exploited already. They do have training set and test set and some additional checks, so unless they did something "bad", the results seem to be real.

Edit: After some thinking, I have some questions/ideas/reservations/inquisitions to test if this really works or not. If anyone is familiar with DNNs and want to run some experiments, shoot me a message and we can play around. Results could be contribution to science or monetary (if you get a model that works, you can just use it to invest  ;D - most likely you won't ). I might do it myself, but I need time and motivation... yeah, yuge pile of money is not motivation enough.  :P  ::)  8)  ;D

I'll point to the paper when it becomes publicly available.  8)

Just cross validation during bull market years? I've played around with it a bit but never been comfortable enough with the algo [even worst with NNs]. I'm very scared of blowing up with these over-fitted models that have only seen rising markets...

I think the main criticism against these "paper" strategies is you have 1000s of academics looking for signals and the winners publish a paper. The signals they find basically are the result of survivor bias.

Do you guys have slack? Maybe its time we start a CoBF slack group 

So I spent a bunch of time reimplementing what these guys presumably implemented.
I do not get their results. My results are pretty much at level of random guessing.
It's quite possible I am not doing something the same way they did.
As I said before, I'll link to their paper once it's publicly available and someone else might be able to replicate their results ... or not.
I may also post or send my implementation to anyone interested after the paper is publicly available so people can shoot holes in what I did... Although I don't promise to clean up the code hugely... Right now it's a prototype-level mess.  8)
"Before you can be rich, you must be poor." - Nef Anyo
--------------------------------------------------------------------
"American History X", "Milk", "The Insider", "Dirty Money", "LBJ"

frommi

  • Hero Member
  • *****
  • Posts: 1180
Re: AI - Artificial Intelligence
« Reply #13 on: July 01, 2018, 12:14:03 AM »
I think where AI can really be helpful is to predict earnings and then you can use these to build better value portfolios. If you feed an AI with the noise of the markets you will get all types of correlations that don`t hold up in reality. I read an article not long ago on this where they reduced the analysts error rate on earnings projections from 40% to 20%. I can imagine that when you use additional data like credit card information you can get really good earnings forecasts.
Personally i wouldn`t trust an AI black box and i am 100% sure that i would leave that approach when the first larger drawdown happens. Its really hard to determine if you just have a "normal" drawdown or if the model has stopped working, so in the end the human will always be the weak link in this regardless of how automated the whole approach is.

cameronfen

  • Sr. Member
  • ****
  • Posts: 334
Re: AI thread
« Reply #14 on: July 01, 2018, 06:49:13 AM »
Looks like the name of the thread was not great. Can't find it using search function.  :( Should have named AI - Artificial Intelligence.  ::) I don't think I can rename right now.

Anyway, today I've read pretty impressive paper on predicting index X using deep neural nets using 20XX-20XX dataset. Great results. Not sure if this is one of these situations where back testing works and currently the approach stopped working or not. I would have thought that their approach should not have worked on historical data either, since this seems should have been exploited already. They do have training set and test set and some additional checks, so unless they did something "bad", the results seem to be real.

Edit: After some thinking, I have some questions/ideas/reservations/inquisitions to test if this really works or not. If anyone is familiar with DNNs and want to run some experiments, shoot me a message and we can play around. Results could be contribution to science or monetary (if you get a model that works, you can just use it to invest  ;D - most likely you won't ). I might do it myself, but I need time and motivation... yeah, yuge pile of money is not motivation enough.  :P  ::)  8)  ;D

I'll point to the paper when it becomes publicly available.  8)

Just cross validation during bull market years? I've played around with it a bit but never been comfortable enough with the algo [even worst with NNs]. I'm very scared of blowing up with these over-fitted models that have only seen rising markets...

I think the main criticism against these "paper" strategies is you have 1000s of academics looking for signals and the winners publish a paper. The signals they find basically are the result of survivor bias.

Do you guys have slack? Maybe its time we start a CoBF slack group 

So I spent a bunch of time reimplementing what these guys presumably implemented.
I do not get their results. My results are pretty much at level of random guessing.
It's quite possible I am not doing something the same way they did.
As I said before, I'll link to their paper once it's publicly available and someone else might be able to replicate their results ... or not.
I may also post or send my implementation to anyone interested after the paper is publicly available so people can shoot holes in what I did... Although I don't promise to clean up the code hugely... Right now it's a prototype-level mess.  8)

The dirty secret in AI research is everyone is secretly overfitting their ANNs by by fiddling with the archtecture of the model and peaking at test set results.  Only the papers with actual impressive results get published so you have a publication bias.  Doesn't mean a lot of techniques don't work but they likely don't work as well as the paper would lead you to believe. 

Jurgis

  • Hero Member
  • *****
  • Posts: 4312
    • Porfolio
Re: AI thread
« Reply #15 on: July 01, 2018, 07:47:35 AM »
Looks like the name of the thread was not great. Can't find it using search function.  :( Should have named AI - Artificial Intelligence.  ::) I don't think I can rename right now.

Anyway, today I've read pretty impressive paper on predicting index X using deep neural nets using 20XX-20XX dataset. Great results. Not sure if this is one of these situations where back testing works and currently the approach stopped working or not. I would have thought that their approach should not have worked on historical data either, since this seems should have been exploited already. They do have training set and test set and some additional checks, so unless they did something "bad", the results seem to be real.

Edit: After some thinking, I have some questions/ideas/reservations/inquisitions to test if this really works or not. If anyone is familiar with DNNs and want to run some experiments, shoot me a message and we can play around. Results could be contribution to science or monetary (if you get a model that works, you can just use it to invest  ;D - most likely you won't ). I might do it myself, but I need time and motivation... yeah, yuge pile of money is not motivation enough.  :P  ::)  8)  ;D

I'll point to the paper when it becomes publicly available.  8)

Just cross validation during bull market years? I've played around with it a bit but never been comfortable enough with the algo [even worst with NNs]. I'm very scared of blowing up with these over-fitted models that have only seen rising markets...

I think the main criticism against these "paper" strategies is you have 1000s of academics looking for signals and the winners publish a paper. The signals they find basically are the result of survivor bias.

Do you guys have slack? Maybe its time we start a CoBF slack group 

So I spent a bunch of time reimplementing what these guys presumably implemented.
I do not get their results. My results are pretty much at level of random guessing.
It's quite possible I am not doing something the same way they did.
As I said before, I'll link to their paper once it's publicly available and someone else might be able to replicate their results ... or not.
I may also post or send my implementation to anyone interested after the paper is publicly available so people can shoot holes in what I did... Although I don't promise to clean up the code hugely... Right now it's a prototype-level mess.  8)

The dirty secret in AI research is everyone is secretly overfitting their ANNs by by fiddling with the archtecture of the model and peaking at test set results.  Only the papers with actual impressive results get published so you have a publication bias.  Doesn't mean a lot of techniques don't work but they likely don't work as well as the paper would lead you to believe.

This was brought up upthread. In general it is true.

I don't think this is what's happening in this case though, but I'd rather not get into abstract discussions of why I don't think that's the case. OTOH, I can't really explain their results either, so who knows. Let's push out this discussion until you guys have the paper.  ;)
"Before you can be rich, you must be poor." - Nef Anyo
--------------------------------------------------------------------
"American History X", "Milk", "The Insider", "Dirty Money", "LBJ"

cameronfen

  • Sr. Member
  • ****
  • Posts: 334
Re: AI - Artificial Intelligence
« Reply #16 on: July 01, 2018, 08:01:13 AM »
https://blog.openai.com/openai-five/

You could try a reinforcement learning approach rather than just a supervised learning approach.  The upside here is the algorithm could learn to deal with risks and optimize a portfolio.  The methods discussed in the openai post TRPO  and PPO are very powerful both theoretically and practically and PPO is really easy to implement. 

Jurgis

  • Hero Member
  • *****
  • Posts: 4312
    • Porfolio
Re: AI - Artificial Intelligence
« Reply #17 on: July 01, 2018, 08:34:12 AM »
https://blog.openai.com/openai-five/

You could try a reinforcement learning approach rather than just a supervised learning approach.  The upside here is the algorithm could learn to deal with risks and optimize a portfolio.  The methods discussed in the openai post TRPO  and PPO are very powerful both theoretically and practically and PPO is really easy to implement.

I don't know reinforcement learning in depth. I wonder if there's enough data to run RL on stock prices. Unless you do it on intraday pricing, which I don't really want to do. I think it's the same issue as with supervised learning: 10 years of daily data is only 3500 data points or so. With only 2-3 crashes in data set.

But I'd have to read up on RL to see if there's a way to apply it. If/when I have time. Thanks for bringing it up. 8)
"Before you can be rich, you must be poor." - Nef Anyo
--------------------------------------------------------------------
"American History X", "Milk", "The Insider", "Dirty Money", "LBJ"

SharperDingaan

  • Lifetime Member
  • Hero Member
  • *****
  • Posts: 3043
Re: AI - Artificial Intelligence
« Reply #18 on: July 01, 2018, 10:16:43 AM »
A few additions, as one of my partners has significant expertise in this area.

Trying to predict outcome from a disparate data flow is a fools game.
At best the prediction is just a more precise guess, but it just has to be better than the other guys. Back testing is typically against a VaR model, with an AI algorithm that is 'fitted' to the data. Hence the predictive power has to be truly awful to fail the test parameters, yet most do. They can all predict a number, but the +/- standard deviation is so high as to be essentially useless.

Noise versus signal is typically addressed by applying opposing white noise (randomly generated) against the source. Viewed on an oscilloscope you would see a flat line with spikes/valleys suggesting signal. Increase the opposing white noise and you will see fewer but stronger signals - if they exist. The process is well understood, and widely used in robotic industial bottling to poduce a 'fill' within preset upper and lower boundaries at a CI of 95% or better.

An AI robot continually sniffing, continually sees 'new' signal, and could trade accordingly - we call this 'learning'. Problem is that for this to work, the future data stream has to look similar to the 'sampled' historic data stream. The repeated back testing failures tell us that this isn't the case. It also ignores competitors deliberately introducing toxic 'data points' into the market, to screw up your algorithm - & trade against it.

End of the day it essentially remains a zero-sum game.
The AI slice of industry profit barely covers its costs, and comes at the cost of smaller slices of instutional and retail clients.
Speed, # of transactions, and trading volumes increase - but no net benefit.

Not quite what we're being led to believe.

SD

 

 

« Last Edit: July 01, 2018, 10:19:11 AM by SharperDingaan »

cameronfen

  • Sr. Member
  • ****
  • Posts: 334
Re: AI - Artificial Intelligence
« Reply #19 on: July 01, 2018, 05:58:21 PM »
https://blog.openai.com/openai-five/

You could try a reinforcement learning approach rather than just a supervised learning approach.  The upside here is the algorithm could learn to deal with risks and optimize a portfolio.  The methods discussed in the openai post TRPO  and PPO are very powerful both theoretically and practically and PPO is really easy to implement.

I don't know reinforcement learning in depth. I wonder if there's enough data to run RL on stock prices. Unless you do it on intraday pricing, which I don't really want to do. I think it's the same issue as with supervised learning: 10 years of daily data is only 3500 data points or so. With only 2-3 crashes in data set.

But I'd have to read up on RL to see if there's a way to apply it. If/when I have time. Thanks for bringing it up. 8)

The best returns come from intraday data algorithms.  Not the fundemental type analysis we are all used to.  The reason is these algorithms may be able to average like 10 basis points after costs (just an example your algos probably arent that good).  But if your holding periods are a couple of hours or even minutes, you can make 100%+ in a year which is just not attainable with any longer horizon algorithm.