Author Topic: AI - Artificial Intelligence  (Read 17006 times)

Jurgis

  • Hero Member
  • *****
  • Posts: 4399
    • Porfolio
Re: AI - Artificial Intelligence
« Reply #20 on: July 01, 2018, 09:03:48 PM »
https://blog.openai.com/openai-five/

You could try a reinforcement learning approach rather than just a supervised learning approach.  The upside here is the algorithm could learn to deal with risks and optimize a portfolio.  The methods discussed in the openai post TRPO  and PPO are very powerful both theoretically and practically and PPO is really easy to implement.

I don't know reinforcement learning in depth. I wonder if there's enough data to run RL on stock prices. Unless you do it on intraday pricing, which I don't really want to do. I think it's the same issue as with supervised learning: 10 years of daily data is only 3500 data points or so. With only 2-3 crashes in data set.

But I'd have to read up on RL to see if there's a way to apply it. If/when I have time. Thanks for bringing it up. 8)

The best returns come from intraday data algorithms.  Not the fundemental type analysis we are all used to.  The reason is these algorithms may be able to average like 10 basis points after costs (just an example your algos probably arent that good).  But if your holding periods are a couple of hours or even minutes, you can make 100%+ in a year which is just not attainable with any longer horizon algorithm.

Thanks for comments. You are likely right, but I have very little interest in intraday-based algos for variety of reasons. 8)
"Before you can be rich, you must be poor." - Nef Anyo
--------------------------------------------------------------------
"American History X", "Milk", "The Insider", "Dirty Money", "LBJ"


Jurgis

  • Hero Member
  • *****
  • Posts: 4399
    • Porfolio
Re: AI - Artificial Intelligence
« Reply #21 on: July 21, 2018, 12:24:27 PM »
The paper I was talking about is
"Dow Jones Trading with Deep Learning: The Unreasonable Effectiveness of Recurrent Neural Networks"
to be presented at http://insticc.org/node/TechnicalProgram/data/presentationDetails/69221

The paper is not publicly available, but you can ask the authors for copy. I have a copy and can send it to people interested, but I won't post it here publicly. PM me if you want a copy.

Couple comments on various things previously mentioned now that the paper is semi-public:

- The paper predicts daily close of DJIA from daily open value + opens of previous n days (2-10).
- The trading algorithm is simply buy if predicted close > open and sell otherwise. If you cannot buy (already have position), then hold. If you cannot sell (already hold cash), then hold cash.
- Authors use training data from 01/01/2000-06/30/2009 and test data from 07/01/2009 and 12/31/2017. This somewhat answers the critique that training is from bull market: it's not. Testing is not completely from bull market either.
- Authors use pretty much vanilla LSTM, so IMO the critique that "1000s of academics looking for signals and the winners publish a paper" or that they have tweaked/over-fitted the model until it worked does not seem to apply. (It's possible that they messed up somehow and used testing data in training, but they seem to be careful, so it doesn't seem very likely). This is really vanilla IMO without much tweaking at all. Which makes the results surprising BTW.
- I have some other comments, but I'd rather discuss this further with people who have read the paper, so I won't post them now.  8)

As I mentioned, I spent a bunch of time reimplementing what these guys presumably implemented.
I do not get their results. My results are pretty much at level of random guessing, i.e. the accuracy is around 48-52% while they get up to 80% accuracy.
It's quite possible I am not doing something the same way they did.
It's possible that their implementation or testing is messed up somehow too. But it's hard to prove that. Maybe they'll opensource their implementation sometime in the future.  8)

If anyone is interested to get together (online through some tools?) and go through the paper and/or my implementation, we can do it. PM me and we'll try to figure out what would work best.  8)
"Before you can be rich, you must be poor." - Nef Anyo
--------------------------------------------------------------------
"American History X", "Milk", "The Insider", "Dirty Money", "LBJ"

cameronfen

  • Sr. Member
  • ****
  • Posts: 407
Re: AI - Artificial Intelligence
« Reply #22 on: July 21, 2018, 06:24:45 PM »
The paper I was talking about is
"Dow Jones Trading with Deep Learning: The Unreasonable Effectiveness of Recurrent Neural Networks"
to be presented at http://insticc.org/node/TechnicalProgram/data/presentationDetails/69221

The paper is not publicly available, but you can ask the authors for copy. I have a copy and can send it to people interested, but I won't post it here publicly. PM me if you want a copy.

Couple comments on various things previously mentioned now that the paper is semi-public:

- The paper predicts daily close of DJIA from daily open value + opens of previous n days (2-10).
- The trading algorithm is simply buy if predicted close > open and sell otherwise. If you cannot buy (already have position), then hold. If you cannot sell (already hold cash), then hold cash.
- Authors use training data from 01/01/2000-06/30/2009 and test data from 07/01/2009 and 12/31/2017. This somewhat answers the critique that training is from bull market: it's not. Testing is not completely from bull market either.
- Authors use pretty much vanilla LSTM, so IMO the critique that "1000s of academics looking for signals and the winners publish a paper" or that they have tweaked/over-fitted the model until it worked does not seem to apply. (It's possible that they messed up somehow and used testing data in training, but they seem to be careful, so it doesn't seem very likely). This is really vanilla IMO without much tweaking at all. Which makes the results surprising BTW.
- I have some other comments, but I'd rather discuss this further with people who have read the paper, so I won't post them now.  8)

As I mentioned, I spent a bunch of time reimplementing what these guys presumably implemented.
I do not get their results. My results are pretty much at level of random guessing, i.e. the accuracy is around 48-52% while they get up to 80% accuracy.
It's quite possible I am not doing something the same way they did.
It's possible that their implementation or testing is messed up somehow too. But it's hard to prove that. Maybe they'll opensource their implementation sometime in the future.  8)

If anyone is interested to get together (online through some tools?) and go through the paper and/or my implementation, we can do it. PM me and we'll try to figure out what would work best.  8)

I dont know what the authors did but ill reiterate from before vanilla LSTMs do little better than guess on the stock market.  They probably had like 1000 GPU and tested thousands of hyperparameter configurations and "overfit" the test set.  This is why typically papers like this are not believed anymore in the ML literature.  Try adding some stuff like attention or skip connections and whatever else is hot now (I'm not sure) and didnt someone recommend GRUs instead.  I have some other ideas you can use like Gaussian Processes to estimate realtime covariance matrices, but your better off looking at the literature first than trying out hairbrained ideas that might not work.   It's really not a trivial excerse to outperform the market with ML. 

Jurgis

  • Hero Member
  • *****
  • Posts: 4399
    • Porfolio
Re: AI - Artificial Intelligence
« Reply #23 on: July 21, 2018, 06:41:25 PM »
The paper I was talking about is
"Dow Jones Trading with Deep Learning: The Unreasonable Effectiveness of Recurrent Neural Networks"
to be presented at http://insticc.org/node/TechnicalProgram/data/presentationDetails/69221

The paper is not publicly available, but you can ask the authors for copy. I have a copy and can send it to people interested, but I won't post it here publicly. PM me if you want a copy.

Couple comments on various things previously mentioned now that the paper is semi-public:

- The paper predicts daily close of DJIA from daily open value + opens of previous n days (2-10).
- The trading algorithm is simply buy if predicted close > open and sell otherwise. If you cannot buy (already have position), then hold. If you cannot sell (already hold cash), then hold cash.
- Authors use training data from 01/01/2000-06/30/2009 and test data from 07/01/2009 and 12/31/2017. This somewhat answers the critique that training is from bull market: it's not. Testing is not completely from bull market either.
- Authors use pretty much vanilla LSTM, so IMO the critique that "1000s of academics looking for signals and the winners publish a paper" or that they have tweaked/over-fitted the model until it worked does not seem to apply. (It's possible that they messed up somehow and used testing data in training, but they seem to be careful, so it doesn't seem very likely). This is really vanilla IMO without much tweaking at all. Which makes the results surprising BTW.
- I have some other comments, but I'd rather discuss this further with people who have read the paper, so I won't post them now.  8)

As I mentioned, I spent a bunch of time reimplementing what these guys presumably implemented.
I do not get their results. My results are pretty much at level of random guessing, i.e. the accuracy is around 48-52% while they get up to 80% accuracy.
It's quite possible I am not doing something the same way they did.
It's possible that their implementation or testing is messed up somehow too. But it's hard to prove that. Maybe they'll opensource their implementation sometime in the future.  8)

If anyone is interested to get together (online through some tools?) and go through the paper and/or my implementation, we can do it. PM me and we'll try to figure out what would work best.  8)

I dont know what the authors did but ill reiterate from before vanilla LSTMs do little better than guess on the stock market.  They probably had like 1000 GPU and tested thousands of hyperparameter configurations and "overfit" the test set.  This is why typically papers like this are not believed anymore in the ML literature.  Try adding some stuff like attention or skip connections and whatever else is hot now (I'm not sure) and didnt someone recommend GRUs instead.  I have some other ideas you can use like Gaussian Processes to estimate realtime covariance matrices, but your better off looking at the literature first than trying out hairbrained ideas that might not work.   It's really not a trivial excerse to outperform the market with ML.

Ah, I think I see where there is a miscommunication between us. :)

My goal is not to outperform market with ML. My goal is to understand whether what is proposed in this paper works and if it does not then why.  8)

You are possibly completely right that what authors propose does not work.
I just want to understand how they got the results they got.

You've said "probably had like 1000 GPU and tested thousands of hyperparameter configurations and "overfit" the test set." before.
I don't think that's the case at all. If you read the paper - which you haven't so far - you can see that their training is really simple and there's no "thousands of hyperparameter configurations". Which is baffling in itself. I have some suspicions of what could be wrong, but it's not productive to discuss it if you just dismiss the paper offhand. Which is BTW your prerogative - if that's where you stand, that's fine and I won't bother you with this further.  8)
« Last Edit: July 21, 2018, 06:43:45 PM by Jurgis »
"Before you can be rich, you must be poor." - Nef Anyo
--------------------------------------------------------------------
"American History X", "Milk", "The Insider", "Dirty Money", "LBJ"

cameronfen

  • Sr. Member
  • ****
  • Posts: 407
Re: AI - Artificial Intelligence
« Reply #24 on: July 21, 2018, 07:58:30 PM »
The paper I was talking about is
"Dow Jones Trading with Deep Learning: The Unreasonable Effectiveness of Recurrent Neural Networks"
to be presented at http://insticc.org/node/TechnicalProgram/data/presentationDetails/69221

The paper is not publicly available, but you can ask the authors for copy. I have a copy and can send it to people interested, but I won't post it here publicly. PM me if you want a copy.

Couple comments on various things previously mentioned now that the paper is semi-public:

- The paper predicts daily close of DJIA from daily open value + opens of previous n days (2-10).
- The trading algorithm is simply buy if predicted close > open and sell otherwise. If you cannot buy (already have position), then hold. If you cannot sell (already hold cash), then hold cash.
- Authors use training data from 01/01/2000-06/30/2009 and test data from 07/01/2009 and 12/31/2017. This somewhat answers the critique that training is from bull market: it's not. Testing is not completely from bull market either.
- Authors use pretty much vanilla LSTM, so IMO the critique that "1000s of academics looking for signals and the winners publish a paper" or that they have tweaked/over-fitted the model until it worked does not seem to apply. (It's possible that they messed up somehow and used testing data in training, but they seem to be careful, so it doesn't seem very likely). This is really vanilla IMO without much tweaking at all. Which makes the results surprising BTW.
- I have some other comments, but I'd rather discuss this further with people who have read the paper, so I won't post them now.  8)

As I mentioned, I spent a bunch of time reimplementing what these guys presumably implemented.
I do not get their results. My results are pretty much at level of random guessing, i.e. the accuracy is around 48-52% while they get up to 80% accuracy.
It's quite possible I am not doing something the same way they did.
It's possible that their implementation or testing is messed up somehow too. But it's hard to prove that. Maybe they'll opensource their implementation sometime in the future.  8)

If anyone is interested to get together (online through some tools?) and go through the paper and/or my implementation, we can do it. PM me and we'll try to figure out what would work best.  8)

I dont know what the authors did but ill reiterate from before vanilla LSTMs do little better than guess on the stock market.  They probably had like 1000 GPU and tested thousands of hyperparameter configurations and "overfit" the test set.  This is why typically papers like this are not believed anymore in the ML literature.  Try adding some stuff like attention or skip connections and whatever else is hot now (I'm not sure) and didnt someone recommend GRUs instead.  I have some other ideas you can use like Gaussian Processes to estimate realtime covariance matrices, but your better off looking at the literature first than trying out hairbrained ideas that might not work.   It's really not a trivial excerse to outperform the market with ML.

Ah, I think I see where there is a miscommunication between us. :)

My goal is not to outperform market with ML. My goal is to understand whether what is proposed in this paper works and if it does not then why.  8)

You are possibly completely right that what authors propose does not work.
I just want to understand how they got the results they got.

You've said "probably had like 1000 GPU and tested thousands of hyperparameter configurations and "overfit" the test set." before.
I don't think that's the case at all. If you read the paper - which you haven't so far - you can see that their training is really simple and there's no "thousands of hyperparameter configurations". Which is baffling in itself. I have some suspicions of what could be wrong, but it's not productive to discuss it if you just dismiss the paper offhand. Which is BTW your prerogative - if that's where you stand, that's fine and I won't bother you with this further.  8)

You are entirely correct that I haven't read the paper and maybe I was too hasty in dismissing the paper.  I wouldn't mind a copy of the paper if you don't mind sending me one. 

That being said here is my reasoning in more depth.  The authors seem like they are in ML acadamia, so I made a couple assumptions.  1.) It didnt look like their paper made it to one of the premier conferences and maybe its because they aren't big names but likely its because people have been training LSTMs on stocks for a long time and vanilla LSTMs dont work well and I think everyone in the ML community is suspicious of 80% hit rates using a vanilla LSTM on indices for good reason and they likely didn't do anything special to assume that they didn't just get "lucky" with their model.  the  reason they got "lucky" is number 2) typically papers dont discuss the hyperparameter search they go through to find the exact correct configuration, so even if they didn't say they tested 100s/1000s of hyperparameters they might have and likely did (although yes i didnt read the paper). Unless they specifically say there were few or no hyperparameters to test or they tested only a few of them, you should assume they did test many.  This is a  dirty secret in ML, you come up with a new technique and you dont stop testing hyperparameter choices the model until you get good results on both the test set and validation set.  Then you submit to to a journal saying this method did really well because it outperformed on both the validation set and test set.  But you stopped right after you get a hyperparameter choice that met those criteria which strongly bias your results upward.  This is related to p-hacking.  This is a perfectly natural, but bad thing people do and usually means most papers have performance that can't be matched when trying to reproduce them.  You can pick basically any method of the thousands that have been proposed and if it doesn't have over 1000 citations (and the method actually seems useful) this is probably one reason why. 

Now you maybe you are right and something else may be missing, but if I had to guess I think its a good chance the authors just got "lucky".  BTW why dont ask the authors for their code.  Its customary to either give this stuff out or post it on github. 

As a side note: Even a vanilla LSTM has many hyperparameters: number of states, activation type, number of variables to predict, test/train/validation breakdown, number of epochs, choice of k in k fold validation, size of batches, random seed, how they intialized weights (glorot, random nomal, variance scaling..) for each weight in the ANN, the use of pca or other methods to whiten data, momentum hyperparameter for hillclimbing, learning rate initialization, choice of optimizer...

My point is that even with a vanilla LSTM the author can pull more levers than can be hope to be reproduced if you don't know absolutely everything maybe even down to the version of python installed to reproduce the pseudorandom number generator.  No doubt some of these choices will be mentioned in the paper, but many of these choices won't be typically, which makes any reproduction difficult.  And typically the authors are the only ones who are incetivized to keep trying hyperparam configurations until one works. 

The real papers that are sucessful are typically methods where either its not impossible to get a reproducible and externally valid hyperparamter configuration, or something that is relatively robust to hyperaprameter choices. 

Jurgis

  • Hero Member
  • *****
  • Posts: 4399
    • Porfolio
Re: AI - Artificial Intelligence
« Reply #25 on: July 21, 2018, 11:47:47 PM »
The paper I was talking about is
"Dow Jones Trading with Deep Learning: The Unreasonable Effectiveness of Recurrent Neural Networks"
to be presented at http://insticc.org/node/TechnicalProgram/data/presentationDetails/69221

The paper is not publicly available, but you can ask the authors for copy. I have a copy and can send it to people interested, but I won't post it here publicly. PM me if you want a copy.

Couple comments on various things previously mentioned now that the paper is semi-public:

- The paper predicts daily close of DJIA from daily open value + opens of previous n days (2-10).
- The trading algorithm is simply buy if predicted close > open and sell otherwise. If you cannot buy (already have position), then hold. If you cannot sell (already hold cash), then hold cash.
- Authors use training data from 01/01/2000-06/30/2009 and test data from 07/01/2009 and 12/31/2017. This somewhat answers the critique that training is from bull market: it's not. Testing is not completely from bull market either.
- Authors use pretty much vanilla LSTM, so IMO the critique that "1000s of academics looking for signals and the winners publish a paper" or that they have tweaked/over-fitted the model until it worked does not seem to apply. (It's possible that they messed up somehow and used testing data in training, but they seem to be careful, so it doesn't seem very likely). This is really vanilla IMO without much tweaking at all. Which makes the results surprising BTW.
- I have some other comments, but I'd rather discuss this further with people who have read the paper, so I won't post them now.  8)

As I mentioned, I spent a bunch of time reimplementing what these guys presumably implemented.
I do not get their results. My results are pretty much at level of random guessing, i.e. the accuracy is around 48-52% while they get up to 80% accuracy.
It's quite possible I am not doing something the same way they did.
It's possible that their implementation or testing is messed up somehow too. But it's hard to prove that. Maybe they'll opensource their implementation sometime in the future.  8)

If anyone is interested to get together (online through some tools?) and go through the paper and/or my implementation, we can do it. PM me and we'll try to figure out what would work best.  8)

I dont know what the authors did but ill reiterate from before vanilla LSTMs do little better than guess on the stock market.  They probably had like 1000 GPU and tested thousands of hyperparameter configurations and "overfit" the test set.  This is why typically papers like this are not believed anymore in the ML literature.  Try adding some stuff like attention or skip connections and whatever else is hot now (I'm not sure) and didnt someone recommend GRUs instead.  I have some other ideas you can use like Gaussian Processes to estimate realtime covariance matrices, but your better off looking at the literature first than trying out hairbrained ideas that might not work.   It's really not a trivial excerse to outperform the market with ML.

Ah, I think I see where there is a miscommunication between us. :)

My goal is not to outperform market with ML. My goal is to understand whether what is proposed in this paper works and if it does not then why.  8)

You are possibly completely right that what authors propose does not work.
I just want to understand how they got the results they got.

You've said "probably had like 1000 GPU and tested thousands of hyperparameter configurations and "overfit" the test set." before.
I don't think that's the case at all. If you read the paper - which you haven't so far - you can see that their training is really simple and there's no "thousands of hyperparameter configurations". Which is baffling in itself. I have some suspicions of what could be wrong, but it's not productive to discuss it if you just dismiss the paper offhand. Which is BTW your prerogative - if that's where you stand, that's fine and I won't bother you with this further.  8)

You are entirely correct that I haven't read the paper and maybe I was too hasty in dismissing the paper.  I wouldn't mind a copy of the paper if you don't mind sending me one. 

That being said here is my reasoning in more depth.  The authors seem like they are in ML acadamia, so I made a couple assumptions.  1.) It didnt look like their paper made it to one of the premier conferences and maybe its because they aren't big names but likely its because people have been training LSTMs on stocks for a long time and vanilla LSTMs dont work well and I think everyone in the ML community is suspicious of 80% hit rates using a vanilla LSTM on indices for good reason and they likely didn't do anything special to assume that they didn't just get "lucky" with their model.  the  reason they got "lucky" is number 2) typically papers dont discuss the hyperparameter search they go through to find the exact correct configuration, so even if they didn't say they tested 100s/1000s of hyperparameters they might have and likely did (although yes i didnt read the paper). Unless they specifically say there were few or no hyperparameters to test or they tested only a few of them, you should assume they did test many.  This is a  dirty secret in ML, you come up with a new technique and you dont stop testing hyperparameter choices the model until you get good results on both the test set and validation set.  Then you submit to to a journal saying this method did really well because it outperformed on both the validation set and test set.  But you stopped right after you get a hyperparameter choice that met those criteria which strongly bias your results upward.  This is related to p-hacking.  This is a perfectly natural, but bad thing people do and usually means most papers have performance that can't be matched when trying to reproduce them.  You can pick basically any method of the thousands that have been proposed and if it doesn't have over 1000 citations (and the method actually seems useful) this is probably one reason why. 

Now you maybe you are right and something else may be missing, but if I had to guess I think its a good chance the authors just got "lucky".  BTW why dont ask the authors for their code.  Its customary to either give this stuff out or post it on github. 

As a side note: Even a vanilla LSTM has many hyperparameters: number of states, activation type, number of variables to predict, test/train/validation breakdown, number of epochs, choice of k in k fold validation, size of batches, random seed, how they intialized weights (glorot, random nomal, variance scaling..) for each weight in the ANN, the use of pca or other methods to whiten data, momentum hyperparameter for hillclimbing, learning rate initialization, choice of optimizer...

My point is that even with a vanilla LSTM the author can pull more levers than can be hope to be reproduced if you don't know absolutely everything maybe even down to the version of python installed to reproduce the pseudorandom number generator.  No doubt some of these choices will be mentioned in the paper, but many of these choices won't be typically, which makes any reproduction difficult.  And typically the authors are the only ones who are incetivized to keep trying hyperparam configurations until one works. 

The real papers that are sucessful are typically methods where either its not impossible to get a reproducible and externally valid hyperparamter configuration, or something that is relatively robust to hyperaprameter choices.

I sent you the link to the paper.
If you look at table 1, there's couple things to notice:
Yeah, for Adagrad, the accuracies are all over the place. But for Momentum and Rmsprop they are all quite similar and way higher than 50% (which would be random guess). So I think this somewhat shows that they did not just pick a single lucky combination of what you call hyperparams. You can still argue that perhaps there's a lucky hyperparam that is not shown in Table 1. That's possible, but I guess it's becoming less convincing. ;)

OTOH, I did not run all the combinations they presented in Table 1, but from what I ran, the results were way more stable and clustered at 48-52% range. So I wonder why they are getting much wider dispersion than I do and why their results have so much better accuracy. So I wonder if their results are correct.

In other words, you question their results because you think they hyperparam hacked. I question their results because I think there's another issue somewhere. But I don't know what it is.

I think you're a bit overstating the instability of the runs. Yeah, there's definitely hyperparam hacking, but IMO - and I'm not a huge expert - the big difference comes from network architecture hacking rather than version of python, random seed, etc.
Also I think you're mostly talking about papers/work where someone tries to squeeze out couple % gain on a widely studied problem where tons of methods have been applied in the past. I'd be more inclined to agree with you if these guys were at 53% accuracy in single or couple tests. But with the number of results in 70% range, I think there's something else going on. But since I don't know what it is, your argument might be still weightier than mine. ;)


"Before you can be rich, you must be poor." - Nef Anyo
--------------------------------------------------------------------
"American History X", "Milk", "The Insider", "Dirty Money", "LBJ"

cameronfen

  • Sr. Member
  • ****
  • Posts: 407
Re: AI - Artificial Intelligence
« Reply #26 on: July 22, 2018, 06:20:48 AM »
not going to quote again but I took a look at their paper.  You are certainly right the results seem to be robust to the param choice they show.  That being said I wonder if there was something to how they cleaned the data because 65% performance on a feed forward ANN is phenomonal.  usually LSTMs only perform like 54-56% at most with a bit of work (but the papers I've read use intraday data).  I'm curious also why they published if there model performed so well.  why not use it themselves if they get good results?

Jurgis

  • Hero Member
  • *****
  • Posts: 4399
    • Porfolio
Re: AI - Artificial Intelligence
« Reply #27 on: July 22, 2018, 07:18:34 AM »
not going to quote again but I took a look at their paper.  You are certainly right the results seem to be robust to the param choice they show.  That being said I wonder if there was something to how they cleaned the data because 65% performance on a feed forward ANN is phenomonal.  usually LSTMs only perform like 54-56% at most with a bit of work (but the papers I've read use intraday data).  I'm curious also why they published if there model performed so well.  why not use it themselves if they get good results?

I have to compliment you: you do hit the right spots.  8)

I also thought that their data cleaning/preparation was suspect. So I ran on data without their "missing dates" transformation. I thought I'll get bad results with that and then the results will jump when I do their transformation, which would prove that their results are caused by bad data preparation. No cigar. I get the same bad results with original data and with data prepared based on their description.
Which does not prove that their preparation wasn't broken... it might be broken in a way that's not described in the paper. Or maybe the description does not match what's in the code.

Anyway, maybe this is enough time spent on this paper. Maybe the right thing to do is to wait if they gonna publish the code (or ask for the code). Or just conclude that their results are broken and we just don't know why.  8)

I'd still be interested to discuss with someone my implementation and where it might be different from theirs. Just to see if I missed something obvious or did something wrong. But my code is hacked up mess, so it's probably not a high ROI for anyone to look at it. 8) I could put my code on github... oh noes, I'm too ashamed of the quality to do it...  ::)

Anyway, thanks for discussion so far.  8)


Ah, regarding
Quote
I'm curious also why they published if there model performed so well.  why not use it themselves if they get good results?

I write this off as academia. People may be more interested in results/papers/thesis (I think this was master's thesis for one author) than in applying it in real life.
Almost nobody from the people I know transferred their thesis/papers into actual startup work. Maybe it's more common nowadays and in certain areas, but it's likely not very prevalent. I guess this area/paper would be easier to transfer into money making than other theses, but they still might not be interested.
A valid question though. I'm not cynical enough to suggest that they know their results are broken and that's why they published instead of using them themselves. I somewhat believe people don't consciously publish incorrect results. But who knows.  ::)
"Before you can be rich, you must be poor." - Nef Anyo
--------------------------------------------------------------------
"American History X", "Milk", "The Insider", "Dirty Money", "LBJ"

cameronfen

  • Sr. Member
  • ****
  • Posts: 407
Re: AI - Artificial Intelligence
« Reply #28 on: July 22, 2018, 01:52:08 PM »
haha thanks.  Just to clarify though, I dont think academics are conciously cheating.  But when you are pushed to get positive results, you typically search until you find something, then you just don't check to make sure you didnt screw something up. 

Liberty

  • Lifetime Member
  • Hero Member
  • *****
  • Posts: 11067
  • twitter.com/libertyRPF
    • twitter.com/libertyRPF
Re: AI - Artificial Intelligence
« Reply #29 on: August 06, 2018, 12:29:30 PM »
https://blog.openai.com/openai-five-benchmark-results/

"Yesterday, OpenAI Five won a best-of-three against a team of 99.95th percentile Dota players: Blitz, Cap, Fogged, Merlini, and MoonMeander four of whom have played Dota professionally in front of a live audience and 100,000 concurrent livestream viewers."

https://arstechnica.com/gaming/2018/08/elon-musks-dota-2-bots-spank-top-tier-humans-and-they-know-how-to-trash-talk/

"OpenAI is using 128,000 cores on Google's Cloud Platform. The bots learn the game from scratch: initial versions will just wander aimlessly and at random as the game plays itself out. As thousands upon thousands of games are played, it figures out which actions will improve its chance of winning."
"Most haystacks don't even have a needle." |  I'm on Twitter  | This podcast episode is a must-listen