That is a good summary, but it is a little bit more complex than what you have stated. But your comment is helpful because this is exactly the understanding that Goldbloom has and it is incorrect in an important way.
The 23 entries that were submitted are for the public leaderboard set. The purpose of this set is so over the duration of the competition, competitors can see how they are performing relative to each other. These 23 submissions have no impact on the final prize winners.
A different data set, called the final data set is what actually determines the final prize winners. I submitted this file exactly one time. It was processed and scored by Kaggle and I followed the directions exactly as the competition admin stated. I did not check a checkbox because I did not believe it applied - I only submitted a single file and I recieved the 'all zeros' response from Kaggle's system as it said it would.
Also, the text Anthony quoted is not applicable to the Flight Quest competition either. This is the text that is used for the single phase competitions, not a two phase competition like Flight Quest.
If you haven't competed before I know the difference might not be clear. But if you read the response from the other competitors in the forum, you can see there is general agreement that the submission page is faulty for a two phase competition.
1. So your stance is that Anthony, the CEO and Founder of Kaggle, is mistaken about the rules and execution of the 2 phase competition that his company is running, even after 8 days of his company communicating back and forth in email correspondence with you?
2. In the forum you initially state:
"As I have been saying over email repeatedly over the past 8 days, and yesterday on the phone with you and the Sorkin, there is no checkbox to check!"
..then later state:
"What on earth is a checkbox doing on the submission screen of a two phase competition? There is no ambiguity - there is only a single file type that is valid for the final leaderboard file. Why on earth is your software not smart enough to figure that out?"
and later:
"I submitted exactly one final submission. At this point the checkboxes should not matter"
and in your reply above state:
"I did not check a checkbox because I did not believe it applied"
While I'm not familiar with all of the intricacies of the competition, conflicting statements like those have me a bit confused over this whole thing.
3. $30k is a lot of money, so I sympathize with you for missing out on that given that your algorithm is better than what the final results suggest. That being said, the other entrants that won competed using the same submission process that you did, was it just bad luck that your entry was not accepted in the manner you had expected or why did they not also encounter this submission bug?
4. Is this your first two phase competition with Kaggle? Was this competition executed different than other competitions? I ask as you seem surprised/confused about this "checkbox" that appears to be at the center of the controversy.
Thank you for reading the forum trying to better understand my POV.
re: #1. Yes, my stance with Anthony is that he doesn't think there is a bug in the software because he thinks this is a typical one phase competition. Flight Quest was more complicated than the typical Kaggle competition.
re: #2. I have done a poor job explaining the checkbox. I understand the confusion, as my statements are contradictory. In a single phase competition, the checkbox is important because over the course of the competition, numerous submissions are scored and you ultimately need to pick the one you want to apply as your final submission. In the case you do not make a final selection, the system selects the most recent submission.
The same UI is used for the two phase competition, including the same text that Anthony posted in his response to me, however the submissions work slightly differently as I mentioned.
> "As I have been saying over email repeatedly over the past 8 days, and yesterday on the phone with you and the Sorkin, there is no checkbox to check!"
I said this because I don't believe there should have been a checkbox on the screen. That is what I meant by saying there is no checkbox to check.
You have a dispute with them. They're running a business and want the dispute (and you) to go away. Eventually, you will go away, either because you gave up or lost in court; or because they paid you to do so. If you're ethically entitled to compensation, this is a perfectly decent thing for them to do.
I will do some research on how I can find some representation. Any tips, or experience with who I might get in contact with? Am I looking for some sort of technology or competition specialist?
I suggest you google for lawyer sweepstake New York
And lawyer contest law New York - talk to a couple till you get a feel for it. It about exploring your options not going to court tomorrow, so the gung-ho ones might not be best. But that's a Brit talking :-)
I get the instant replay analogy, but I have a problem with a bug in their software impacting the results of the competition. I work in software, so I get living with bugs and understanding they are always going to be there. But I am unhappy that Kaggle/GE announced the winners via press release at the same the rest of the competition field was notified. It makes it impossible for a competitor to dispute the final results.
From the response I have received from Kaggle/GE I get the impression, under no circumstances what so ever would they go back and change the published results of this competition.
>"It makes it impossible for a competitor to dispute the final results."
Yes. That's the purpose.
The teleology of gamified competition is toward declaring a winner. The rules are structured with that in mind. Those rules are different from the marketplace where protesting and litigation are allowed and consequently, declared winners are simply a matter of opinion, not fact.
Accepting mistakes as part of the game is sportsmanship. And it was a game. Best to move on to the next one: http://www.kaggle.com/competitions
>'... declared winners are simply a matter of opinion, not fact'
I certainly agree with that part of your statement.
I have a hard time thinking about moving on to a new competition, this one has left such a bad taste in my mouth. Maybe time will change my mind, but certainly not happening any time soon.
A bit of my background - I referee futbol (or as we call it, soccer). Every game, I try to give my best performance to the players. Sometimes, I make bad calls. The score is still the score.
Obviously, you're pretty damn good. Your competitors deserve to compete against you. Cheating is unfair. Mistakes aren't. Play long enough and it all evens out. At least statistically.
Ironically, that has been the best part about this ordeal - a number of the top competitors have shown support through the forum as well as private emails. I compete alone, so this issue has given me the opportunity to get to know some of the other folks.
I am not certain I want to work on anymore Kaggle competitions. I put (just a guess) somewhere between 100 and 200 hours into FlightQuest. It is hard for me to rationalize putting this amount of effort into another Kaggle hosted competition after going through the experiences I went through for this competition.
I have not consulted a lawyer yet, the reason being the competition rules state the following:
"By participating in the Competition, each Entrant agrees to release, indemnify and hold harmless GE, Kaggle Inc., and their respective affiliates, subsidiaries, advertising and promotions agencies, as applicable, and each of their respective agents, representatives, officers, directors, shareholders, and employees from and against any injuries, losses, damages, claims, actions and any liability of any kind resulting from or arising out of your participation in or association with the Competition. GE is not responsible for any miscommunications such as technical failures related to computer, telephone, cable, and unavailable network or server connections, related technical failures, or other failures related to hardware, software or virus, or incomplete, late or misdirected Entries. GE reserves the right to cancel, modify or suspend the Competition should any computer virus, bug or other technical difficulty or other causes beyond the control of GE corrupt the administration, security or proper play of the Competition, and to determine winners from among Entries not affected by the corruption, if any, in its sole discretion."
I interpret that as saying: GE can pick the winners using its sole discretion. So I don't believe this is a legal issue I can win. I have a lawyer in the family, I will run it by her and see what her opinion is.
Please, get a lawyer outside the family with experience of competition law - maybe your sister/aunt can tell you the area of law that might apply and point to some local firms
You need objective advice, not coloured by "what if my advi e leads to not speaking to half the family" - don't put that on her.
Secondly, consult a lawyer because they have - if according to the common-sense rules you should get 30k (and I don't at all understand the situation) then no amount of disclaimers or indemnification deprives you - statute on competitions run for marketing purposes are widespread.
Maybe you have a case, maybe not, but without a lawyer you won't know and without a lawyer they (GE?Kaggle) won't take you seriously - which kind of screws the only other reason you seem to have for not getting a lawyer - the idea that maybe you will pick up some work later on. If they see you as a push over on this, why will they respect you next time?
This does not mean drop a writ for 10 million in the morning - get advice, if there is a case, ask if the lawyer can directly contact the organisers lawyers for clarification or discovery. If you have a case they will have to do something - push now, so that their decision has not been set in concrete and them moved on.
I am a 'Kaggler', I have done well (prize winner) in some past competitions and I could not agree more.
In the last week of a competition, you go into kitchen sink mode trying anything and everything to squeeze the last few points out of your models. The objective of the competition from the competitors standpoint is to win, not create a solution that could be deployed into the sponsor's production environment. On going maintenance of the model is not a consideration.
As far as the netflix competition goes though, the final solutions did help publicize the potential of ensembling and RBMs - I am grateful for that. My personal approach to the competitions is to use them as an opportunity to try out new modeling techniques (i.e. RMBs, deep beliefs etc...) on real world data. It has the added bonus of potentially paying off with prize money (as long as you don't get screwed[1]).
I would like to hear from past Kaggle sponsors and see what they have done with the winning models.
The 23 entries that were submitted are for the public leaderboard set. The purpose of this set is so over the duration of the competition, competitors can see how they are performing relative to each other. These 23 submissions have no impact on the final prize winners.
A different data set, called the final data set is what actually determines the final prize winners. I submitted this file exactly one time. It was processed and scored by Kaggle and I followed the directions exactly as the competition admin stated. I did not check a checkbox because I did not believe it applied - I only submitted a single file and I recieved the 'all zeros' response from Kaggle's system as it said it would.
Also, the text Anthony quoted is not applicable to the Flight Quest competition either. This is the text that is used for the single phase competitions, not a two phase competition like Flight Quest.
If you haven't competed before I know the difference might not be clear. But if you read the response from the other competitors in the forum, you can see there is general agreement that the submission page is faulty for a two phase competition.