Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

The only person who really understood how to do independent validation. Of course the code MUST be different.


Why is it not a good idea to first redo with the original code?

Also doing an independent implementation doesn't mean you should start there - what if it doesn't replicate with the same code?


It could be a nice quick sanity check, in the sense that if they've completely lied or you've completely misunderstood how to use the program, you won't get the same results. So it could tell you that you shouldn't even bother trying to replicate their claims. But there's a risk that people might mistake re-running the code for reproducing the findings of the paper.

The paper is entered into the scientific record, not the code, which will inevitably become obsolete (old language, old frameworks, implementation details tied to old hardware or operating systems, maybe some source code will be lost, github will go out of business). If the code is necessary, then crucial details have been left out of the paper, so it is not reproducible (although there are some journals that let you submit code as an artifact).

Pseudocode is better, IMO.


The risk here is that the original code contains a bug that is inadvertently reproduced by the replicating scientists after reading it. It can easily happen in some fields that some computation looks very plausible in code, but is actually incorrect.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: