Research is Difficult

It takes years of training and mentorship to understand an industry, learn what's important in a field, how to organize a team, how to plan and forecast research, what combination of activities produce research, and how that research earns the currency of the realm, both financial and social. I still don't claim to have a handle on all of that, and I admit I've been slow to learn. But the picture is starting to take shape in the final year of my PhD.

Maybe I'll blog later about the other aspects mentioned above, but now I want to focus on just one research activity: experimentation. A lot of data science and ML papers focus on showing that a method can be useful without thorough proofs on the distribution of problem class. And that's fine. So long as methods are well-tested, the empirical approach can and should convince people, and convincing people is one of the currencies of the realm.

I'd like to put together my thoughts on what makes experiments good and convincing. The basic point is "engineering and collaboration", but lets see the technical forms that takes.