I’m now a semester into my Ph.D. at Brandeis, and after a couple of projects, I’ve now got some thoughts on a suitable research workflow. Note that this is simply what works for me at this time, and may change in the future.
What was worth it
- Experiment management software – Guild.AI
- The TUI for
guild compare is great, for example
- Scripts with verbose arguments
- Do:
--input, --output, --mode
- Don’t:
-i, -o, -m
- Use click for creating CLIs
- Coreutils + command line tools like
grep, sed, awk, cut, paste, ...
- In particular, the above tools are very, very useful inside little shell scripts
- Infrastructure-as-code is suprisingly worthwhile (e.g.
guild.yml)
- Symlinking is also very worthwhile
- Create an
experiments folder that is the sole entry point for data on that experiment
- Inside each experiment folder, simply symlink to data / checkpoints etc.
- This way, any script only needs to navigate to the folder of that particular experiment
What I never found the use case for
- A lot of examples seem to have train & evaluation done in the same script
- This is often not true when you train with one metric & evaluate using another.
- Case in point: evaluating MT / seq2seq outputs decoded using beam search
- I also haven’t yet felt the need for
cookiecutter or similar tools that produce boilerplate code.