Tests are part of source project.
Tests, which are pretty large, are currently part of the source distro. Consequently, checkouts are slow. Since most people don't develop on STARS, this is an unnecessary overhead for many people.
Requirements:
- A separate project/repo for tests and testing harness files.
- Large annotations files should not be committed. This places undue burden on the repo.
- Does it make sense to keep .exe files in the repo? We could integrate with zith (or a zith-like mechanism) to provide a reasonable suite of exe files without storing large files in the repo. Alternately, consider using "git lfs". Investigation of how this happens would be required.
- Testing should support understanding changes in annotations after changes in STARS.
- Testing should support command line switches, such as reduced mode, etc.
- Testing should support STARS/Ida and STARS/IRDB. Mechanism should be nominally extensible in case we want STARS/binja, STARS/objdump, etc.
- Tests should report a "pass/fail" metric. This may require some discussion. Here are some ideas to start said discussion:
- any failure to produce an annotation file -> fail
- all expected files exist, else fail
- thresholding on annotation file size?
- changes that change more than 10% of annotations -> fail?
- Specify which annotations are allowed to change w/o failure?