Is your feature request related to a problem? Please describe.
Current evals implementation is serial, and takes up a long time if using a complex evaluator / large dataset

Describe the solution you'd like
Add a batch option on the CLI to batch multiple samples together when running evaluation (not necessarily inference, since this may interfere with the trace collection process). This has already been implemented as POC in 5bded06

Additional Requirements:

• have better span names,
• link a metric to the corresponding span in the evaluator trace.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
5bded06