Skip to content

Conversation

@ssbushi
Copy link
Contributor

@ssbushi ssbushi commented Aug 12, 2024

The context array in the eval-flow command currently holds all context across multiple traces. Since we only support non-durable flows in evals, we should actually just use the first context element, which corresponds to the the context array extracted from the first trace.

Before:
image
After:
image

@ssbushi ssbushi changed the title [Evals] Make context actually a string array. Aug 12, 2024
@ssbushi ssbushi requested a review from MichaelDoyle August 12, 2024 16:48
@ssbushi ssbushi changed the title [Evals] Make context use the right content from the extracted values. Aug 12, 2024
@ssbushi ssbushi requested a review from shrutip90 August 13, 2024 15:26
@ssbushi ssbushi merged commit 33e9e19 into main Aug 13, 2024
@ssbushi ssbushi deleted the sb/fixContextField branch August 13, 2024 21:15
ssbushi added a commit that referenced this pull request Aug 13, 2024
i2amsam added a commit that referenced this pull request Aug 16, 2024
* Update index.md (#764)

* test(go): add live tests to go ollama plugin (#720)

* test(go): add ollama live test

* test(go): retrieve port info for test container

* chore(go): refactor ollama plugin live test

* test(go): remove docker from ollama live test

* fix context array (#777)

* Update to pnpm v9.7.1 (#786)

---------

Co-authored-by: Peter Friese <peter@peterfriese.de>
Co-authored-by: Jacob Cable <32874567+cabljac@users.noreply.github.com>
Co-authored-by: ssbushi <66321939+ssbushi@users.noreply.github.com>
Co-authored-by: Anthony Barone <tonybaroneee@gmail.com>
cabljac added a commit that referenced this pull request Sep 16, 2024
* feat: do not allow defining new actions from within other actions/flows (#725)

* refactor: [JS] introduced a generate utility action to make generate veneer logic reusable (#759)

Co-authored-by: Michael Bleigh <bleigh@google.com>

* refactor: consolidated registry into a class, made registry hierarchical (#639)

* Add DatasetStore interfaces for evals datasets (#781)

* [Fix] Apply #777 to `next` branch (#779)

* Merge main to Next (#792)

* Update index.md (#764)

* test(go): add live tests to go ollama plugin (#720)

* test(go): add ollama live test

* test(go): retrieve port info for test container

* chore(go): refactor ollama plugin live test

* test(go): remove docker from ollama live test

* fix context array (#777)

* Update to pnpm v9.7.1 (#786)

---------

Co-authored-by: Peter Friese <peter@peterfriese.de>
Co-authored-by: Jacob Cable <32874567+cabljac@users.noreply.github.com>
Co-authored-by: ssbushi <66321939+ssbushi@users.noreply.github.com>
Co-authored-by: Anthony Barone <tonybaroneee@gmail.com>

* [Eval] Breaking change -- Modify EvalRunKey (#755)

* Makes inputSchema optional for tools. (#822)

* Add datasetId field to eval run key (#827)

* Fixes history rendering of Dotprompt system role. (#837)

* Local file based DatasetStore implementation and Tools API changes (#797)

* feat(js/plugins/ollama): add initial embedding support

* feat(js/plugins/ollama): integrate ollama embeddings into plugin proper

* Update js/plugins/ollama/src/embeddings.ts

Co-authored-by: Pavel Jbanov <pavelj@google.com>

---------

Co-authored-by: Pavel Jbanov <pavelj@google.com>
Co-authored-by: Michael Bleigh <bleigh@google.com>
Co-authored-by: shrutip90 <shruti.p90@gmail.com>
Co-authored-by: ssbushi <66321939+ssbushi@users.noreply.github.com>
Co-authored-by: Sam Phillips <samphillips@google.com>
Co-authored-by: Peter Friese <peter@peterfriese.de>
Co-authored-by: Anthony Barone <tonybaroneee@gmail.com>
Co-authored-by: Michael Doyle <michaeldoyle@google.com>
cabljac added a commit to invertase/genkit that referenced this pull request Nov 25, 2025
* feat: do not allow defining new actions from within other actions/flows (firebase#725)

* refactor: [JS] introduced a generate utility action to make generate veneer logic reusable (firebase#759)

Co-authored-by: Michael Bleigh <bleigh@google.com>

* refactor: consolidated registry into a class, made registry hierarchical (firebase#639)

* Add DatasetStore interfaces for evals datasets (firebase#781)

* [Fix] Apply firebase#777 to `next` branch (firebase#779)

* Merge main to Next (firebase#792)

* Update index.md (firebase#764)

* test(go): add live tests to go ollama plugin (firebase#720)

* test(go): add ollama live test

* test(go): retrieve port info for test container

* chore(go): refactor ollama plugin live test

* test(go): remove docker from ollama live test

* fix context array (firebase#777)

* Update to pnpm v9.7.1 (firebase#786)

---------

Co-authored-by: Peter Friese <peter@peterfriese.de>
Co-authored-by: Jacob Cable <32874567+cabljac@users.noreply.github.com>
Co-authored-by: ssbushi <66321939+ssbushi@users.noreply.github.com>
Co-authored-by: Anthony Barone <tonybaroneee@gmail.com>

* [Eval] Breaking change -- Modify EvalRunKey (firebase#755)

* Makes inputSchema optional for tools. (firebase#822)

* Add datasetId field to eval run key (firebase#827)

* Fixes history rendering of Dotprompt system role. (firebase#837)

* Local file based DatasetStore implementation and Tools API changes (firebase#797)

* feat(js/plugins/ollama): add initial embedding support

* feat(js/plugins/ollama): integrate ollama embeddings into plugin proper

* Update js/plugins/ollama/src/embeddings.ts

Co-authored-by: Pavel Jbanov <pavelj@google.com>

---------

Co-authored-by: Pavel Jbanov <pavelj@google.com>
Co-authored-by: Michael Bleigh <bleigh@google.com>
Co-authored-by: shrutip90 <shruti.p90@gmail.com>
Co-authored-by: ssbushi <66321939+ssbushi@users.noreply.github.com>
Co-authored-by: Sam Phillips <samphillips@google.com>
Co-authored-by: Peter Friese <peter@peterfriese.de>
Co-authored-by: Anthony Barone <tonybaroneee@gmail.com>
Co-authored-by: Michael Doyle <michaeldoyle@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants