test: add simple e2e tests for AI SDK #75

dionlow · 2025-11-25T17:05:47Z

Summary

Add e2e test directory with simple connection verification tests
Implement test:simple script to verify checkConnection and checkInstallation tools
Document testing workflow in README with environment setup and local development instructions
Configure pnpm workspace to include test package for monorepo testing
Clean up vite.config.ts formatting for consistency

Testing Steps

Configure test environment variables:

cd test
cp .env.example .env  # Add your OPENAI_API_KEY, AMPERSAND_API_KEY, etc.

Build the SDK and run tests:

pnpm --filter @amp-labs/ai build
pnpm --filter ai-e2e-test test:simple

Verify test output shows successful connection checks for Salesforce

- add check installation test

Copilot

Pull request overview

This PR adds end-to-end testing infrastructure for the AI SDK with a focus on verifying basic tool functionality. The changes introduce a new test package within the pnpm workspace to validate checkConnection and checkInstallation tools using the Vercel AI SDK, along with documentation updates and code formatting improvements.

Add a new test directory with e2e tests for AI SDK tools
Configure pnpm workspace to include the test package for monorepo testing
Document testing workflow in README with setup and local development instructions
Clean up vite.config.ts formatting for consistency (single quotes, cleaner structure)

Reviewed changes

Copilot reviewed 6 out of 8 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
test/package.json	Defines test package with dependencies and test scripts (though some scripts reference missing files)
test/tsconfig.json	TypeScript configuration for the test package
test/ai-e2e-simple.ts	E2E test script to verify checkConnection and checkInstallation tools with OpenAI
pnpm-workspace.yaml	Adds test directory to workspace for monorepo integration
pnpm-lock.yaml	Adds dependency entries for the new test package
README.md	Documents testing setup, prerequisites, and workflow for running e2e tests
sdk/vite.config.ts	Formatting cleanup (single quotes, simplified define property)

Files not reviewed (1)

pnpm-lock.yaml: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-25T17:10:59Z

test/ai-e2e-simple.ts

+  console.log();
+
+  // comment out the following code to test the tool call (will use OpenAI credits)
+  // Test 1: Check connection - simple string parameter


Duplicate comment. The comment "Test 1: Check connection - simple string parameter" appears both on line 32 (as a standalone comment) and line 35 (in the code). Remove the duplicate on line 32.

Suggested change

// Test 1: Check connection - simple string parameter

Copilot · 2025-11-25T17:10:59Z

README.md

+- OpenAI API key (required for AI model)
+- Ampersand API credentials
+
+1. Configure environment variables in `test/.env`:


Missing .env.example file. The README and PR description mention cp .env.example .env but no .env.example file was added to the test directory. This file should be included to help developers set up the required environment variables.

Copilot · 2025-11-25T17:11:00Z

test/package.json

+    "test": "tsx ai-e2e-vercel-ai.ts",
+    "test:simple": "tsx ai-e2e-simple.ts",
+    "test:ts-node": "ts-node ai-e2e-vercel-ai.ts"


The package.json references a test file ai-e2e-vercel-ai.ts on lines 7 and 9, but this file does not exist in the test directory. Either add the missing file or remove these script entries.

Suggested change

"test": "tsx ai-e2e-vercel-ai.ts",

"test:simple": "tsx ai-e2e-simple.ts",

"test:ts-node": "ts-node ai-e2e-vercel-ai.ts"

"test:simple": "tsx ai-e2e-simple.ts"

Copilot · 2025-11-25T17:11:00Z

test/ai-e2e-simple.ts

+  console.log('- AMPERSAND_GROUP_REF:', process.env.AMPERSAND_GROUP_REF);
+  console.log();
+


Missing OPENAI_API_KEY validation. The test logs AMPERSAND_API_KEY, AMPERSAND_PROJECT_ID, and AMPERSAND_GROUP_REF but doesn't log or validate OPENAI_API_KEY, which is required according to the README. Add a check for OPENAI_API_KEY to help developers identify missing configuration.

Suggested change

console.log('- AMPERSAND_GROUP_REF:', process.env.AMPERSAND_GROUP_REF);

console.log();

console.log('- AMPERSAND_GROUP_REF:', process.env.AMPERSAND_GROUP_REF);

console.log('- OPENAI_API_KEY:', process.env.OPENAI_API_KEY ? 'SET' : 'NOT SET');

console.log();

// Validate required environment variables

if (!process.env.OPENAI_API_KEY) {

console.error('Error: OPENAI_API_KEY environment variable is not set. Please set it in your .env file.');

process.exit(1);

}

Copilot · 2025-11-25T17:11:00Z

test/tsconfig.json

+    "checkJs": false,
+    "types": ["node"]
+  },
+  "include": ["./**/*"],


[nitpick] The tsconfig.json has a broad include pattern ["./**/*"] which will include all files in the test directory, including potentially unwanted files like package-lock.json. Consider being more specific, e.g., ["**/*.ts"] to only include TypeScript files.

Suggested change

"include": ["./**/*"],

"include": ["**/*.ts", "**/*.js"],

Copilot · 2025-11-25T17:11:00Z

test/ai-e2e-simple.ts

+      checkInstallationResult.steps[0].toolCalls?.[0]?.toolName,
+    );
+    console.log(
+      '- Tool args:',
+      JSON.stringify(checkInstallationResult.steps[0].toolCalls?.[0]?.args),


Potential runtime error when accessing array index without checking if array exists or has elements. If checkInstallationResult.steps is empty or undefined, accessing steps[0] will cause a runtime error. Add a check before accessing the array.

Suggested change

checkInstallationResult.steps[0].toolCalls?.[0]?.toolName,

);

console.log(

'- Tool args:',

JSON.stringify(checkInstallationResult.steps[0].toolCalls?.[0]?.args),

Array.isArray(checkInstallationResult.steps) && checkInstallationResult.steps.length > 0

? checkInstallationResult.steps[0].toolCalls?.[0]?.toolName

: 'No steps available',

);

console.log(

'- Tool args:',

Array.isArray(checkInstallationResult.steps) && checkInstallationResult.steps.length > 0

? JSON.stringify(checkInstallationResult.steps[0].toolCalls?.[0]?.args)

: 'No steps available',

Copilot · 2025-11-25T17:11:01Z

test/ai-e2e-simple.ts

+async function testSimpleToolCall() {
+  console.log('Starting simple tool call test...\n');
+  console.log('Environment variables:');
+  console.log(
+    '- AMPERSAND_API_KEY:',
+    process.env.AMPERSAND_API_KEY ? 'SET' : 'NOT SET',
+  );
+  console.log('- AMPERSAND_PROJECT_ID:', process.env.AMPERSAND_PROJECT_ID);
+  console.log('- AMPERSAND_GROUP_REF:', process.env.AMPERSAND_GROUP_REF);
+  console.log();
+
+  // comment out the following code to test the tool call (will use OpenAI credits)
+  // Test 1: Check connection - simple string parameter
+
+  try {
+    // Test 1: Check connection - simple string parameter
+    console.log('Test 1: Checking connection for Salesforce...');
+    const checkResult = await generateText({
+      model: openai('gpt-4o-mini'),
+      tools: {
+        checkConnection: checkConnectionTool,
+      },
+      maxSteps: 5,
+      prompt: 'Check if there is an active connection for salesforce',
+    });
+
+    console.log('Check Connection Result:');
+    console.log('- Text:', checkResult.text);
+    console.log('- Tool Calls:', JSON.stringify(checkResult.steps, null, 2));
+    console.log('\n');
+
+    console.log('Test completed successfully!');
+  } catch (error) {
+    console.error('Error during test execution:', error);
+    process.exit(1);
+  }
+
+  // Test 2: Check installation - simple string parameter
+  try {
+    console.log('Test 2: Checking installation for Salesforce...');
+    const checkInstallationResult = await generateText({
+      model: openai('gpt-4o-mini'),
+      tools: {
+        checkInstallation: checkInstallationTool,
+      },
+      maxSteps: 5,
+      prompt: 'Check if there is an active installation for salesforce',
+    });
+
+    console.log('Check Installation Result:');
+    console.log('- Text:', checkInstallationResult.text);
+    console.log(
+      '- Tool called:',
+      checkInstallationResult.steps[0].toolCalls?.[0]?.toolName,
+    );
+    console.log(
+      '- Tool args:',
+      JSON.stringify(checkInstallationResult.steps[0].toolCalls?.[0]?.args),
+    );
+    console.log('\n');
+
+    console.log('Installation test completed successfully!');
+  } catch (error) {
+    console.error('Error during installation test execution:', error);
+    process.exit(1);
+  }


The test lacks assertions and doesn't verify the actual behavior. The test only logs outputs and doesn't validate that the tools work correctly (e.g., checking if checkResult.text contains expected values, or if found is true/false as expected). Consider adding assertions or converting this to a proper automated test with a testing framework.

Copilot · 2025-11-25T17:11:01Z

README.md

+
+   # Or from the test directory
+   cd test
+   npm run test:simple


[nitpick] Inconsistent command examples. Line 82 uses pnpm --filter ai-e2e-test test:simple but line 86 uses npm run test:simple. Since this is a pnpm workspace, it's better to be consistent and use pnpm commands. Consider changing line 86 to pnpm test:simple instead of npm run test:simple.

Suggested change

npm run test:simple

pnpm test:simple

Copilot · 2025-11-25T17:11:01Z

test/ai-e2e-simple.ts

+ * It uses the checkConnection and checkInstallation tools to check the connection and installation.
+ * It uses the generateText function to generate the text response.
+ *
+ * note: this test will use OpenAI credits


[nitpick] Inconsistent capitalization in comments. The docstring on line 17 says "note: this test will use OpenAI credits" with lowercase "note". Consider capitalizing "Note:" to match standard documentation style.

Suggested change

* note: this test will use OpenAI credits

* Note: this test will use OpenAI credits

Copilot · 2025-11-25T17:11:02Z

test/ai-e2e-simple.ts

+  console.log('- AMPERSAND_GROUP_REF:', process.env.AMPERSAND_GROUP_REF);
+  console.log();
+
+  // comment out the following code to test the tool call (will use OpenAI credits)


The comment is confusing and appears to be misplaced. It says "comment out the following code to test the tool call" but the code is not commented out. Consider either removing this comment or clarifying the instruction (e.g., "Uncomment the following code to skip test 1" if that's the intent).

Suggested change

// comment out the following code to test the tool call (will use OpenAI credits)

// To skip Test 1 and save OpenAI credits, comment out the following block.

dionlow · 2025-11-26T00:37:57Z

sdk/vite.config.ts


 export default defineConfig({
  define: {
-    'process.env': {


strips out the the other variables like the API key

Dion Low added 2 commits November 24, 2025 17:37

test: adds test script for basic connections

9370ccc

test: add simple test to README

928e5cd

- add check installation test

Copilot AI review requested due to automatic review settings November 25, 2025 17:05

Copilot started reviewing on behalf of dionlow November 25, 2025 17:06 View session

Copilot finished reviewing on behalf of dionlow November 25, 2025 17:09

Copilot AI reviewed Nov 25, 2025

View reviewed changes

dionlow commented Nov 26, 2025

View reviewed changes

sdk/vite.config.ts

export default defineConfig({

define: {

'process.env': {

Copy link

Author

dionlow Nov 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

strips out the the other variables like the API key

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test: add simple e2e tests for AI SDK #75

test: add simple e2e tests for AI SDK #75

Uh oh!

dionlow commented Nov 25, 2025 •

edited

Loading

Copilot AI left a comment

Copilot AI Nov 25, 2025

Copilot AI Nov 25, 2025

Copilot AI Nov 25, 2025

Copilot AI Nov 25, 2025

Copilot AI Nov 25, 2025

Copilot AI Nov 25, 2025

Copilot AI Nov 25, 2025

Copilot AI Nov 25, 2025

Copilot AI Nov 25, 2025

Copilot AI Nov 25, 2025

dionlow Nov 26, 2025

Labels

2 participants

		console.log('- AMPERSAND_GROUP_REF:', process.env.AMPERSAND_GROUP_REF);
		console.log();

	* note: this test will use OpenAI credits
	* Note: this test will use OpenAI credits

	// comment out the following code to test the tool call (will use OpenAI credits)
	// To skip Test 1 and save OpenAI credits, comment out the following block.

test: add simple e2e tests for AI SDK #75

Are you sure you want to change the base?

test: add simple e2e tests for AI SDK #75

Uh oh!

Conversation

dionlow commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing Steps

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Copilot AI Nov 25, 2025

Choose a reason for hiding this comment

Copilot AI Nov 25, 2025

Choose a reason for hiding this comment

Copilot AI Nov 25, 2025

Choose a reason for hiding this comment

Copilot AI Nov 25, 2025

Choose a reason for hiding this comment

Copilot AI Nov 25, 2025

Choose a reason for hiding this comment

Copilot AI Nov 25, 2025

Choose a reason for hiding this comment

Copilot AI Nov 25, 2025

Choose a reason for hiding this comment

Copilot AI Nov 25, 2025

Choose a reason for hiding this comment

Copilot AI Nov 25, 2025

Choose a reason for hiding this comment

Copilot AI Nov 25, 2025

Choose a reason for hiding this comment

dionlow Nov 26, 2025

Choose a reason for hiding this comment

Labels

2 participants

dionlow commented Nov 25, 2025 •

edited

Loading