ci: split workflows and add E2E simulator run by mattleibow · Pull Request #36 · Redth/AppleDev.Tools

mattleibow · 2026-03-05T02:21:25Z

Summary

Splits the monolithic CI workflow into 3 focused workflows, parallelizes tests, and adds an end-to-end simulator test pipeline.

Changes

Workflow restructuring

build.yml — Renamed to "Build and Publish", removed test job, added dotnet-version: 10.0.x
test.yml (NEW) — Dedicated test workflow using matrix strategy to run each test project in parallel, with individual test result reporting via dorny/test-reporter
run.yml (NEW) — Comprehensive E2E simulator workflow

Test parallelization (`test.yml`)

Each test project runs as a separate matrix job (3 parallel runners instead of sequential)
Only the FbIdb job installs idb-companion (no wasted time for other jobs)
Each job publishes its own test results check

E2E simulator pipeline (`run.yml`)

Full app lifecycle following the Do/Validate pattern (modeled after AndroidSdk.Tools run.yml):

Create simulator → validate it appears in simulator list
Boot simulator → validate state is Booted
Install pre-built MAUI test app (extracted from committed zip) → validate via simulator app list
Launch app → validate E2E_APP_STARTED_SUCCESSFULLY marker found in simulator logs output
Screenshot → validate file exists and size > 1KB
Gather logs → validate log file created
Uninstall → validate app removed from list
Cleanup → delete simulator with --force

Every validation step uses only dotnet $TOOL CLI commands (no raw xcrun/simctl) — the whole point is to test the CLI tool itself.

Pre-built test app

AppleDev.Test/testdata/com.companyname.e2etestapp.zip (NEW) — Pre-built MAUI iOS simulator app, matching the Android approach of committing a pre-built APK. The app logs E2E_APP_STARTED_SUCCESSFULLY via Console.WriteLine in MainPage.OnAppearing.

Rename

AppleDev.Tool.Tests → AppleDev.Tool.Test — Consistent *.Test naming convention across all test projects

Bug fix

AppleDev.Tool/Extensions.cs — Fixed --output path validation: GetInvalidFileNameChars() → GetInvalidPathChars() so paths like artifacts/screenshot.png are accepted (the / separator was incorrectly rejected)

- build.yml: renamed to 'Build and Publish', contains only build/pack and publish jobs. Tests extracted to dedicated workflow. - test.yml: new 'Tests' workflow running all test suites (AppleDev, Tool.Tests, FbIdb) with test result reporting. - run.yml: new 'Run iOS Simulator' E2E workflow that creates a simulator, boots it, verifies it is running, and cleans up. Both build.yml and test.yml add explicit dotnet-version: 10.0.x. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR refactors the monolithic CI workflow (formerly build.yml) into three focused workflows: a pure build-and-publish workflow, a dedicated test workflow (test.yml), and a new end-to-end simulator workflow (run.yml). This fits cleanly into the repository's CI infrastructure.

Changes:

build.yml renamed to "Build and Publish", test job removed, explicit dotnet-version: 10.0.x added to both setup-dotnet steps.
test.yml (new) runs the three test projects on every PR and push to main, with its own concurrency group and dorny/test-reporter@v2 for publishing TRX results.
run.yml (new) exercises the CLI's simulator commands end-to-end: build → list → create → boot → verify → delete.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
`.github/workflows/build.yml`	Renamed to "Build and Publish"; removed test job; pinned `dotnet-version: 10.0.x` on both setup-dotnet steps
`.github/workflows/test.yml`	New dedicated test workflow with concurrency group, IDB Companion install, and TRX-based test reporting
`.github/workflows/run.yml`	New E2E simulator workflow that builds the CLI, creates/boots/verifies/deletes a simulator

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Follow the Do/Validate pattern from the Android SDK Tools run workflow. Uses MauiApp7.app test fixture for install/launch/uninstall validation. Captures screenshot and logs as uploadable artifacts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

E2E workflow now tests: create, boot, install (MauiApp7.app), launch (Settings), screenshot, logs, uninstall, delete. Follows Do/Validate pattern from AndroidSdk.Tools. Fix bug in Extensions.cs where --output path validation used GetInvalidFileNameChars() (which includes '/') instead of GetInvalidPathChars(), rejecting valid paths like 'artifacts/file.png'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The launch command already fails on error. The screenshot taken right after serves as real validation that the simulator is responsive. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Validate the app actually launched by checking system logs for the Preferences process and verifying it appears in the app list, with a retry loop matching the Android workflow pattern. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Instead of using a stub .app from testdata, build a real iOS app on CI using 'dotnet new ios' + 'dotnet build -r iossimulator-arm64'. Adds the ios workload install step. All Do/Validate steps now use the real app: install, launch (verified via app list), screenshot, logs, uninstall. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The .NET 10 iOS workload requires Xcode 26.2 but the runner defaults to Xcode 16.4. Add xcode-select step to pick the right version. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The iOS workload may expect a specific Xcode version (e.g. 26.2) while the runner has a newer one (26.3). Use -p:_XcodeVersionCheck=false to allow building the test app with a compatible but not exact Xcode match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Select the latest Xcode 26.x before installing the iOS workload, then run 'dotnet workload update' to ensure the workload version matches the selected Xcode. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The iOS workload may lag behind the runner's Xcode version. Use -p:ValidateXcodeVersion=false (from Xamarin.Shared.Sdk.targets) to allow building the E2E test app with a slightly newer Xcode. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The runner's Xcode 26.3 RC2 has incomplete SDKs and the .NET 10 iOS workload requires exactly Xcode 26.2. Build the E2E test app with .NET 9 and the default Xcode 16.4 which are compatible. The CLI tool still runs on .NET 10. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The ios template doesn't support --framework. Create with dotnet new, then sed the csproj to net9.0-ios for Xcode 16.4 compatibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Select Xcode 26.3, create MacOSX.sdk symlink if missing (RC builds may have versioned SDK names), skip Xcode version check for the build, and use net10.0-ios directly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The .NET iOS workload requires an exact Xcode version match (26.2) but the CI runner only has Xcode 26.3 RC2 with broken MacOSX.sdk. Switch to building a minimal native Obj-C iOS app with clang directly using the iphonesimulator SDK - no workload dependency needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add NSLog(E2E_APP_STARTED_SUCCESSFULLY) to the native test app and validate launch by grepping simulator logs for the marker in a retry loop — matching the Do/Validate pattern from AndroidSdk.Tools. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Use xcrun simctl spawn directly with --predicate filter and --info flag instead of going through the CLI tool, which may garble output in table format. Adds debug log dump on failure for easier troubleshooting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Use 'dotnet $TOOL simulator logs' with --format json and --predicate to validate the app launch marker. The E2E workflow should only use the CLI tool - that's the whole point of testing it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add a pre-built MAUI iOS simulator app (com.companyname.e2etestapp) to testdata as a zip, matching the Android approach of committing a pre-built APK. The app logs E2E_APP_STARTED_SUCCESSFULLY on MainPage.OnAppearing. The workflow now extracts the zip instead of building with clang, removing the Xcode version dependency entirely. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

….Test - Rename AppleDev.Tool.Tests -> AppleDev.Tool.Test for consistent *.Test.csproj naming - Update solution file, InternalsVisibleTo, and namespace - Split test workflow into matrix: each test project runs in parallel - Only install idb-companion for the FbIdb test job - Each matrix job publishes its own test results via dorny/test-reporter Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 9 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@v5

- Add explicit dotnet-version: 10.0.x to setup-dotnet in build.yml and test.yml - Update upload-artifact@v4 to @v5 in run.yml for consistency - Fix timestamp race condition: capture 5s before launch for log filtering - Fix log validation: check file size > 0, not just existence - Fix uninstall validation: remove || true that masked failures Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 9 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Add env.SIMULATOR_UDID != '' condition to uninstall/validate/cleanup steps so they skip gracefully if the simulator was never created. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

actions/setup-dotnet respects global.json for SDK version detection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mattleibow and others added 2 commits March 5, 2026 04:21

Fix jq property names to match lowercase JSON serialization

463cf31

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mattleibow requested a review from Copilot March 5, 2026 14:38

Copilot started reviewing on behalf of mattleibow March 5, 2026 14:38

View session

mattleibow added 2 commits March 5, 2026 16:38

Update build.yml

77d703a

Update build.yml

3ca1f62

Copilot AI reviewed Mar 5, 2026

View reviewed changes

Comment thread .github/workflows/run.yml Resolved

mattleibow and others added 20 commits March 5, 2026 21:48

Remove fake launch validation step

0e10fdd

The launch command already fails on error. The screenshot taken right after serves as real validation that the simulator is responsive. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Select Xcode 26.x for iOS workload compatibility

0558fc1

The .NET 10 iOS workload requires Xcode 26.2 but the runner defaults to Xcode 16.4. Add xcode-select step to pick the right version. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Select Xcode before workload install, add workload update

9402b71

Select the latest Xcode 26.x before installing the iOS workload, then run 'dotnet workload update' to ensure the workload version matches the selected Xcode. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove unsupported --sdk-version flag from workload install

0360aa3

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Retarget test app csproj to net9.0-ios via sed after dotnet new

2b3d219

The ios template doesn't support --framework. Create with dotnet new, then sed the csproj to net9.0-ios for Xcode 16.4 compatibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Update GitHub Actions workflow for testing

d17e798

mattleibow requested a review from Copilot March 6, 2026 19:51

Copilot started reviewing on behalf of mattleibow March 6, 2026 19:51

View session

Copilot AI reviewed Mar 6, 2026

View reviewed changes

Comment thread .github/workflows/run.yml Resolved

Comment thread .github/workflows/run.yml Resolved

mattleibow requested a review from Copilot March 6, 2026 21:24

Copilot started reviewing on behalf of mattleibow March 6, 2026 21:25

View session

Copilot AI reviewed Mar 6, 2026

View reviewed changes

Comment thread .github/workflows/run.yml Resolved

mattleibow and others added 2 commits March 6, 2026 23:38

fix: guard cleanup steps against empty SIMULATOR_UDID

4717dff

Add env.SIMULATOR_UDID != '' condition to uninstall/validate/cleanup steps so they skip gracefully if the simulator was never created. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fix: remove explicit dotnet-version, rely on global.json

ad4e5d2

actions/setup-dotnet respects global.json for SDK version detection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mattleibow merged commit d0011c4 into main Mar 6, 2026
5 of 6 checks passed

mattleibow deleted the pr/ci-restructure branch March 6, 2026 22:22

mattleibow mentioned this pull request Mar 6, 2026

Enhance simulator commands: create UDID, boot format, delete force, JSON compact #28

Closed

Navigation Menu

Repository navigation

ci: split workflows and add E2E simulator run - #36

ci: split workflows and add E2E simulator run#36
mattleibow merged 27 commits into
mainfrom
pr/ci-restructure

mattleibow commented Mar 5, 2026 •

edited

Loading

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Labels

2 participants

Conversation

mattleibow commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Workflow restructuring

Test parallelization (test.yml)

E2E simulator pipeline (run.yml)

Pre-built test app

Rename

Bug fix

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Labels

2 participants

mattleibow commented Mar 5, 2026 •

edited

Loading

Test parallelization (`test.yml`)

E2E simulator pipeline (`run.yml`)