Adding Validation and Eval Gates to CI with WIQD

A validation workflow becomes an engineering system when it gives the same answer on a laptop and in a pull request. Work IQ Developer Tools makes that possible without translating its lifecycle into a pile of grep, sed, and hopeful string matching: every command has a non-interactive JSON mode, a shared output envelope, and a stable exit-code contract.

The hard part is not writing the YAML. It is deciding which code is trusted enough to receive a Microsoft 365 identity.

Write the Gate Policy First

This is the policy I want reviewers to approve before anyone opens .github/workflows/wiqd.yml:

Gate	When	Network or tenant	Blocks
Environment diagnostics	Every run	No tenant required in the offline job	A broken runner from masquerading as a project failure
Static manifest validation	Every pull request and push to main	Offline	Merge
Package build	Every pull request and push to main	Offline	Merge
Deep validation	Trusted main or an approved manual run	Dev endpoints and development tenant	Promotion
Smoke eval suite	After deep validation	Pre-provisioned development agent only	Promotion
Provision or publish	Never in this workflow	Not applicable	Nothing

The job boundary is a security boundary, not an optimization. The offline job checks out and executes pull-request code with contents: read and no secrets. The tenant-backed job runs only after code reaches trusted main, targets a protected GitHub environment, and consumes the package produced by the offline job. It validates and evaluates an agent that already exists in development; it does not provision, share, or publish anything.

That split also keeps failures intelligible. A schema error belongs to the pull request. An endpoint timeout belongs to the protected environment. Mixing both in one job turns every red check into an investigation.

Pin the Runtime and the Contract

WIQD is a pre-release preview, so pinning is part of correctness:

env:
  NODE_VERSION: "24.15.0"
  WIQD_VERSION: "0.12.2"

The package currently requires Node 24.15 or newer. Pin the exact Node and WIQD versions together, then upgrade them in a dedicated pull request after reading wiqd changelog --from 0.12.2. Do not install @microsoft/wiqd@latest in a merge gate: a preview release can change a command-specific data payload while your repository is unchanged.

The WIQD output contract guarantees these top-level JSON envelopes:

{
  "status": "success",
  "command": "agent.validate",
  "data": {}
}

{
  "status": "error",
  "command": "agent.validate",
  "exitCode": 1,
  "error": {
    "code": "COMMAND_SPECIFIC_CODE",
    "message": "Actionable failure message"
  }
}

Parse only status, command, data, exitCode, and error plus fields documented for the pinned command. Never parse the human table, and never infer success from an empty array. The process exit code owns pass or fail:

Exit	Meaning	Pipeline response
0	Command succeeded	Continue
1	Input, upstream, or postcondition failure	Block the gate; fix the project, endpoint, or quality result
2	Configuration or infrastructure failure	Block the gate; repair the runner, provider session, or configuration
130	Cancelled	Report cancellation; do not present it as a product regression

The eval input is versioned separately. Keep its schemaVersion in source control, review changes to it, and pass a selected smoke file rather than discovering prompts dynamically during CI.

The Two-Stage Workflow

The workflow below uses immutable action SHAs, exact tool versions, least-privilege permissions, timeouts, and one concurrency group per ref. Replace the example development secrets with the non-interactive authentication settings documented by the providers your WIQD extensions use.

name: WIQD quality gates

on:
  pull_request:
    paths:
      - "appPackage/**"
      - "evals/**"
      - "m365agents.yml"
      - ".github/workflows/wiqd.yml"
  push:
    branches: [main]
    paths:
      - "appPackage/**"
      - "evals/**"
      - "m365agents.yml"
      - ".github/workflows/wiqd.yml"
  workflow_dispatch:

permissions:
  contents: read

concurrency:
  group: wiqd-${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

env:
  NODE_VERSION: "24.15.0"
  WIQD_VERSION: "0.12.2"

jobs:
  offline:
    name: Offline validation and package
    runs-on: ubuntu-24.04
    timeout-minutes: 10

    steps:
      - name: Check out source
        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2

      - name: Set up Node
        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: npm

      - name: Install pinned WIQD
        run: npm install --global "@microsoft/wiqd@${WIQD_VERSION}"

      - name: Collect runner diagnostics
        continue-on-error: true
        run: |
          mkdir -p artifacts
          wiqd doctor --json > artifacts/doctor.json

      - name: Validate manifests offline
        run: |
          wiqd agent validate \
            --mode static \
            --json > artifacts/static-validation.json

      - name: Build the package
        run: |
          wiqd agent package \
            --env ci \
            --output artifacts/agent.zip \
            --json > artifacts/package.json

      - name: Upload offline reports
        if: always()
        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
        with:
          name: wiqd-offline-reports
          path: artifacts/*.json
          if-no-files-found: error
          retention-days: 7

      - name: Upload validated package
        if: success()
        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
        with:
          name: wiqd-agent-package
          path: artifacts/agent.zip
          if-no-files-found: error
          retention-days: 7

  protected-dev:
    name: Protected development validation and eval
    needs: offline
    if: github.event_name == 'push' || github.event_name == 'workflow_dispatch'
    runs-on: ubuntu-24.04
    timeout-minutes: 25
    environment: wiqd-development

    env:
      M365_ACCOUNT_NAME: ${{ secrets.WIQD_DEV_ACCOUNT_NAME }}
      M365_ACCOUNT_PASSWORD: ${{ secrets.WIQD_DEV_ACCOUNT_PASSWORD }}
      M365_TITLE_ID: ${{ secrets.WIQD_DEV_TITLE_ID }}
      AZURE_OPENAI_ENDPOINT: ${{ secrets.WIQD_DEV_AZURE_OPENAI_ENDPOINT }}
      AZURE_OPENAI_API_KEY: ${{ secrets.WIQD_DEV_AZURE_OPENAI_API_KEY }}

    steps:
      - name: Check out trusted source
        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2

      - name: Set up Node
        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: npm

      - name: Install pinned WIQD
        run: npm install --global "@microsoft/wiqd@${WIQD_VERSION}"

      - name: Download validated package
        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
        with:
          name: wiqd-agent-package
          path: artifacts

      - name: Verify provider sessions
        run: |
          mkdir -p artifacts
          wiqd auth status --json > artifacts/auth-status.json
          wiqd doctor --json > artifacts/doctor.json

      - name: Deep-validate the package
        run: |
          set +e
          wiqd agent validate \
            --mode deep \
            --env dev \
            --package-file artifacts/agent.zip \
            --json > /tmp/deep-validation.raw.json
          code=$?
          set -e
          jq 'walk(
            if type == "object" then
              with_entries(select(.key | test(
                "token|secret|password|credential"; "i"
              ) | not))
            elif type == "string" then
              gsub("https?://[^[:space:]\"]+"; "[REDACTED_URL]")
            else . end
          )' /tmp/deep-validation.raw.json > artifacts/deep-validation.json
          exit "$code"

      - name: Run the bounded smoke suite
        run: |
          set +e
          wiqd agent eval \
            --env dev \
            --config evals/smoke.json \
            --concurrency 2 \
            --threshold 0.85 \
            --json > /tmp/eval.raw.json
          code=$?
          set -e
          jq 'walk(
            if type == "object" then
              with_entries(select(.key | test(
                "token|secret|password|credential|prompt|response"; "i"
              ) | not))
            elif type == "string" then
              gsub("https?://[^[:space:]\"]+"; "[REDACTED_URL]")
            else . end
          )' /tmp/eval.raw.json > artifacts/eval.json
          exit "$code"

      - name: Write a concise job summary
        if: always()
        run: |
          {
            echo "## WIQD protected-development gates"
            for report in artifacts/deep-validation.json artifacts/eval.json; do
              if jq -e . "$report" >/dev/null 2>&1; then
                jq -r '"- `\(.command)`: **\(.status)**"' "$report"
              fi
            done
            echo
            echo "### Validation errors"
            jq -r '
              .data.diagnostics[]?
              | select(.severity == "error")
              | "- `\(.file // "project"):\(.line // 1)` — \(.code // "validation error")"
            ' artifacts/deep-validation.json 2>/dev/null | head -20
          } >> "$GITHUB_STEP_SUMMARY"

      - name: Upload redacted protected reports
        if: always()
        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
        with:
          name: wiqd-protected-dev-reports
          path: |
            artifacts/deep-validation.json
            artifacts/eval.json
          if-no-files-found: warn
          retention-days: 3

There are three deliberate omissions:

1. No pull_request_target. It would put base-repository secrets near code controlled by the pull request. Use pull_request for untrusted CI and keep privileged automation separate.
2. No provisioning or publishing. The eval targets the title ID of an already provisioned development agent. Promotion changes evidence; deployment changes state. They should not share a job.
3. No raw tenant-backed output in logs or artifacts. Stdout goes to a temporary file, known credential fields, URLs, prompts, and responses are removed, and only the redacted file is uploaded.

The summary deliberately prints codes and source locations, not diagnostic messages or model responses. Reviewers get a file to fix without receiving a tenant URL, access token, prompt, or grounded answer.

Configure the Protected Environment

Create a GitHub environment named wiqd-development and require reviewers. Scope its identity to a development tenant and only the agent, grounding sources, endpoints, and model deployment needed by the smoke suite. Keep production consent and production content out of reach.

The environment should provide:

• A dedicated development identity using a provider-supported non-interactive authentication method.
• The title ID of an already provisioned development agent.
• Eval-provider configuration for the selected evaluators.
• Network access to the development endpoints exercised by deep validation.

Do not put values in env/.env.dev.user, echo them into $GITHUB_ENV, cache a provider token directory, or upload auth-status.json. Add .env*.user, raw reports, token caches, and local WIQD state to .gitignore.

If your organization uses self-hosted runners, make them ephemeral and deny unrelated internal network access. Environment approval does not make a reused machine clean.

Three Failures, Three Recoveries

WIQD does not invent one numeric exit code for every diagnostic. That would make scripts brittle. The process code decides whether the gate passes; the documented envelope tells you what failed.

1. The manifest violates its schema

Static validation exits 1. The agent.validate envelope points to the manifest diagnostic, so the offline job annotates the pull request and blocks merge. Reproduce it without credentials:

wiqd agent validate --mode static --json
echo $?
# 1

Fix the manifest, rerun the same command, and expect 0. Do not rotate the manifest version just to silence a rule; validation must preserve the project’s intent.

2. Deep validation cannot reach a dependency

Deep validation also fails the gate, but its command and diagnostics identify reachability rather than schema. An unreachable endpoint is unknown, not an invalid manifest. Confirm the development runner’s DNS, firewall, and endpoint health, then retry from a host with the intended network path.

If WIQD cannot attempt the command because a required provider, project, or configuration is absent, it exits 2 instead. That is a runner incident. Do not “fix” project files and do not convert it into a skipped green check.

3. The smoke suite misses its quality bar

The eval run completes, calculates its pass rate, and exits 1 because it is below --threshold 0.85:

wiqd agent eval \
  --env dev \
  --config evals/smoke.json \
  --concurrency 2 \
  --threshold 0.85 \
  --json
echo $?
# 1

The recovery is to inspect the protected report, classify the failure as grounding, reasoning, or action selection, and repair the agent. Do not lower the threshold in the same pull request that changes the agent. Treat the eval suite as the specification, not as an obstacle to green CI.

Reproduce the Gates Locally

Use the same pinned versions as CI:

nvm install 24.15.0
nvm use 24.15.0
npm install --global @microsoft/[email protected]

wiqd doctor --json
wiqd agent validate --mode static --json
wiqd agent package --env ci --output artifacts/agent.zip --json

The tenant-backed checks require access to the approved development environment:

wiqd auth status --json
wiqd agent validate \
  --mode deep \
  --env dev \
  --package-file artifacts/agent.zip \
  --json
wiqd agent eval \
  --env dev \
  --config evals/smoke.json \
  --concurrency 2 \
  --threshold 0.85 \
  --json

A pull request can merge only after static validation and packaging pass. A merge to main can be promoted only after protected deep validation and the smoke suite pass against development. Deployment and publishing remain explicit downstream operations with their own approval, identity, and concurrency boundary.

That is the transformation WIQD enables: the local lifecycle does not get rewritten for CI. It gets pinned, isolated, and run unattended.

The Value You Just Unlocked

• One quality contract everywhere: The same pinned WIQD commands decide whether a change is ready on a laptop and in GitHub Actions.
• Fast feedback without tenant risk: Every pull request gets static validation and packaging while untrusted code stays isolated from Microsoft 365 credentials.
• Evidence before promotion: Deep validation and a bounded smoke suite prove that the trusted package still works against its development dependencies.
• Failures with clear owners: Exit codes and structured JSON separate project defects, quality regressions, and runner incidents.
• Reviewable security boundaries: Protected environments, least-privilege identities, and redacted artifacts make access to tenant-backed checks explicit.
• A pipeline that validates without deploying: Quality gates produce promotion evidence without quietly provisioning, sharing, or publishing an agent.

CI is no longer a second implementation of your agent lifecycle. It is the same lifecycle made repeatable, inspectable, and safe enough to run unattended.

Resources

• WIQD CLI command reference
• WIQD exit codes and JSON output contract
• WIQD authentication guidance
• Data, privacy, and security for Microsoft 365 Copilot extensibility
• Secure use reference for GitHub Actions
• Securely using pull_request_target
• The declarative agent lifecycle with WIQD
• Generating evals with WIQD