SparkTrace

Agentic Spark pipeline debugging copilot

This is a showcase deployment — the loop is real, the models are stubbed.

Everything you are about to watch actually executes: the planner loop picks each next action, the generated SQL is real SQL, it is checked by the safety guard, and it runs against the bundled scenario. What is not running here is Claude — a deterministic reasoner stands in for the five model tiers so this page needs no API keys, no AWS account, and can be opened by anyone without spending anything.

Each step below is tagged with the model tier that handles it in live mode. Point the kit at your own pipeline repo and Athena workgroup, and the same orchestration runs against real infrastructure — nothing about the loop changes.

Start an investigation

Describe the data symptom you're seeing and point SparkTrace at the pipeline repo, or run the bundled demo scenario offline.
bundled demo scenario
demo

How SparkTrace works

A data engineer debugging a silent pipeline regression spends most of their time on the same loop: form a guess, write a query to test it, read the result, guess again. The knowledge is rarely the bottleneck — the round trips are. SparkTrace runs that loop as an agent, and keeps a human-readable record of why it did each thing.

A planner decides every turn

There is no scripted plan. Each turn the planner sees the symptom, the pipeline, and the evidence gathered so far, then picks one action: read the repo, generate a diagnostic query, or conclude. An investigation that finds its answer in two steps stops in two steps.

Two layers the model cannot skip

Every generated query passes a deterministic guard before execution — read-only, no exceptions, rejections are recorded and never run. Every result is compacted to a bounded digest before the analyst sees it, so a query returning a million rows can never blow up a prompt.

The right model per step, not one big model

Deciding what to investigate is hard reasoning. Reading a digest and saying what it shows is not. Paying top-tier rates for both is how agent demos become too expensive to run in production.
  • PlannerOpus 4.8sparktrace-planner

    Decides the next investigative action every turn from the evidence so far. This is the only genuinely open-ended judgement in the loop, so it is the only step that earns the top tier.

  • Repo readerSonnet 5sparktrace-repo-reader

    Reads pipeline source and extracts the relevant mechanism. Long context and careful reading, but the question is already framed by the planner.

  • Query generatorSonnet 5sparktrace-query-gen

    Writes read-only diagnostic SQL against a known schema — a constrained translation task, and its output is checked by the deterministic guard before anything runs.

  • AnalystHaiku 4.5sparktrace-analyst

    Reads a compacted result digest — never raw rows — and states what it shows. Small, bounded input; the cheapest tier is the correct tier.

  • ReporterSonnet 5sparktrace-reporter

    Synthesizes the accumulated evidence chain into a root-cause report a data engineer can act on. Quality of writing matters; novel reasoning does not.

What changes in live mode

The orchestrator only ever talks to interfaces, so live mode swaps four implementations and changes nothing else: the deterministic reasoner becomes the five deployed Lamatic flows, the bundled CSVs become your Athena workgroup, the fixed schema becomes your Glue catalog, and the sample scenario becomes a clone of your pipeline repo.

That is also why this page can exist. The same loop you are watching is the loop that runs in production — it is being fed by fixtures instead of by AWS.