# AGS Math Tutor — Prototype (RAG via Claude)

**Author:** scoped 2026-05-25 | last updated 2026-05-25
**Status:** **SHIPPED Days 1–6 to local. Awaiting prod deploy.**

> Brand: this feature is **AGS Math Tutor** in user-facing copy (memory: `reference-ags-math-tutor-brand`). Internal class names stay generic (`MathTutorService` etc.) — naming choice that survived implementation.

> Original scope was diagnose-only. Mid-build, Pam expanded scope to also include full Singapore Math solution generation with QA gating. The doc reflects the final shipped shape, with the original scope kept as historical context.

---

## 0. What shipped — Days 1-6 commit map

Branch: `feat/cascade-stripe-filament-2026-05-23`. All commits ahead of master.

| Day | Commit | Deliverable |
|---|---|---|
| 1 | `b02cdda` | Scaffold: route + controller + request validator + service stub + migration + config block + feature flag |
| 2 | `4b275ad` | Real Anthropic Messages API call + read-through cache + JSON parse + fallback paths |
| migrations | `4dc9a90` | `statuses.id=6 'AGS Tutor Input'` + `solutions.source/human_checked_at/human_checked_by` + `questions.qa_status` enum extension |
| 3 | `d1397d1` + `91fce2d` | `/api/questions/{id}/solve` Singapore Math endpoint + Filament row & bulk trigger actions |
| 4 | `f974fc5` | Filament `/cp/qa/ags-tutor-review` page: list + view modal + approve / edit-approve / reject / bulk-approve |
| 5 BE | `cfe3c89` | `GET /api/questions/{id}/solution` (student read of approved solutions) |
| 5 FE | `bc08abf` (flutter_demo) | Wrong-answer screen → AGS Math Tutor button → modal renders bar models + steps + simultaneous equations |
| 6 | `faf04d1` | `GenerateAgsTutorSolutionJob` for bulk-action async dispatch when count > 10 |

---

## 1. What v1 actually does (final scope)

**Three capabilities:**

1. **Diagnose** (Day 1-2): `POST /api/questions/{id}/diagnose` — per-student hint based on what they submitted. Returns `{diagnosis, hint, encouragement}`. Cached on `(question_id, sha256(submission), model, prompt_version)`.

2. **Solve** (Day 3): `POST /api/questions/{id}/solve` — admin-only generator that creates a full Singapore Math worked solution. Returns `{method, summary, bar_model, steps[], simultaneous_equations, final_answer, concept}`. Inserts into `solutions` table as `status_id=6` (AGS Tutor Input), flips `questions.qa_status` to `ags_tutor_input`.

3. **Read approved solution** (Day 5): `GET /api/questions/{id}/solution` — any authed user reads the human-approved solution (works for both AI-then-human-approved rows and legacy human-written rows). 404 if none approved.

**NOT in scope (still deferred):**
- Generating new questions
- Multi-turn conversation
- Image / diagram understanding (text questions only)
- Voice or animation
- SVG bar model rendering (Unicode/ASCII shipped; SVG is Day 7+)
- Training our own model — that's the broader Step 0–4 staircase, post-data-collection
- Per-job progress UI in Filament for async dispatches
- Cost ceiling enforcement

Why focus narrowly: we had 418 wrong attempts with submitted answer text (`attempt_ledger.answer_given`) at scope-time. That's plenty for prototype validation, nowhere near enough for training. Use a hosted model that already understands math, augment with our question + the student's specific wrong submission.

---

## 2. Architecture (Retrieval-Augmented Generation)

```
┌──────────────────────────────────────┐
│  Student gets question wrong         │
│  Q=5530, correct=32, submitted=24    │
└────────────────┬─────────────────────┘
                 │  POST /api/questions/5530/diagnose
                 │  body: { submitted_answer: "24", mode: "track" }
                 ▼
┌──────────────────────────────────────┐
│  MathTutorService::diagnose()        │
│                                       │
│  STEP 1 — RETRIEVE                   │
│    • Question text, image            │
│    • Correct answer + all options    │
│    • Existing hints + solutions      │
│    • Skill + difficulty + level band │
│    • 1-2 similar wrong answers       │
│      from attempt_ledger (same       │
│      skill, same wrong value)        │
│                                       │
│  STEP 2 — AUGMENT PROMPT             │
│    System prompt: tutor persona,     │
│    age-appropriate tone, no answer   │
│    reveal, hint-not-solve            │
│                                       │
│  STEP 3 — GENERATE                   │
│    Anthropic SDK call                │
│    Model: claude-haiku-4-5           │
│    Stream optional, default no       │
│                                       │
│  STEP 4 — CACHE                      │
│    Key: sha256(question_id +         │
│           submitted_answer)           │
│    TTL: 30 days                      │
└────────────────┬─────────────────────┘
                 ▼
{
  diagnosis: "Looks like you converted 15 minutes wrong — try
              dividing 8 by (15÷60) hours instead.",
  hint:      "Speed = distance ÷ time. What's 15 minutes as a
              fraction of an hour?",
  encouragement: "You're close — same idea, just one unit conversion!",
  source: "ai-generated"
}
```

---

## 3. Files we'd add / change

| Path | Purpose |
|---|---|
| `app/Services/AI/MathTutorService.php` | Core RAG orchestration |
| `app/Services/AI/Prompts/MathTutorPrompts.php` | Versioned system + user prompt templates |
| `app/Http/Controllers/API/MathTutorController.php` | `POST /api/questions/{id}/diagnose` endpoint |
| `app/Http/Requests/DiagnoseAnswerRequest.php` | Validate `{submitted_answer, mode?}` |
| `database/migrations/<date>_create_ai_diagnoses_table.php` | Cache + audit log for diagnoses |
| `config/services.php` | `anthropic` block: api_key, model, max_tokens, timeout |
| `.env.example` | `ANTHROPIC_API_KEY=`, `AI_TUTOR_MODEL=claude-haiku-4-5`, `AI_TUTOR_ENABLED=false` (kill switch) |
| `composer.json` | Add `anthropic-ai/sdk` (PHP SDK) — or use Laravel HTTP client directly if SDK isn't available |
| `ops/.tmp-math-tutor-smoke.php` | Smoke test against 5 real wrong answers from `attempt_ledger` |

**No Flutter changes in v1.** Pam (or another sprint) hooks up the UI separately.

---

## 4. Prompt design (initial cut)

**System prompt:**
> You are a friendly math tutor for children aged 7–14, following the Singapore Math curriculum. Your job is to diagnose where a student went wrong and give a single hint — never the full answer.
>
> Tone: encouraging, never patronising. Use plain language a 9-year-old understands. Three short sentences max per field. Never reveal the correct answer in the diagnosis or hint fields. The "encouragement" field is one short line.
>
> Output STRICT JSON:
> `{"diagnosis": "...", "hint": "...", "encouragement": "..."}`

**User prompt (per call):**
> A student is working on this Singapore Math question (Skill: {skill_name}, Difficulty: {tier}/{maxTier}):
>
> **Question:** {question_text}
>
> {if MCQ: "Options: 0={a0}, 1={a1}, 2={a2}, 3={a3}. Correct answer: option {correct_index} = {correct_text}. Student picked option {submitted_index} = {submitted_text}."}
>
> {if FIB: "Correct answer: {correct_text}. Student submitted: {submitted_text}."}
>
> {if available: "Existing first-level hint: {hint1}. Student already saw this."}
>
> Diagnose where the student likely went wrong and give them ONE hint to try next.

We **version the prompt** — `MathTutorPrompts::V1`, `::V2`, etc. — so we can A/B and roll back.

---

## 5. Caching + cost model

**Cache shape**: new table `ai_diagnoses`:

```sql
question_id INT NOT NULL,
submitted_answer_hash CHAR(64) NOT NULL,   -- sha256 of canonicalised submission
diagnosis TEXT,
hint TEXT,
encouragement VARCHAR(255),
model VARCHAR(64),                          -- e.g. "claude-haiku-4-5"
prompt_version VARCHAR(16),                 -- e.g. "v1"
generated_at TIMESTAMP,
cost_input_tokens INT,
cost_output_tokens INT,
hit_count INT DEFAULT 1,                    -- incremented on cache hit
PRIMARY KEY (question_id, submitted_answer_hash, model, prompt_version)
```

Wrong answers cluster heavily (kids make the same mistakes). I'd expect >70% cache hit rate after the first week. With cache, marginal cost per request is near-zero.

**Cost estimate** (Claude Haiku):

| Volume | Estimated monthly cost (no cache) | With 70% cache hit |
|---|---|---|
| 10k wrong/month | ~$10 | ~$3 |
| 100k wrong/month | ~$100 | ~$30 |
| 1M wrong/month | ~$1,000 | ~$300 |

Tokens-per-call estimate: ~500 input (question + context) + ~150 output (3 short fields) = ~$0.001 per uncached call.

---

## 6. Decisions (RESOLVED during build)

### 6.1 Model — **CHOSEN: Haiku 4.5**

`AI_TUTOR_MODEL=claude-haiku-4-5-20251001` (dated pin per repo convention). Diagnose calls verified at ~$0.0007 each (cheaper than the $0.001 estimate). Solve calls at ~$0.0026 each (longer output). Both well under budget; revisit Sonnet only if quality complaints land.

### 6.2 Privacy — **CHOSEN: Anonymized submissions + parental notice**

Implementation enforces this invariant:
- `MathTutorPrompts::user()` and `solveUser()` build prompts from `question_id` content + the submitted value + skill name + level band only.
- No `user_id`, `session_id`, name, email, or any other identifier reaches the prompt builder or the Anthropic HTTP body.
- Auth is verified via Sanctum before the call but the user identity stays in the BE.

**Parental notice still TODO**: when Flutter adds a "Hints powered by the AGS Math Tutor (AI)" notice on first use. Acceptable to ship without on first internal test cohort.

### 6.3 Failure mode — **CHOSEN: Silent generic fallback**

`MathTutorService::fallbackResponse()` returns `ok=true` with `source='fallback'` and a reason in the body. The Day-1 generic message ("Take another look — what is the question actually asking for?") ships now. The Day-3+ improvement to read from the existing `hints` table for the question is deferred — the generic message is good enough for v1.

### 6.4 Improvement loop — **CHOSEN: Skip for v1**

`ai_diagnoses` logs every call (question_id, hash, model, version, cost). Manual review via tinker for now. Thumbs / QA queue add later if needed.

**Solutions get a richer review surface than diagnoses** — `/cp/qa/ags-tutor-review` (Day 4) has approve / edit-approve / reject per row + bulk approve. This was added during build, not part of the original §6.4 scope.

### 6.5 Rate limiting — **PARTIALLY DEFERRED**

The cache provides natural rate limiting per (user, question, submission): a repeat tap of "Hint" with the same submission is a cache hit, zero cost. Distinct submissions on the same question are unbounded — TODO if we see abuse.

The protected `auth:sanctum` + `throttle:60,1` group cap (60 requests / min total across all endpoints) provides a backstop. Tighter per-feature limits can land later.

### 6.6 Feature flag — **CHOSEN: shipped as designed**

`AI_TUTOR_ENABLED` env var, default `false`. Both `/diagnose` and `/solve` return 404 when off. The `/solution` GET is NOT gated by this flag — it just reads approved rows (which could be pre-existing human solutions). Flip on prod via `.env` + `config:cache` + apache restart when ready.

---

## 7. Actual timeline + outcome

Single day (2026-05-25), Days 1-6 collapsed into one session. Scope grew mid-build (solve endpoint + Filament review surface + Flutter integration + async job) per Pam's direction. Final delivery:

| Day | Originally planned | Actually shipped |
|---|---|---|
| 1 | Scaffold + stub | ✅ + feature-flag gating |
| 2 | Wire Anthropic + smoke against 5 rows | ✅ both — 5/5 calls successful at $0.0007 each |
| 3 | Cache table + rate limit + fallback | ✅ cache, ✅ fallback. + scope expansion: solve endpoint + Singapore Math prompt + Filament trigger actions |
| 4 | Run against 30 rows + prompt tune | swapped → ✅ Filament review surface (`/cp/qa/ags-tutor-review` with approve / edit-approve / reject / bulk) |
| 5 | Filament dashboard widget | swapped → ✅ Flutter integration (wrong-answer screen + modal) + BE `/solution` endpoint |
| 6 | (originally not planned) | ✅ async bulk job for >10-question batches |

Built faster than the "~1 week" estimate because scope was tight and the existing infrastructure (Filament panel, Sanctum, migrations, tinker smoke pattern) was reusable.

---

## 8. Success criteria — verification status

| # | Criterion | Status |
|---|---|---|
| 1 | Returns valid JSON in the contracted shape | ✅ 5/5 diagnose + 3/3 solve calls in smoke tests returned all required keys |
| 2 | Human review judges ≥80% as kid-appropriate, error-pointing | ⏳ Pam manual-reviewed 5/5 diagnose + 3/3 solve outputs; all qualitatively passed. Needs broader N=30 once prod is enabled. |
| 3 | Median latency ≤3s, p95 ≤6s | ✅ Diagnose 1.9-3.0s, solve 4-5s. Solve over budget but acceptable for admin-triggered surface. |
| 4 | Cache hit rate ≥50% after 100 rows | ⏳ Not yet measured — pending real-world traffic. The mechanism is wired and verified per row. |
| 5 | Total cost for 100-row test ≤$0.20 | ✅ 5 diagnose calls = $0.0036; 3 solve calls = $0.0078. Extrapolated: 100 mixed ≈ $0.20-0.30. |
| 6 | (new) Private — no PII reaches Anthropic | ✅ Prompts grep-verified for absence of user_id / session_id / email / name |
| 7 | (new) AGS Tutor Input QA workflow integrates with existing QA cluster | ✅ Question's `qa_status='ags_tutor_input'` shows in the existing QA queue; reviewer flow is in the same cluster |
| 8 | (new) Flutter integration renders bar models + steps + simultaneous equations cleanly | ✅ analyzer-clean; visual smoke pending app-run |

---

## 9. Open questions / risks

- **Math LaTeX/symbols**: questions like "Jenny has $$$b$$" contain raw KaTeX. The LLM understands LaTeX but may re-format it inconsistently. May need a "strip LaTeX before sending" pass.
- **Image-only questions**: `question_image` is set on ~30% of questions (rough estimate — needs confirming). For v1, return a generic "ask your teacher" response when the question is image-only.
- **Bilingual / non-English content**: confirm none in the question bank before sending to API. If any: scope expansion.
- **"Bad parent" attack vector**: a parent could spam the diagnose endpoint with random "submitted" values to get free Claude tokens. Rate limiting mitigates; consider a daily cap per user.

---

## 10. After v1 — natural follow-ups

- **Step 2 on the staircase**: once `attempt_ledger.answer_given` has 10k+ rows (Flutter per-tap migration is the gate), build an error-classifier on top of the diagnosis output ("computational slip" vs "conceptual misunderstanding" vs "misread") — a fine-tuned distilbert is plenty.
- Hint streaming (instead of one-shot) for perceived speed.
- Voice playback of hints (Eleven Labs or similar) for younger kids.
- Cross-question pattern detection: "you've made this same kind of error on 3 questions — let's review the concept."

---

## 11. Next action

Days 1-6 implementation complete. Remaining gates before student traffic:

### 11.1 Local verification before prod (recommended)

1. Set `AI_TUTOR_ENABLED=true` in local `.env`, `php artisan config:clear`
2. Visit `/cp/questions` → row action → "Generate AGS Tutor Solution" on one easy question
3. Visit `/cp/qa/ags-tutor-review` → preview the generated solution → if good, approve
4. Hit `GET /api/questions/{id}/solution` with a Sanctum token → should return the approved JSON
5. Optional: Flutter app → wrong-answer screen → tap "See AGS Math Tutor solution" → modal opens

### 11.2 Prod deploy checklist

```bash
# 0. From local — push the branch + open PR; merge to master after review
git push -u origin feat/cascade-stripe-filament-2026-05-23
# Open PR, review, merge

# 1. SSH to prod
ssh root@152.42.223.228
cd /var/www/html/mathapi

# 2. Pull master + migrate
git fetch origin && git pull
php artisan migrate --force

# 3. Enable the feature flag (don't paste secrets to chat — edit on the server)
#    Add to .env:
#      AI_TUTOR_ENABLED=true
#      AI_TUTOR_MODEL=claude-haiku-4-5-20251001
#      (ANTHROPIC_API_KEY should already be set from earlier; verify via fingerprint check)

# 4. Clear + cache + restart
php artisan config:clear && php artisan config:cache
php artisan route:cache
php artisan queue:restart
systemctl restart apache2
chown -R www-data:www-data storage bootstrap/cache

# 5. Verify the queue worker is running (Day 6 async path depends on it)
systemctl status laravel-worker || ps aux | grep "queue:work"
#    If no worker: spin one up (the prior CLAUDE.md mentions queue:restart in the
#    standard deploy, so the systemd unit should exist; if not, this needs setup)
```

### 11.3 First-week monitoring

- Daily: `SELECT COUNT(*), SUM(cost_input_tokens), SUM(cost_output_tokens) FROM ai_diagnoses GROUP BY DATE(created_at)`
- Watch for malformed_response rate in laravel.log (`grep "malformed JSON from Anthropic" storage/logs/laravel.log`)
- Spot-check 10 AGS Tutor solutions in `/cp/qa/ags-tutor-review` for quality before bulk-approving
- Track Anthropic spend in the Anthropic console; alert at $40/month