Assignment Level Feedback

# Assignment Level Feedback

After a teacher grades student submissions on an assignment, AI reviews all of the individual feedback and grades in aggregate and surfaces a condensed summary of class-wide strengths, weaknesses, and recommended next steps — giving the teacher an actionable picture of where their students are, without attributing struggles to individual students.

## Current status

Draft/planning `yawp-2.0` PR #104 is open. Kevin's May 7 email frames this as prototype-first: draft a preview, demo it to Brian, then decide whether it is ready for engineering. Bryant's current architecture bias is to generate summaries in a background job triggered by grading milestones, not inside the interactive grade-save/tutor path.

## Problem

Teachers already get individual-level feedback through the tutor/integrating assistant, but that data lives submission-by-submission. After grading a full class set, a teacher has no quick way to see: *What did my students tend to get right? Where did they collectively fall short? What should I reteach?* They'd have to re-read every submission or trust memory to synthesize that picture. At 150+ students (UA professor), that's not realistic.

## Goals

- Give teachers an AI-generated class-level summary per assignment: strengths, weaknesses, and suggested follow-up actions.
- Surface this summary in context — on the assignment view, near the student grade list, so it's visible right when the teacher is thinking about the assignment.
- Make the summary actionable but suggestive in tone — pointing to two or three high-leverage skills or rubric components to focus on, not prescribing specific lesson plans. Teachers know how to teach; the summary surfaces *where* to focus, not *how*.
- Keep the summary anonymous — trends only, no student attribution.

## Non-goals

- **No cross-class aggregation.** The summary covers exactly one Assignment (one class section, one assignment). A teacher with 3rd period and 5th period both writing the Macbeth essay gets a separate summary per class — never merged. Cross-section comparison is a natural future ask but explicitly punted from this scope.
- **No cross-assignment aggregation.** This is not a longitudinal view of how a class trends over time. That's a separate, future scope.
- **No summary history.** Only the latest summary is stored; regenerating overwrites the previous one.
- No student-level identification in the summary (that's the individual tutor assistant's job).
- Not a replacement for individual feedback or grades — purely a layer on top.
- No real-time/live update as submissions come in — generates on demand (or after all/most submissions are graded).

## Domain notes

- **Assignment** — the instance of an assignment given to students in a course. This is the anchor for the summary; one summary per Assignment.
- **AssignmentType** — the reusable template. Relevant because assignment-specific rubric context (e.g., "this was a close-reading essay on Macbeth") should inform what the AI looks at.
- **Submission** — each student's graded snapshot. The AI reads across all Submissions linked to the Assignment. Specifically: grades, rubric scores if present, and teacher comment text.
- **Course** — scopes which students belong to this assignment, which matters for the aggregate.

The AI has access to the same individual feedback data that the tutor assistant generates; this feature reads that data and folds it up one level.

## UX sketch

The summary lives on the **per-assignment overview page** introduced in [clickable-assignment-titles.md](clickable-assignment-titles.md) — the tabbed page a teacher lands on when they click an assignment title (or status badge) in the class's Assignments tab. The panel sits between the assignment header and the Submitted/Graded/Released/In Progress tab strip, so it's visible no matter which tab is selected (the summary describes the assignment as a whole, not any single status slice).

```
Assignment overview page:
+------------------------------------------------------------------+
| Grade 11 • Period 3                                              |
| St. Joe's Prep                                                   |
+------------------------------------------------------------------+
| < Back to assignments                                            |
|                                                                  |
| Daily Pages: Year End Review                                     |
| Daily Pages   Due 5/19/2026                                      |
|                                                                  |
| +-------------------------------------------------+              |
| | Class Summary  [Regenerate]                     |              |
| |                                                 |              |
| | Strengths                                       |              |
| |  • Strong use of textual evidence overall       |              |
| |  • Clear understanding of Act 1 themes          |              |
| |                                                 |              |
| | Weaknesses                                      |              |
| |  • Thesis statements often too broad            |              |
| |  • Conclusions didn't link back to thesis       |              |
| |  • Evidence cited but not analyzed (quote-drop) |              |
| |                                                 |              |
| | Suggested focus areas                           |              |
| |  • Thesis specificity                           |              |
| |  • Conclusions that link back to the thesis     |              |
| |  • Analyzing evidence (vs. quote-dropping)      |              |
| +-------------------------------------------------+              |
|                                                                  |
| [ Submitted ]  Graded   Released   In Progress                   |
| ──────────────                                                   |
|                                                                  |
| [ tab content — student list for the selected status ]           |
+------------------------------------------------------------------+
```

- Panel is collapsed by default on first load; expanded once a summary has been generated. Collapsed/expanded state persists per teacher per assignment.
- The panel is tab-agnostic — it stays put while the teacher switches between Submitted/Graded/Released/In Progress.
- **Generation triggers.** Summary auto-generates the first time the teacher crosses 50% graded, then auto-regenerates at 75% and 90% milestones. The "Regenerate" button is always available on-demand once the 50% threshold is reached — visually it "colors in" / activates at 50%, grayed-out before that.
- Below 50% graded: panel shows the threshold prompt — "Grade more submissions to get a meaningful class summary. X of Y graded." — and the Regenerate button is inactive.
- Empty state when no submissions exist yet: panel not shown.
- Loading state: spinner with "Reviewing submissions…" while the AI generates.
- The "Suggested focus areas" section surfaces two or three high-leverage skills or rubric components the AI noticed — phrased as opportunities the teacher might want to lean into (e.g., "Suggested focus area: thesis specificity"), never as instructions or prescribed lesson plans. The framing should consistently feel like the AI is *pointing something out*, not telling the teacher what to do.

## Data model implications

- New table or column to store the generated summary per Assignment — probably an `assignment_class_summaries` table with `assignment_id` (unique), `generated_at`, `graded_at_generation` (count, so the UI can tell the teacher how stale the summary is), `summary_json` (structured: strengths[], weaknesses[], focus_areas[]).
- Only the latest summary per Assignment is stored — regeneration overwrites in place. No history table. If we later decide to keep history, that change should be additive (append-only rows on the same table) — design with that future shape in mind so it's not a destructive migration.
- Backward compat: additive only, no changes to existing Assignment or Submission tables.
- `summary_json` schema should be versioned so we can evolve the structure without a migration.

## File paths in `yawp-2.0` likely to change

Anchored to the assignment overview route introduced in [clickable-assignment-titles.md](clickable-assignment-titles.md):

- `services/web-app/app/routes/app.my-classes.$courseId.$assignmentId/route.tsx` — host the summary panel above the tab strip.
- `services/web-app/app/routes/app.my-classes.$courseId.$assignmentId/components/class-summary-panel.tsx` — new `ClassSummaryPanel` component.
- `packages/db/` — new `assignment_class_summaries` table migration.
- AI/LLM service layer (wherever the tutor assistant calls live) — new prompt + endpoint for aggregate summarization, invoked from the background job that fires on grade-save milestones.

## Open questions

- [ ] Bryant: Where does the AI call live — same service as the tutor assistant, or a separate job? Latency matters if it's synchronous on page load; auto-generation at 50/75/90 milestones could happen as a background job triggered on grade-save.

### Resolved

- [x] **Generation thresholds.** 50% graded to start; auto-regen at 75% and 90% milestones. Regenerate button is always present and activates once 50% is crossed (grayed-out before that).
- [x] **Auto-generate vs. on-demand.** Both — auto at 50/75/90, plus on-demand Regenerate button thereafter.
- [x] **Summary history.** Latest only — regeneration overwrites in place. No history table.
- [x] **Tone.** Suggestive — surface 2-3 high-leverage focus areas (skills or rubric components), not prescribed lesson plans.
- [x] **Cross-section comparison.** Explicitly punted from this scope. One Assignment = one summary; sections never merged.

## Edge cases

- Teacher hasn't graded any submissions yet — panel not shown.
- Below 50% graded — show threshold prompt, don't generate a potentially misleading summary; Regenerate button inactive.
- Crossing 50%, 75%, 90% milestones — auto-(re)generates once per milestone per Assignment so the teacher doesn't get spammed by repeat regenerations on every grade-save in between.
- Small classes (e.g., 3-student tutorial) — 50% may be only 2 submissions. Consider a minimum absolute floor (e.g., at least 3 graded) before auto-generating, regardless of percentage.
- Teacher re-grades after summary is generated — summary becomes stale; "Regenerate" is the escape hatch and the milestone auto-regen will catch the next threshold crossing.
- Students in multiple sections of the same AssignmentType — summary should scope to the specific Assignment instance, not aggregate across sections (teachers own their section view).

## Test plan

- Unit: AI prompt produces well-structured output (strengths / weaknesses / focus areas — 2-3 focus areas, suggestive tone) given sample submission + comment data.
- Unit: Summary not generated when graded count is below 50% (and below the small-class absolute floor).
- Unit: Auto-regen fires exactly once per 50/75/90 milestone, not on every grade-save in between.
- Integration: Generating a summary upserts into `assignment_class_summaries` (latest only — no history rows) and is returned on subsequent page loads without re-calling the AI.
- Integration: Crossing 50%, then 75%, then 90% on a single Assignment produces three distinct summary updates; intermediate grade-saves do not.
- Manual QA: Load an assignment with a full set of graded submissions; verify the summary panel appears, content is sensible, and Regenerate works.
- Manual QA: Load an assignment with 0%, sub-50%, and just-crossed-50% states; verify panel UI and Regenerate-button activation.
- Regression: Existing assignment and submission views unchanged.

## Rollout

**Pre-handoff gate.** Bryant drafts a preview PR from this spec; Kevin demos it to Brian for approval before this spec flips to ready-for-engineering. The draft PR exists to validate the summary's content quality and panel placement against a real graded assignment — not to ship. Brian's sign-off is the gate to formal engineering handoff.

Feature flag (`assignment_level_feedback`). Roll out to UA professor first — highest volume, most meaningful test of the summary quality at scale. Then Washington and Birmingham City after validating summary quality and performance. Default flag OFF until pilot validation.

## Engineering handoff checklist

- [ ] Domain context covered
- [ ] File paths in `yawp-2.0` listed
- [ ] Data model implications spelled out, including backward-compat plan
- [ ] UX sketch in prose
- [ ] Edge cases enumerated
- [ ] Test plan written
- [ ] Rollout plan decided