Daily Pages holistic GA

# Daily Pages holistic GA

A separate Grading Assistant pipeline for Daily Pages submissions that scores completion and engagement with ideas, not grammar, syntax, or essay structure. Selected at GA-time by AssignmentType, runs alongside the existing thesis-essay GA without changing it.

## Problem

Daily Pages are exploratory writing — students respond to a prompt to think on the page. The existing GA scores them with the same rubric as a thesis essay (thesis quality, evidence, mechanics), which:

- Penalizes the writing for not being something it was never trying to be. A strong, idea-rich Daily Page that's a little rough on syntax scores low under the thesis-essay rubric.
- Teaches the wrong lesson. Students learn that Daily Pages "don't really count" or that the goal is to clean up mechanics, defeating the pedagogical point.
- Produces feedback that's useless to the teacher. A grade out of 100 with "weak thesis, evidence underdeveloped" tells the teacher nothing about whether the student engaged with the prompt.
- Wastes the signal. Daily Pages happen every day; they are the highest-volume source of student thinking in the platform. Grading them on mechanics throws away the signal that should feed the student-level insights work.

Teachers currently work around this by ignoring GA grades on Daily Pages or not submitting them for grade at all. The Submit-for-Grade toggle spec gives them the *option* to leave Daily Pages ungraded, but a teacher who *wants* a grade still has nothing usable. This spec fixes that.

## Goals

- Produce a holistic Daily Pages grade that rewards completion + engagement with ideas.
- Produce feedback in a register that matches the assignment: encouraging, curious, idea-forward — never mechanics-policing.
- Generate structured signal per submission (themes engaged with, depth of engagement, prompt fit) that downstream features (student-level insights, class-level summaries) can read.
- Slot cleanly into the existing GA pipeline: same I/O shape, selected by AssignmentType, no special-casing in the rest of the system.

## Non-goals

- Replacing or modifying the existing thesis-essay GA. That pipeline stays exactly as-is.
- Grading mechanics, grammar, or essay structure on Daily Pages. Mechanics issues are explicitly out of scope — the holistic GA ignores them.
- Detecting effort fraud (very short submissions, copy-paste, off-prompt rambling). Worth a follow-up but not in v1; teacher override is the v1 escape hatch.
- Building the student-level insights view that consumes this signal — separate spec, downstream.
- Cross-submission longitudinal analysis (how a student's engagement has trended over weeks). Separate scope.
- Cross-assignment grade weighting or category math inside Yawp. That stays in the teacher's gradebook of record.

## Domain notes

- **AssignmentType** drives pipeline selection. `AssignmentType = "daily-pages"` (or whatever the canonical slug is in `yawp-2.0`) routes to this GA; everything else stays on the existing GA.
- **Submission** is the input. The GA reads the prompt (from the parent Assignment) and the student's response text. No prior submissions, no comparison against peers.
- **GA output** has the same shape as the existing GA — grade percentage, comments — so the submission view and gradebook sync don't need to branch on which GA ran. New: a structured `engagement_signal` payload (see Data model) that downstream features can read but isn't shown directly to students.
- **Teacher override** preserved as-is: any grade or comment is editable before release.
- **No AI branding in user-facing UI** (platform rule). Feedback reads as the teacher's voice — curious, encouraging — never "the AI noticed…".

## What "holistic" means here

The scoring rubric, in plain words:

- **Completion** — Did the student actually engage with the prompt? Length is a weak proxy; substance is the real signal. A four-sentence response that wrestles with the prompt scores higher than two paragraphs of throat-clearing.
- **Engagement with the idea** — Does the response take a position, complicate the prompt, connect to lived experience, or push past the obvious first answer? This is the heart of the score.
- **Originality of thought** — Not "novel" in a publishable sense; "not the response a generic student would give." A surprising angle, a personal connection, an unexpected complication.
- **Prompt fit** — Did the student actually respond to *this* prompt, or did they drift into something unrelated? Drift is fine if they wrestle with it productively; pure off-prompt isn't.

Explicitly NOT in the rubric:

- Grammar, spelling, punctuation, capitalization.
- Sentence-level style, voice consistency, tone.
- Essay structure — thesis, topic sentences, transitions, conclusions.
- Citation, evidence, source use.
- Length minimums beyond what's needed to demonstrate engagement.

The feedback voice should match: comments lean toward "what's interesting about what you wrote" and "here's a thread you might pull on next time" — not "consider revising for clarity."

## UX sketch

The student-facing and teacher-facing UI is identical to today's GA output — same panels, same grade slot, same comment surface. The only differences are the score model behind it and the voice of the comments. No new screens.

Where this *does* show up visually is in the submission view's grade summary. For Daily Pages submissions:

```
Daily Pages — Year End Review (5/19)
Grade: 9 / 10        (holistic)        [ Override ]

What's working
  • Real wrestling with the prompt — the move from "I want
    to be remembered" to "I want to be misremembered" is
    the kind of thinking that essays grow out of.
  • Concrete details ground the abstraction (the band photo,
    the cracked phone).

A thread to pull next time
  • You raised the misremembered idea and then dropped it.
    Worth coming back to — what would it take to want that?
```

Notes:

- Grade displayed as `points / point_value` per the Submit-for-Grade spec.
- A small `(holistic)` tag next to the grade signals this used the Daily Pages pipeline. Teacher-only; students don't see the tag.
- Comment shape: a "What's working" section and a "Thread to pull next time" section. No "weaknesses." No mechanics call-outs.
- Override button identical to the existing GA flow.

## Current model caveat

Production does not currently have per-assignment `submit_for_grade` or `point_value` fields. Submission/grading availability is controlled by document-submission feature flags at the school/global level, and the current `Assignment` row stores class, AssignmentType, title, prompt, tutor context, and due date.

The earlier submit-for-grade toggle idea may still be right, but it is not already a settled production capability. Treat graded/ungraded assignment intent and configurable point values as a model decision to make alongside [Daily Pages AssignmentType model](daily-pages-assignment-type-model.md), not as a prerequisite silently assumed by this GA spec.

May 26 update: the team discussed point value as a broader assignment-creation requirement, not only a Daily Pages requirement. Brian's direction on the call was that `Submit for grade` should default checked, with the teacher able to uncheck it, and that point values should support low-stakes assignments such as Daily Pages being worth 5 or 10 points instead of 100. See [Assignment creation standardization](assignment-creation-standardization.md).

## Graded vs. ungraded assignments and configurable point values

Working hypothesis if we decide per-assignment grading intent belongs in the model:

- **Graded/ungraded control.** A "Submit for Grade" checkbox on the assignment creator lets teachers decide whether an assignment records a formal grade. May 26 call direction: checked by default, teacher can uncheck. If unchecked, GA can still give feedback, but no grade is recorded.
- **Configurable point values.** Teachers set the point value per assignment (default 100, max 1000, positive integer). The displayed grade is `GA_percentage × point_value`. The native GA percentage stays stored alongside so re-scaling is non-destructive if the teacher edits the point value.
- **Possible new columns on `assignments`:** `submit_for_grade BOOLEAN NOT NULL DEFAULT FALSE` and `point_value INTEGER NULL` (only meaningful when `submit_for_grade = TRUE`).
- **Possible new columns on submissions:** `ga_percentage NUMERIC NULL` (native GA score) and `displayed_grade NUMERIC NULL` (derived: `ga_percentage × point_value`, rounded).
- **Backward compat requirement:** Any new columns must be additive/default-safe. Existing thesis-essay behavior must remain unchanged.
- **Grade display when graded:** `180 / 200 (GA: 90%) [ Override ]`
- **Grade display when ungraded:** `Feedback ready. No grade recorded.`

## Data model implications

- No new tables required for the GA itself — output slots into the existing GA result storage.
- New structured payload on each Daily Pages GA result: `engagement_signal` (JSON). Versioned. Shape (v1 working draft):

  ```json
  {
    "version": 1,
    "themes_engaged": ["identity", "memory"],
    "engagement_depth": "high",
    "originality": "moderate",
    "prompt_fit": "strong",
    "notable_moves": [
      "complicated the prompt",
      "personal connection"
    ]
  }
  ```

  Enum values: `engagement_depth` ∈ {low, moderate, high}; `originality` ∈ {low, moderate, high}; `prompt_fit` ∈ {weak, partial, strong}.

- Stored alongside the existing GA result row, not in a separate table. Nullable for existing rows and for non-Daily-Pages submissions.
- Versioning the schema means downstream consumers (student-level insights) can evolve without forcing a migration on historical data.
- Backward compat: purely additive. Existing GA results untouched.

## File paths in `yawp-2.0` likely to change

- `services/grading-assistant/` — pipeline selection branching on AssignmentType. Likely a router-style entry point that dispatches to the existing GA or the new one.
- `services/grading-assistant/prompts/daily-pages-holistic.ts` (new) — the prompt, with the rubric described above and the voice/tone guardrails.
- `services/grading-assistant/scoring/daily-pages.ts` (new) — score computation from model output to a percentage.
- `services/grading-assistant/types.ts` — extend the GA result type to include the optional `engagement_signal` payload.
- `packages/db/...` — additive migration for the `engagement_signal` column on the GA result table.
- `services/web-app/app/routes/app.submissions.$submissionId/...` — small UI change: the `(holistic)` teacher-only tag and the "What's working / Thread to pull" comment shape. Also: graded vs. ungraded display path.
- `services/web-app/app/routes/app.my-classes.$courseId.assignments.new/route.tsx` — "Submit for Grade" checkbox + point value input in the assignment creator form.
- `services/web-app/app/components/assignment-form/...` — form schema + validation for the grading fields.

## Open questions

- [ ] **Owner.** Who builds and trains this — Bryant, or a new contributor? The prompt work is meaty and benefits from someone who can iterate against real submissions; the engineering plumbing is straightforward but needs the same person to close the loop.
- [ ] Brian: Should there be a minimum grade floor for any submitted Daily Page (e.g., never below 60%) to reinforce that the assignment is low-stakes? Or does that flatten the signal we want?
- [ ] Brian: What's the right grade for a clearly off-prompt response — score it on engagement with whatever they *did* write, or penalize for missing the prompt?
- [ ] Calibration set — how many graded Daily Pages submissions (with Kevin's or Brian's ground-truth scoring) do we need to validate the prompt before flag-on? Propose 30–50 spanning low/medium/high engagement.
- [ ] Should the `engagement_signal` payload be visible to teachers in any form (e.g., a small "themes engaged" chip), or strictly internal for now? Default: strictly internal in v1.
- [ ] Detection of effort fraud (one-word submissions, copy-paste, off-prompt drift) — v1 leans on teacher override; is a low-confidence flag worth adding?
- [ ] Brian: Rounding precision on the displayed grade — whole points, or one decimal (e.g., 90.5 / 100)?
- [ ] Bryant: Confirm gradebook integration handles non-100 point scales without special-casing.
- [ ] Bryant: Confirm whether per-assignment grading intent/point value belongs in this model at all, given production currently gates submission/grading by school feature flags.
- [ ] Brian/Bryant: Confirm May 26 direction that submit-for-grade defaults ON globally, with assignment-type suggested point values for low-stakes work.
- [ ] Should the point value input remember the teacher's last-used value per AssignmentType, or always default to 100?
- [ ] If a teacher unchecks "Submit for Grade" on an assignment that already has graded submissions, what happens to the existing grades — preserved, hidden, or blocked with a confirmation?

## Edge cases

- Very short submission (one sentence, a few words): GA still runs but flags low engagement_depth. Teacher override is the recovery path.
- Off-prompt submission: scored on what they wrote, with `prompt_fit: weak` in the signal payload. Teacher decides whether to override.
- Empty submission: GA does not run; grade slot reads "No submission."
- Mechanics-heavy student work that's idea-rich: the holistic GA should score well regardless of mechanics. Calibration set should include examples like this.
- Mechanics-clean student work that's idea-thin: should score modestly. Calibration set should include examples like this.
- Submission in a language other than English (rare but possible in pilot schools): out of scope for v1; falls back to ungraded with a teacher-visible note.
- Daily Pages submission where the Submit-for-Grade toggle is unchecked: GA still runs, produces the holistic feedback comments, but no grade is recorded. Same I/O shape minus the grade field.
- Existing Daily Pages submissions graded under the old (thesis-essay) GA: leave as-is. Re-grading is a teacher-initiated action via override or re-submission, not a background backfill.

## Test plan

- Unit: pipeline router selects the holistic GA when `AssignmentType = daily-pages`, the existing GA otherwise.
- Unit: holistic GA prompt produces well-structured output (grade + "what's working" + "thread to pull" + engagement_signal) on a synthetic submission.
- Unit: scoring function maps model output to a percentage in `[0, 1]`.
- Unit: `engagement_signal` payload validates against the v1 schema.
- Integration: a Daily Pages submission flowing through the GA produces a holistic grade, holistic comments, and an engagement_signal row; a thesis-essay submission is byte-identical to today's GA output (regression).
- Calibration: against a 30–50 submission calibration set with Kevin/Brian ground-truth scores, the holistic GA agrees within an acceptable band (target TBD — propose ±10% on grade, qualitative review on comments).
- Manual QA: Kevin (and/or Brian) reviews 20 freshly graded Daily Pages submissions end-to-end before flag-on for any school.
- Regression: existing thesis-essay GA behavior unchanged. Submission view, gradebook sync, override flow unchanged for thesis essays.

## Rollout

Feature flag (`daily_pages_holistic_ga`). Default OFF.

Staged path:

1. **Calibration phase.** Owner builds the prompt and scoring pipeline against the calibration set. No user-facing rollout. Iterate until calibration targets are met.
2. **Internal review.** Kevin and Brian review 20 holistic-graded submissions in a staging environment. Sign-off here is the gate to step 3.
3. **UA professor, flag on for Daily Pages only.** Highest volume of Daily Pages submissions; most informative real-world signal. Holistic GA runs on new Daily Pages submissions; old submissions stay on their original grades.
4. **Washington and Birmingham City** after UA validates the grades and feedback are landing well with teachers.
5. **Default ON** for all schools once pilot validation passes.

The Submit-for-Grade toggle spec gates the Daily Pages graded path on this work shipping. Until step 3, teachers can leave Daily Pages ungraded (toggle off) and still get the existing GA feedback — that's the fallback.

## North star context

Daily Pages happen every day in classrooms that adopt them. That's the highest-volume, lowest-friction source of student thinking in the platform. If GA grades that thinking the way it grades a thesis essay, the platform actively punishes the kind of writing it most wants to encourage. Fix this, and every Daily Pages submission becomes structured signal for the student-level insights work — without forcing teachers to wade through it manually, and without rework when we layer insights on top.

## Engineering handoff checklist

- [x] Domain context covered
- [x] File paths in `yawp-2.0` listed
- [x] Data model implications spelled out, including backward-compat plan
- [x] UX sketch in prose
- [x] Edge cases enumerated
- [x] Test plan written
- [x] Rollout plan decided
- [ ] Owner assigned
- [ ] Calibration set size and target accuracy band confirmed
- [ ] Brian signoff on minimum grade floor and off-prompt handling