# PSD Writer Structural Fix Plan — Example3-mini round-trip

## Problem

`/mnt/extra/00_PROJECTS/hcie-rust-v4/_images/_test_images/example3/Example3-mini-hcie.psd` is produced from the original `Example3-mini.psd` via `hcie-psd-saver/src/lib.rs`. Side-by-side binary analysis shows several structural problems that prevent Photoshop from opening the generated file correctly and corrupt the layer/folder structure:

1. **Group / section-divider layers are lost** — original has 19 layers (including a section divider for the `Dodge and Burn` group); HCIE output had 18. The divider layer with `lsct` section type 2 was dropped, so Photoshop could not reconstruct the folder.
2. **Layer blend mode string mismatch for group pass-through** — original group divider uses `pass`; HCIE wrote `norm` for the group layer. `BlendMode::PassThrough` exists in `hcie_protocol` but the PSD writer mapped it to `b"pass"` only in the record. However the imported group layer came in as `BlendMode::Normal` because `hcie-psd/src/psd_import.rs` converted "PassThrough" to `Normal`.
3. **Layer flags were always 0** — original uses flags `8` (transparency protected / group flags), `24` (group/adjustment), `56` (adjustment with vector mask).
4. **Group layers were emitted at full canvas size** — imported group layer had `width=1920 height=1080` and was written as a normal raster layer with transparent pixels. Original group divider layers have `bounds=(0,0,0,0)` and only an `lsct` block.
5. **Adjustment layers lost their extra blocks** — original `curv`, `grdm`, `hue2` layers have 40+ byte adjustment descriptors. HCIE imported them as `Raster` with no adjustment data, then wrote them as 1×1 transparent raster layers.
6. **Layer extra block order/format differences** — original uses `luni` immediately after the Pascal name. Extra block ordering in general did not match Photoshop's observed sequence.
7. **`lfx2` / effects data was missing for layers that had effects** — original `Rear Light` / `Jaw Light` / `Cheek Redness` contain `lfx2` blocks; HCIE output had none, so all layer effects were lost on export.
8. **Adjustment blocks (`curv`, `grdm`, `hue2`) were not written** — even if imported, the writer had no code to emit these.
9. **`iOpa` / `brst` ordering inside extra data was non-standard** — writer placed `iOpa` and `brst` before `lfx2`.
10. **Merged image data differs** — expected because compositing in HCIE is not pixel-identical to Photoshop, but the layer/image structure must be valid first.

## Root Causes

- `hcie-psd/src/psd_import.rs` collapsed group section-divider layers and `PassThrough` blend mode into `LayerType::Group` + `BlendMode::Normal`, so the writer could not emit the correct divider record.
- `hcie-psd/src/psd_import.rs` did not preserve adjustment data (`curv`, `grdm`, `hue2`) into `layer.adjustment`; it only built curve-based `Adjustment` for curves, but the writer never read `layer.adjustment`.
- `hcie-psd-saver/src/lib.rs` treated every layer as a 4-channel raster layer with full canvas or content bounds, and emitted the same extra block sequence regardless of layer type.
- The writer did not emit section-divider records (`lsct` type 2) for group end markers, and did not write pass-through blend mode for group dividers.
- Layer flags were hard-coded to `0`; the writer did not compute protection/group/adjustment flags.
- The writer's image resources section is a minimal stub; the original PSD contains ~12 KB of resources (thumbnail, guides, print info, paths, etc.).
- `hcie-fx::parser::parse_lfx2` cannot yet decode the newer `lfx2` descriptor format used by `Example3-mini.psd`, so effects are lost during import and therefore cannot be re-emitted.

## Files to Modify

| File | Layer | Why |
|------|-------|-----|
| `hcie-psd-saver/src/lib.rs` | GUI/open (crate not in locked list) | Main PSD writer; group/adjustment/flag/effect handling and image resources |
| `hcie-psd/src/psd_import.rs` | editable | Preserve section-divider type, pass-through blend, adjustment raw data |
| `hcie-protocol/src/lib.rs` | locked (unlocked only for these fields) | `Layer.adjustment_raw`, `Layer.is_section_divider` storage fields |
| `hcie-psd-saver/src/fx_descriptors.rs` | GUI/open | Effect descriptor serialization; needs to stay in sync with `hcie-fx` parser |
| `hcie-psd-saver/src/helpers.rs` | GUI/open | `write_8bim_block` / `write_luni_block` helpers |
| `hcie-fx/src/parser.rs` | engine (locked via AGENTS.md) | `parse_lfx2` / `parse_descriptor_body` need to decode `numM`/`ChFX`/modern descriptors |

## Implementation Status

### ✅ Done

- Added `adjustment_raw: Option<([u8; 4], Vec<u8>)>` and `is_section_divider: bool` to `hcie_protocol::Layer`.
- `psd_import` now preserves `curv`/`grdm`/`hue2` raw bytes and section-divider type/PassThrough blend.
- `psd_import` emits a `</Layer group>` section-divider marker in the correct position.
- `psd_saver` emits section dividers with `bounds=(0,0,0,0)`, empty channels, `lsct` type 2, and `pass` blend mode.
- `psd_saver` computes layer flags (group/adjustment).
- `psd_saver` passes adjustment raw blocks through before `luni`/`lfx2`.
- `psd_saver` orders extra blocks as: mask → blending ranges → Pascal name → adjustment → `lfx2`/`iOpa`/`brst` → `luni` → `lyid` → `lsct` → `clbl`/`infx`/`knko`/`lspf`/`lclr` → `shmd` → `fxrp` → `lyvr`.
- `roundtrip_psd` example now reports structural comparison.

### Current Test Result

```
Original layer records: 19
Saved layer records:    19
  Layer structure matches (names, group/divider flags, adjustment flags).
```

Group divider blend is now `pass` and section-divider marker is present.

## Remaining Work for Full Photoshop Compatibility

### 1. Effect Import (`hcie-fx` parser)

`Example3-mini.psd` stores effects in `lfx2` descriptor format. The current `parse_lfx2` only sees `Scl`, `masterFXSwitch`, `numM` entries and fails to parse the `numM` list of `ChFX`/`DrSh`/etc. objects. Until this is fixed, `layer.effects` remains empty on import and cannot be re-emitted.

**Approach:**
- Extend `parse_descriptor_body` to handle `VlLs`/`Objc` recursion with correct skip offsets for non-ASL descriptors.
- Add explicit handling for `numM` (effect list) and common modern effect keys.
- Add fixture tests using the 100-byte `lfx2` blocks from `Rear Light`, `Jaw Light`, `Cheek Redness`.

### 2. Image Resources Passthrough

The original PSD has 24 resources (~12 KB); the writer only writes 7 minimal ones. Missing resources include:
- `0x040C` thumbnail
- `0x03F3`, `0x03F5`, `0x03F8`, `0x0400`, `0x0404`, `0x0411`, `0x041A`, `0x041E`, `0x0421`, `0x0422`, `0x0424`, `0x0425`, `0x0426`, `0x0428`, `0x0429`, `0x043A`, `0x043B`, `0x2710`

**Approach:**
- During import, store the raw image resources section bytes (or a parsed map of resource ID → data).
- In `save_psd`, if the input came from a PSD, write the original resources back unchanged, replacing only resources we must generate (resolution, global angle, layer selection if needed).
- This avoids reverse-engineering every resource format and keeps Photoshop metadata intact.

### 3. Layer Channel Data for Adjustment Layers

Original adjustment layers have 5 channels (`-1, 0, 1, 2, -2` mask) with zero-length channel data. Our writer currently writes 4 channels for adjustment layers. Photoshop may expect 5 if a mask is declared.

**Approach:**
- Preserve original channel layout during import and re-emit exactly the same channel IDs/lengths for adjustment/group layers.
- For raster layers, keep the 4-channel auto-crop logic.

### 4. `luni` Block Format

Original PSD's `luni` block does **not** have a leading Pascal string; it starts directly with 4-byte UTF-16 length. `write_8bim_block` adds a Pascal string length byte of 0 before the data, which may confuse strict parsers. In practice it parses, but for parity we should support writing `luni` without the Pascal prefix.

**Approach:**
- Add a helper that writes `luni` either with or without the leading Pascal byte, controlled by a compatibility flag.
- After verifying with Photoshop, choose the format that matches the original.

### 5. Merged Image Layout

Historical versions of the writer used interleaved row lengths (`R0 G0 B0 A0 R1 G1 B1 A1 ...`) but per-channel compressed data. Current code is internally consistent (per-channel lengths + per-channel data). The plan file `fix-psd-writer-for-photoshop.md` flagged this as a possible regression.

**Approach:**
- Add a toggle in `write_merged_image_data` for interleaved vs. per-channel row lengths.
- Generate a minimal 256×256 single-layer PSD with both layouts and test which one Photoshop/Photopea opens.
- Lock in the working layout and add a regression test.

### 6. Clipping Mask / Vector Mask Support

Original layers with effects (e.g. `Rear Light`) use an explicit `-2` mask channel (`mask_len=28`). Our writer currently writes an empty mask section. If effects rely on the mask, they will look wrong.

**Approach:**
- Preserve `mask_pixels`, `mask_bounds`, and `mask_default_color` on import (already partially done).
- Re-emit layer masks in the saved PSD when present.

### 7. Group Bounds / Section Divider Channel Count

Original section-divider record has 4 channels with 2-byte `0001` RLE markers. Our writer currently emits 4 channels with a single `0x81` PackBits run marker. Both represent empty data, but byte-for-byte parity requires matching the exact marker format.

**Approach:**
- Match the original exactly: 4 channels, each `00 01` compression type followed by nothing (or minimal empty RLE row).

## Test & Tune System

To reach full compatibility we need automated, repeatable tests. Proposed system:

### A. Structural Regression Suite

Extend `hcie-io/examples/roundtrip_psd.rs` to:
1. Compare layer count, names, order.
2. Compare per-layer bounds, channel count/IDs, channel lengths.
3. Compare blend mode keys, opacity, clipping, flags.
4. Compare extra block keys and order (mask → blending → name → adjustment → lfx2 → iOpa/brst → luni → lyid → lsct → clbl/infx/knko/lspf/lclr → shmd → fxrp → lyvr).
5. Compare image resource IDs.
6. Compare merged image compression type.

Fail on first divergence and print both sides.

### B. Photoshop Batch Validation Harness

Create `_tools/photoshop/validate_psd.jsx` that:
1. Opens a PSD in Photoshop.
2. Reports whether it opened without error.
3. Exports layer count and layer names to a JSON/text file.
4. Saves a flattened PNG for visual comparison.

Wrap it with a Rust/Node runner in `hcie-io/examples/photoshop_validate.rs` so CI/dev can call it locally if Photoshop is installed.

### C. Golden File Regression

Commit `Example3-mini-hcie.psd` and a `tests/psd_roundtrip.rs` integration test that:
1. Imports `Example3-mini.psd`.
2. Saves to a temp file.
3. Asserts structural parity with the original.
4. (Later) asserts pixel MAE under a threshold once effects are preserved.

### D. MAE-Based Effect Tuning

Reuse the existing `tune_mae_*_grid.rs` pattern:
1. Generate a grid of single-effect PSDs with `save_psd`.
2. Batch-export ground-truth PNGs from the same PSDs in Photoshop.
3. Run grid search over effect parameters (blur factor, spread scale, etc.) to minimize MAE.
4. Apply best defaults to `hcie-fx` or `hcie-psd-saver/fx_descriptors.rs`.

This is currently done for drop shadow, outer glow, inner glow, bevel, stroke; inner shadow and satin need similar treatment.

### E. Minimal Photoshop Open Test

Create `hcie-io/examples/test_psd_simple.rs` that writes a 256×256 single-layer PSD. This is the fastest way to verify whether the core PSD container opens in Photoshop before adding complex features.

## Success Criteria (Updated)

1. ✅ `Example3-mini-hcie.psd` has 19 layers and matching structural roles.
2. ✅ `roundtrip_psd` reports structural parity.
3. ⬜ File opens in Photoshop without "not compatible" error.
4. ⬜ Folder/adjustment structure is visually correct.
5. ⬜ Layer effects round-trip and render correctly (requires `hcie-fx` parser fix).
6. ⬜ Merged image MAE is within acceptable threshold vs. Photoshop ground-truth PNG.

## Next Steps (Priority Order)

1. **Image resources passthrough** — lowest risk, biggest Photoshop-compatibility win.
2. **Minimal PSD open test** — verify base container with Photoshop/Photopea.
3. **lfx2 effect parser fix** — unlock full effect round-trip.
4. **Layer mask / adjustment channel layout parity** — fix structural edge cases.
5. **Merged image row-length layout test** — confirm Photoshop-preferred order.
6. **Golden regression test** — lock in progress and prevent regressions.

## Notes

- `hcie-protocol` changes must remain minimal and backward-compatible; new fields have `#[serde(default)]`.
- `hcie-fx` is a locked engine crate; any changes there must go through `unlock.sh` / `lock.sh` and be justified by a failing regression test.
- Do not chase pixel-perfect compositing in this plan; focus on structural validity and Photoshop opening first, then MAE-based tuning.

(Updated: 2026-06-18)