hcie-rust-v3.05/.kilo/plans/fix-psd-writer-structure.md

PSD Writer Structural Fix Plan — Example3-mini round-trip

Problem

/mnt/extra/00_PROJECTS/hcie-rust-v4/_images/_test_images/example3/Example3-mini-hcie.psd is produced from the original Example3-mini.psd via hcie-psd-saver/src/lib.rs. Side-by-side binary analysis shows several structural problems that prevent Photoshop from opening the generated file correctly and corrupt the layer/folder structure:

1. Group / section-divider layers are lost — original has 19 layers (including a section divider for the Dodge and Burn group); HCIE output had 18. The divider layer with lsct section type 2 was dropped, so Photoshop could not reconstruct the folder.
2. Layer blend mode string mismatch for group pass-through — original group divider uses pass; HCIE wrote norm for the group layer. BlendMode::PassThrough exists in hcie_protocol but the PSD writer mapped it to b"pass" only in the record. However the imported group layer came in as BlendMode::Normal because hcie-psd/src/psd_import.rs converted "PassThrough" to Normal.
3. Layer flags were always 0 — original uses flags 8 (transparency protected / group flags), 24 (group/adjustment), 56 (adjustment with vector mask).
4. Group layers were emitted at full canvas size — imported group layer had width=1920 height=1080 and was written as a normal raster layer with transparent pixels. Original group divider layers have bounds=(0,0,0,0) and only an lsct block.
5. Adjustment layers lost their extra blocks — original curv, grdm, hue2 layers have 40+ byte adjustment descriptors. HCIE imported them as Raster with no adjustment data, then wrote them as 1×1 transparent raster layers.
6. Layer extra block order/format differences — original uses luni immediately after the Pascal name. Extra block ordering in general did not match Photoshop's observed sequence.
7. lfx2 / effects data was missing for layers that had effects — original Rear Light / Jaw Light / Cheek Redness contain lfx2 blocks; HCIE output had none, so all layer effects were lost on export.
8. Adjustment blocks (curv, grdm, hue2) were not written — even if imported, the writer had no code to emit these.
9. iOpa / brst ordering inside extra data was non-standard — writer placed iOpa and brst before lfx2.
10. Merged image data differs — expected because compositing in HCIE is not pixel-identical to Photoshop, but the layer/image structure must be valid first.

Root Causes

• hcie-psd/src/psd_import.rs collapsed group section-divider layers and PassThrough blend mode into LayerType::Group + BlendMode::Normal, so the writer could not emit the correct divider record.
• hcie-psd/src/psd_import.rs did not preserve adjustment data (curv, grdm, hue2) into layer.adjustment; it only built curve-based Adjustment for curves, but the writer never read layer.adjustment.
• hcie-psd-saver/src/lib.rs treated every layer as a 4-channel raster layer with full canvas or content bounds, and emitted the same extra block sequence regardless of layer type.
• The writer did not emit section-divider records (lsct type 2) for group end markers, and did not write pass-through blend mode for group dividers.
• Layer flags were hard-coded to 0; the writer did not compute protection/group/adjustment flags.
• The writer's image resources section is a minimal stub; the original PSD contains ~12 KB of resources (thumbnail, guides, print info, paths, etc.).
• hcie-fx::parser::parse_lfx2 cannot yet decode the newer lfx2 descriptor format used by Example3-mini.psd, so effects are lost during import and therefore cannot be re-emitted.

Files to Modify

File	Layer	Why
hcie-psd-saver/src/lib.rs	GUI/open (crate not in locked list)	Main PSD writer; group/adjustment/flag/effect handling and image resources
hcie-psd/src/psd_import.rs	editable	Preserve section-divider type, pass-through blend, adjustment raw data
hcie-protocol/src/lib.rs	locked (unlocked only for these fields)	Layer.adjustment_raw, Layer.is_section_divider storage fields
hcie-psd-saver/src/fx_descriptors.rs	GUI/open	Effect descriptor serialization; needs to stay in sync with hcie-fx parser
hcie-psd-saver/src/helpers.rs	GUI/open	write_8bim_block / write_luni_block helpers
hcie-fx/src/parser.rs	engine (locked via AGENTS.md)	parse_lfx2 / parse_descriptor_body need to decode numM/ChFX/modern descriptors

Implementation Status

✅ Done

• Added adjustment_raw: Option<([u8; 4], Vec<u8>)> and is_section_divider: bool to hcie_protocol::Layer.
• psd_import now preserves curv/grdm/hue2 raw bytes and section-divider type/PassThrough blend.
• psd_import emits a </Layer group> section-divider marker in the correct position.
• psd_saver emits section dividers with bounds=(0,0,0,0), empty channels, lsct type 2, and pass blend mode.
• psd_saver computes layer flags (group/adjustment).
• psd_saver passes adjustment raw blocks through before luni/lfx2.
• psd_saver orders extra blocks as: mask → blending ranges → Pascal name → adjustment → lfx2/iOpa/brst → luni → lyid → lsct → clbl/infx/knko/lspf/lclr → shmd → fxrp → lyvr.
• roundtrip_psd example now reports structural comparison.

Current Test Result

Original layer records: 19
Saved layer records:    19
  Layer structure matches (names, group/divider flags, adjustment flags).

Group divider blend is now pass and section-divider marker is present.

Remaining Work for Full Photoshop Compatibility

1. Effect Import (hcie-fx parser)

Example3-mini.psd stores effects in lfx2 descriptor format. The current parse_lfx2 only sees Scl, masterFXSwitch, numM entries and fails to parse the numM list of ChFX/DrSh/etc. objects. Until this is fixed, layer.effects remains empty on import and cannot be re-emitted.

Approach:

• Extend parse_descriptor_body to handle VlLs/Objc recursion with correct skip offsets for non-ASL descriptors.
• Add explicit handling for numM (effect list) and common modern effect keys.
• Add fixture tests using the 100-byte lfx2 blocks from Rear Light, Jaw Light, Cheek Redness.

2. Image Resources Passthrough

The original PSD has 24 resources (~12 KB); the writer only writes 7 minimal ones. Missing resources include:

• 0x040C thumbnail
• 0x03F3, 0x03F5, 0x03F8, 0x0400, 0x0404, 0x0411, 0x041A, 0x041E, 0x0421, 0x0422, 0x0424, 0x0425, 0x0426, 0x0428, 0x0429, 0x043A, 0x043B, 0x2710

Approach:

• During import, store the raw image resources section bytes (or a parsed map of resource ID → data).
• In save_psd, if the input came from a PSD, write the original resources back unchanged, replacing only resources we must generate (resolution, global angle, layer selection if needed).
• This avoids reverse-engineering every resource format and keeps Photoshop metadata intact.

3. Layer Channel Data for Adjustment Layers

Original adjustment layers have 5 channels (-1, 0, 1, 2, -2 mask) with zero-length channel data. Our writer currently writes 4 channels for adjustment layers. Photoshop may expect 5 if a mask is declared.

Approach:

• Preserve original channel layout during import and re-emit exactly the same channel IDs/lengths for adjustment/group layers.
• For raster layers, keep the 4-channel auto-crop logic.

4. luni Block Format

Original PSD's luni block does not have a leading Pascal string; it starts directly with 4-byte UTF-16 length. write_8bim_block adds a Pascal string length byte of 0 before the data, which may confuse strict parsers. In practice it parses, but for parity we should support writing luni without the Pascal prefix.

Approach:

• Add a helper that writes luni either with or without the leading Pascal byte, controlled by a compatibility flag.
• After verifying with Photoshop, choose the format that matches the original.

5. Merged Image Layout

Historical versions of the writer used interleaved row lengths (R0 G0 B0 A0 R1 G1 B1 A1 ...) but per-channel compressed data. Current code is internally consistent (per-channel lengths + per-channel data). The plan file fix-psd-writer-for-photoshop.md flagged this as a possible regression.

Approach:

• Add a toggle in write_merged_image_data for interleaved vs. per-channel row lengths.
• Generate a minimal 256×256 single-layer PSD with both layouts and test which one Photoshop/Photopea opens.
• Lock in the working layout and add a regression test.

6. Clipping Mask / Vector Mask Support

Original layers with effects (e.g. Rear Light) use an explicit -2 mask channel (mask_len=28). Our writer currently writes an empty mask section. If effects rely on the mask, they will look wrong.

Approach:

• Preserve mask_pixels, mask_bounds, and mask_default_color on import (already partially done).
• Re-emit layer masks in the saved PSD when present.

7. Group Bounds / Section Divider Channel Count

Original section-divider record has 4 channels with 2-byte 0001 RLE markers. Our writer currently emits 4 channels with a single 0x81 PackBits run marker. Both represent empty data, but byte-for-byte parity requires matching the exact marker format.

Approach:

• Match the original exactly: 4 channels, each 00 01 compression type followed by nothing (or minimal empty RLE row).

Test & Tune System

To reach full compatibility we need automated, repeatable tests. Proposed system:

A. Structural Regression Suite

Extend hcie-io/examples/roundtrip_psd.rs to:

1. Compare layer count, names, order.
2. Compare per-layer bounds, channel count/IDs, channel lengths.
3. Compare blend mode keys, opacity, clipping, flags.
4. Compare extra block keys and order (mask → blending → name → adjustment → lfx2 → iOpa/brst → luni → lyid → lsct → clbl/infx/knko/lspf/lclr → shmd → fxrp → lyvr).
5. Compare image resource IDs.
6. Compare merged image compression type.

Fail on first divergence and print both sides.

B. Photoshop Batch Validation Harness

Create _tools/photoshop/validate_psd.jsx that:

1. Opens a PSD in Photoshop.
2. Reports whether it opened without error.
3. Exports layer count and layer names to a JSON/text file.
4. Saves a flattened PNG for visual comparison.

Wrap it with a Rust/Node runner in hcie-io/examples/photoshop_validate.rs so CI/dev can call it locally if Photoshop is installed.

C. Golden File Regression

Commit Example3-mini-hcie.psd and a tests/psd_roundtrip.rs integration test that:

1. Imports Example3-mini.psd.
2. Saves to a temp file.
3. Asserts structural parity with the original.
4. (Later) asserts pixel MAE under a threshold once effects are preserved.

D. MAE-Based Effect Tuning

Reuse the existing tune_mae_*_grid.rs pattern:

1. Generate a grid of single-effect PSDs with save_psd.
2. Batch-export ground-truth PNGs from the same PSDs in Photoshop.
3. Run grid search over effect parameters (blur factor, spread scale, etc.) to minimize MAE.
4. Apply best defaults to hcie-fx or hcie-psd-saver/fx_descriptors.rs.

This is currently done for drop shadow, outer glow, inner glow, bevel, stroke; inner shadow and satin need similar treatment.

E. Minimal Photoshop Open Test

Create hcie-io/examples/test_psd_simple.rs that writes a 256×256 single-layer PSD. This is the fastest way to verify whether the core PSD container opens in Photoshop before adding complex features.

Success Criteria (Updated)

1. ✅ Example3-mini-hcie.psd has 19 layers and matching structural roles.
2. ✅ roundtrip_psd reports structural parity.
3. ⬜ File opens in Photoshop without "not compatible" error.
4. ⬜ Folder/adjustment structure is visually correct.
5. ⬜ Layer effects round-trip and render correctly (requires hcie-fx parser fix).
6. ⬜ Merged image MAE is within acceptable threshold vs. Photoshop ground-truth PNG.

Next Steps (Priority Order)

1. Image resources passthrough — lowest risk, biggest Photoshop-compatibility win.
2. Minimal PSD open test — verify base container with Photoshop/Photopea.
3. lfx2 effect parser fix — unlock full effect round-trip.
4. Layer mask / adjustment channel layout parity — fix structural edge cases.
5. Merged image row-length layout test — confirm Photoshop-preferred order.
6. Golden regression test — lock in progress and prevent regressions.

Notes

• hcie-protocol changes must remain minimal and backward-compatible; new fields have #[serde(default)].
• hcie-fx is a locked engine crate; any changes there must go through unlock.sh / lock.sh and be justified by a failing regression test.
• Do not chase pixel-perfect compositing in this plan; focus on structural validity and Photoshop opening first, then MAE-based tuning.

(Updated: 2026-06-18)