Commit 5bc25c5f authored by yaoke.yk's avatar yaoke.yk

docs: add canvas mode workbench design spec

parent 3b6236dd
# Canvas Mode Project Workbench Design
## Summary
Introduce a new project mode named `canvas` alongside the existing `normal` mode in `yaoaivideo`.
- `normal` mode keeps the current step-based project flow unchanged.
- `canvas` mode opens a new project-level workbench centered on a visual canvas.
- The canvas workbench is the main operating surface for canvas projects, but it reuses the existing domain objects for projects, episodes, characters, scenes, storyboards, video tasks, and assembly tasks.
- Canvas-specific data such as node layout, grouping, links, viewport, and action history is stored separately.
This design intentionally avoids project-mode switching in v1. A project chooses its mode at creation time and keeps it permanently.
## Goals
- Add a second creation path for projects: `normal` or `canvas`.
- Provide a new project-level workbench page for canvas projects.
- Visualize script, episodes, characters, scenes, storyboards, video tasks, and assembly tasks on a single canvas.
- Allow users to control generation actions from the canvas.
- Allow users to organize and edit the canvas using drag, selection, grouping, linking, and batch operations.
- Reuse the existing backend pipeline and API capabilities wherever possible.
- Keep the current normal-mode experience stable and isolated from the new workbench.
## Non-Goals
- No switching between `normal` and `canvas` after project creation.
- No attempt to make the canvas the sole source of truth for all business data in v1.
- No direct copy-paste migration of the Tapnow monolithic `src/App.jsx` architecture.
- No iframe or separately deployed micro-frontend workbench.
- No full free-form workflow engine in v1.
## Product Decisions
### Project Mode
Add a project mode field with at least the following values:
- `normal`
- `canvas`
Behavior:
- New projects default to `normal` unless the user actively chooses `canvas`.
- A `normal` project continues to use the current page sequence.
- A `canvas` project opens into a new workbench route and uses a canvas-first workflow.
- Mode is immutable after creation.
### User Experience Strategy
For `canvas` projects:
- The new workbench becomes the primary entry point.
- Existing detail pages can remain available as support pages for editing or fallback operations.
- The top project navigation changes to expose `工作台` instead of forcing users through the current step-by-step flow.
For `normal` projects:
- No behavior change.
- Existing routes, tabs, and project detail flow remain intact.
## Existing System Fit
This design is aligned with the current `yaoaivideo` structure:
- Frontend routing already supports project-scoped pages in `doc/html/src/app/routes.tsx`.
- Project top tabs already exist in `doc/html/src/app/components/Layout.tsx`.
- New project creation already flows through `doc/html/src/app/pages/NewProject.tsx`.
- Frontend data hooks and APIs already exist for projects and AI flows, including:
- `doc/html/src/lib/api/projects.ts`
- `doc/html/src/lib/api/ai.ts`
- `doc/html/src/hooks/useProjects.ts`
- `doc/html/src/hooks/useAi.ts`
- Backend domain and controller structure already supports projects, storyboards, video tasks, and agent runs.
This makes the workbench a host-integrated feature rather than a separate product.
## Architecture Overview
The recommended architecture has four layers.
### 1. Domain Truth Layer
Keep existing domain tables and services as the source of truth for:
- projects
- outlines
- episodes
- characters
- scenes
- storyboards
- video tasks
- assembly tasks
The canvas will not replace these tables in v1.
### 2. Canvas Orchestration Layer
Introduce a separate workbench data model to store:
- nodes
- edges
- viewport
- groups
- layout
- per-node display state
- canvas action history
- snapshots
This layer references existing business objects instead of duplicating them.
### 3. Workbench Control Layer
Add a dedicated backend module surface for canvas-oriented actions such as:
- bootstrap workbench from project data
- save canvas layout
- trigger extraction and generation actions from nodes
- expose aggregated workbench state
- record action history
This layer should internally call existing services and pipelines whenever possible.
### 4. Frontend Workbench Layer
Build a new modular workbench frontend inside the current `doc/html` app:
- page shell
- canvas renderer
- inspector
- library panel
- action toolbar
- bottom task panel
- workbench store
- workbench API layer
## Data Model Design
### Extend Existing Project Data
Add `projectMode` to project-level data.
Backend changes:
- `projects` table: add `project_mode`
- `Project` entity: add `projectMode`
- `ProjectCreateRequest`: accept `projectMode`
- `ProjectDTO`: expose `projectMode`
- project create/update service: validate and persist it
Recommended values:
- `normal`
- `canvas`
Default:
- `normal`
### New Workbench Tables
Recommended new tables:
#### `project_workbenches`
Purpose:
- One primary workbench record per project.
- Stores global canvas state.
Suggested fields:
- `id`
- `project_id`
- `tenant_id`
- `version`
- `viewport_x`
- `viewport_y`
- `zoom`
- `layout_mode`
- `created_at`
- `updated_at`
Constraints:
- unique key on `project_id`
#### `project_workbench_nodes`
Purpose:
- Stores canvas nodes.
Suggested fields:
- `id`
- `workbench_id`
- `project_id`
- `tenant_id`
- `node_type`
- `ref_type`
- `ref_id`
- `title`
- `status`
- `x`
- `y`
- `width`
- `height`
- `config_json`
- `meta_json`
- `sort_order`
- `created_at`
- `updated_at`
Notes:
- `ref_type` + `ref_id` ties nodes to existing business objects.
- `config_json` stores node-local UI and action parameters.
- `meta_json` stores display and temporary state that should still persist.
#### `project_workbench_edges`
Purpose:
- Stores links between nodes.
Suggested fields:
- `id`
- `workbench_id`
- `project_id`
- `tenant_id`
- `source_node_id`
- `target_node_id`
- `edge_type`
- `label`
- `config_json`
- `created_at`
- `updated_at`
#### `project_workbench_snapshots`
Purpose:
- Save restore points for the workbench.
Suggested fields:
- `id`
- `workbench_id`
- `project_id`
- `version`
- `snapshot_json`
- `created_by`
- `created_at`
#### `project_workbench_actions`
Purpose:
- Records explicit canvas-triggered actions.
Suggested fields:
- `id`
- `project_id`
- `workbench_id`
- `node_id`
- `action_type`
- `status`
- `request_json`
- `result_json`
- `error_message`
- `created_at`
- `updated_at`
### Relationship Model
Canvas nodes should not duplicate full business records.
Recommended mapping examples:
- `character` node -> `ref_type = character`, `ref_id = characters.id`
- `scene` node -> `ref_type = scene`, `ref_id = scenes.id`
- `storyboard` node -> `ref_type = storyboard`, `ref_id = storyboards.id`
- `video_task` node -> `ref_type = video_task`, `ref_id = ai_tasks.id`
- `assembly` node -> `ref_type = assembly_task`, `ref_id = assembly_tasks.id`
This keeps business truth and canvas orchestration cleanly separated.
## Canvas Node System
### Core Node Types
Recommended first-wave node types:
- `project`
- `script`
- `episode_group`
- `episode`
- `character_group`
- `character`
- `scene_group`
- `scene`
- `storyboard_group`
- `storyboard`
- `video_task`
- `assembly`
- `control`
- `note`
- `group`
### Edge Types
Recommended first-wave edge semantics:
- `contains`
- `depends_on`
- `references`
- `produces`
- `controls`
- `related_to`
The UI may render all of these visually, but v1 should restrict which edge types users can create manually.
### Node Status Model
Recommended shared status language:
- `idle`
- `ready`
- `running`
- `success`
- `failed`
- `blocked`
- `missing_dependency`
- `attention`
Statuses should be derived from underlying business data when possible, and only stored on the node if there is real workbench-only meaning.
## Product Interaction Design
### Default Entry Flow
For `canvas` projects:
- user creates project in `NewProject`
- selects `画布模式`
- backend creates project with `projectMode = canvas`
- frontend redirects to `/project/:projectId/workbench`
- first open triggers workbench bootstrap if it does not exist yet
For `normal` projects:
- unchanged current redirect and page flow
### Workbench Layout
Recommended page structure:
- top: project-level command bar
- left: node library, filters, views
- center: main canvas
- right: inspector and node details
- bottom: action queue, logs, task history
### Primary Workbench Capabilities
#### Visualization
- show project production chain in one place
- show current state of episodes, characters, scenes, storyboards, video tasks, and assembly
- expose dependencies and groupings visually
#### Control
- extract characters
- extract scenes
- generate storyboards
- generate video
- retry failed actions
- batch-run selected nodes or groups
#### Editing and Orchestration
- drag nodes
- box select
- multi-select
- group nodes
- create note nodes
- lock and unlock nodes
- collapse and expand groups
- create selected link types
- save custom layout
- restore snapshots
## Bootstrap Strategy
Canvas projects should not start from an empty board.
Recommended behavior:
- first open runs `bootstrap`
- backend reads project state and creates a default graph
- graph is organized into production lanes
Suggested default lanes from left to right:
- project/script
- episodes
- characters/scenes
- storyboards
- video tasks
- assembly
Benefits:
- immediate value on first entry
- no blank-canvas confusion
- existing project content becomes instantly visible and controllable
## Route and Navigation Design
### New Route
Add:
- `/project/:projectId/workbench`
### Existing Routes
Keep existing routes untouched, including:
- `/project/:projectId`
- `/project/:projectId/outline`
- `/project/:projectId/episodes`
- `/project/:projectId/characters`
- `/project/:projectId/scenes`
- `/project/:projectId/props`
- `/project/:projectId/storyboard/:episodeId?`
- `/project/:projectId/video`
- `/project/:projectId/agent`
### Navigation Rules
For `canvas` projects, recommended project tabs in `Layout.tsx`:
- `工作台`
- `资产` or `设定`
- `视频`
- `Agent`
- `设置`
For `normal` projects:
- keep current tab structure
Optional behavior:
- `ProjectDetail` can remain available for canvas projects, but should not be the primary entry page
- alternatively, `ProjectDetail` for canvas projects can become a summary card page with a prominent workbench entry
## API Design
Introduce a new workbench API surface.
### Workbench Read APIs
- `GET /projects/{projectId}/workbench`
- returns full workbench model for rendering
- `POST /projects/{projectId}/workbench/bootstrap`
- creates default workbench from current project data if absent or on explicit reset
- `GET /projects/{projectId}/workbench/actions`
- lists recent workbench actions
- `GET /projects/{projectId}/workbench/snapshots`
- lists snapshots
### Workbench Write APIs
- `PUT /projects/{projectId}/workbench/viewport`
- `PUT /projects/{projectId}/workbench/layout`
- `POST /projects/{projectId}/workbench/nodes`
- `PATCH /projects/{projectId}/workbench/nodes/{nodeId}`
- `DELETE /projects/{projectId}/workbench/nodes/{nodeId}`
- `POST /projects/{projectId}/workbench/edges`
- `DELETE /projects/{projectId}/workbench/edges/{edgeId}`
- `POST /projects/{projectId}/workbench/snapshots`
- `POST /projects/{projectId}/workbench/snapshots/{snapshotId}/restore`
### Workbench Action API
- `POST /projects/{projectId}/workbench/actions`
Suggested request payload:
- `actionType`
- `scope`
- `nodeIds`
- `refType`
- `refIds`
- `params`
Suggested `actionType` values:
- `extract_characters`
- `extract_scenes`
- `generate_storyboards`
- `generate_video`
- `retry_video_task`
- `regenerate_prompt`
- `auto_layout`
- `sync_from_project`
### Reuse of Existing APIs
The frontend workbench should still use or indirectly reuse the existing APIs in `ai.ts` and `projects.ts` where sensible.
The new backend workbench layer should orchestrate those existing domain operations instead of duplicating them.
## Frontend Module Design
Do not migrate Tapnow as one monolithic file.
Recommended new frontend files under `doc/html/src`:
- `app/pages/ProjectWorkbench.tsx`
- `app/components/workbench/WorkbenchShell.tsx`
- `app/components/workbench/WorkbenchCanvas.tsx`
- `app/components/workbench/WorkbenchToolbar.tsx`
- `app/components/workbench/WorkbenchInspector.tsx`
- `app/components/workbench/WorkbenchBottomPanel.tsx`
- `app/components/workbench/WorkbenchNode.tsx`
- `app/components/workbench/nodes/*`
- `hooks/useWorkbench.ts`
- `hooks/useWorkbenchActions.ts`
- `stores/workbenchStore.ts`
- `lib/api/workbench.ts`
- `lib/workbench/nodeTypes.ts`
- `lib/workbench/layout.ts`
- `lib/workbench/mappers.ts`
### State Management Split
Recommended state split:
React Query:
- project data
- workbench fetches
- actions
- polling
- invalidation
Zustand:
- current selection
- hover state
- drag state
- temporary viewport state
- panel open/close state
- in-memory interaction state
Avoid placing full server truth inside Zustand.
## Backend Module Design
Recommended new backend package areas:
- `yaoai-api/.../controller/ProjectWorkbenchController.java`
- `yaoai-api/.../dto/workbench/*`
- `yaoai-api/.../service/ProjectWorkbenchFacade.java`
- `yaoai-api/.../service/impl/ProjectWorkbenchFacadeImpl.java`
- `yaoai-pipeline/.../service/WorkbenchActionService.java`
- `yaoai-domain/.../entity/ProjectWorkbench*.java`
- `yaoai-domain/.../mapper/ProjectWorkbench*.java`
- new Flyway migration for workbench tables and project mode
Service split recommendation:
- `WorkbenchBootstrapService`
- `WorkbenchLayoutService`
- `WorkbenchActionService`
- `WorkbenchSnapshotService`
- `WorkbenchProjectionService`
## Mapping Strategy from Existing Project Data
The workbench is a projection over current project content.
Recommended projection rules:
- project -> one root node
- uploaded script / outline -> one script node
- episodes -> episode nodes grouped under an episode lane
- characters -> character nodes
- scenes -> scene nodes
- storyboards -> storyboard nodes grouped by episode
- video tasks -> video task nodes attached to storyboard or episode nodes
- assembly task -> one assembly node per episode if available
Where relationships are not fully explicit in current data, derive minimal useful links rather than blocking the feature.
## Editing Semantics
There are two kinds of edits.
### Business Object Edits
When the user edits:
- character fields
- scene fields
- storyboard prompt-related fields
- video task launch parameters
The workbench should call the existing formal backend APIs and update the actual business record.
### Canvas-Only Edits
When the user edits:
- x/y position
- node size
- group membership
- manual edge creation
- collapse state
- node note text
- visual category tags
The workbench should only persist into workbench tables.
## Batch Operations
Batch operations are central to the value of the canvas mode.
Recommended batch actions in v1:
- extract all characters for a project
- extract all scenes for a project
- generate storyboards for an episode or selected episodes
- generate videos for selected storyboards
- retry failed video tasks for selected nodes
- auto-layout selected node clusters
Batch actions should show:
- queued
- running
- success count
- failure count
- per-item error details
## Error Handling
Recommended behavior:
- layout save failures do not corrupt business data
- action failures appear at node level and in the bottom log panel
- missing referenced objects become `orphaned` or `missing_dependency` nodes instead of crashing the page
- bootstrap is idempotent where possible
- snapshot restore restores canvas orchestration state, not necessarily every domain object mutation
- destructive actions require confirmation
## Performance Considerations
Key risks:
- very large projects with many storyboard nodes
- frequent node movement causing excessive writes
- many concurrent task polls
Recommended mitigations:
- debounce layout persistence
- virtualize long side panels and lists
- split workbench fetches if graph becomes very large
- poll task status at sensible intervals rather than per-node aggressive polling
- support lazy expansion of episode groups
## Migration Strategy from Tapnow
Recommended rule:
- migrate interaction concepts, not code shape
Carry over:
- canvas-centric mental model
- node-based orchestration
- preview and generation linkage
- batch execution UX ideas
- visual status and queue thinking
Do not carry over directly:
- single giant `App.jsx`
- localStorage-heavy persistence as the primary source
- provider configuration UI embedded into the canvas runtime
- tight coupling between canvas rendering and request-template management
## Security and Multi-Tenant Considerations
Because `yaoaivideo` is tenant-aware, all new workbench records must include tenant scoping.
Requirements:
- all workbench reads and writes validate project ownership within tenant scope
- node refs must only target records inside the same project and tenant
- snapshot restore must not allow cross-project contamination
- action execution must honor the same auth and billing rules as existing flows
## Billing and Usage Considerations
Canvas-triggered actions should still flow through existing billing logic.
Requirements:
- extraction and generation triggered from the workbench count the same as when triggered from existing pages
- action history should record enough metadata for cost attribution
- workbench mode should not bypass quotas or watermark settings
## Rollout Plan
### Phase 1: Project Mode and Host
- add `projectMode` to project data model
- update project create flow UI and DTOs
- add workbench route
- update layout tab logic for canvas projects
### Phase 2: Workbench Persistence and Bootstrap
- add workbench tables
- add bootstrap service and APIs
- render read-only projected graph
- save viewport and layout
### Phase 3: Workbench Controls
- support canvas-triggered extraction and generation
- support node inspector editing for selected domain objects
- show per-node and bottom-panel action status
### Phase 4: Editing and Orchestration Enhancements
- add groups
- add edge editing
- add snapshots
- add advanced batch operations
- add auto-layout variants
## Testing Strategy
### Backend Tests
- migration tests for project mode and workbench tables
- bootstrap graph generation
- workbench node and edge CRUD
- action dispatch tests
- tenant isolation tests
### Frontend Tests
- project create mode selection
- route branching by project mode
- workbench initial render
- node selection and inspector behavior
- layout persistence debounce behavior
- batch action feedback
### E2E Tests
- create canvas project
- open workbench
- bootstrap graph appears
- trigger character extraction
- trigger scene extraction
- trigger storyboard generation
- trigger video generation
- save layout and reload
- restore a snapshot
## Risks
### Main Risks
- underestimating the effort of building a good canvas host inside the current frontend architecture
- letting workbench state and business data drift apart
- trying to make every edge fully editable too early
- rebuilding too much of Tapnow instead of integrating with `yaoaivideo`
- overloading v1 with too many node types and actions
### Mitigations
- keep mode separation strict
- keep business truth outside the workbench tables
- start with a constrained node and edge model
- ship bootstrap plus practical actions first
- preserve existing pages as fallback and support surfaces for canvas projects
## Open Decisions Already Resolved
These decisions are considered fixed for this design:
- use project mode split instead of replacing the existing flow
- create a new workbench route rather than condition-heavy retrofitting into existing pages
- keep mode immutable after project creation
- keep canvas as orchestration and projection layer, not sole truth layer
- avoid direct code transplant of Tapnow monolith
## Final Recommendation
Proceed with:
- `projectMode` on project creation
- dedicated `/project/:projectId/workbench` route
- independent workbench persistence tables
- existing domain object reuse for formal data truth
- modular frontend implementation in `doc/html`
- phased rollout with bootstrap, visualization, control, then advanced orchestration
This is the lowest-risk path that still preserves the product ambition of a project-level visual workbench with control and editing capabilities.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment