# ovui Local Interaction Features ## 7. Add Camera Controls Read `references/viewer-input-routing` first for ovui button normalization, click-vs-drag dispatch, and input/render-loop ownership. Then implement camera math in `local_app/input_controller.py` and `local_app/camera_controller.py`: - Track mouse press, movement, release, buttons, wheel, keyboard modifiers, viewport dimensions, and drag threshold state from ovui callbacks. - Convert ovui screen coordinates into render-image coordinates, accounting for widget position, preserve-aspect scaling, and letterboxing. - Use drag thresholding so a short left press/release selects and a left drag orbits. - Support orbit, pan, dolly/zoom, wheel zoom, fit-to-stage, and optional WASD fly mode when requested. - If selected-prim transform is enabled, test transform-gizmo presses before normal left-drag orbit and before click selection. - Sanitize camera state before input handling and before writing matrices. - Write the viewer camera transform to the camera prim through ovrtx live attributes. Critical contracts: - The camera is a USD prim. Camera movement writes `omni:xform` on the viewer camera path. - Use the row-vector matrix layout expected by USD/ovrtx: basis vectors in rows and translation in the final row. - Clamp elevation away from straight up/down singularities. - Clamp camera distance above a small positive minimum. - Skip camera writes when any matrix value is non-finite. - Use the same world-up convention as the scene when fitting and orbiting. - Left-click selection fires only on release if movement stayed below the drag threshold. - A transform-gizmo mouse-down owns the full gesture until release; it suppresses camera orbit and click-pick for that mouse-down. - Map ovui button IDs to the camera helper's expected button IDs before interpreting gestures. Decision points: - If the user wants simple navigation, use left drag orbit, middle drag pan, right drag dolly, and wheel zoom. - If the user wants DCC-style navigation, use Alt+left for orbit, Alt+middle for pan, and Alt+right for dolly while preserving left-click selection. - If the stage has an authored camera and the app policy is to use it, initialize viewer camera settings from that camera before allowing interaction. - If the user requests camera UI buttons, wire them to local runtime commands rather than duplicating pointer gesture logic in widgets. - If a minimal ovui gizmo renders handles but does not expose handle-drag callbacks, project the selected pivot into viewport space and implement a direct-manipulation fallback that converts mouse deltas into camera-plane world deltas. Common failure modes: - Putting matrix basis vectors in columns instead of rows places the camera inside, behind, or under the scene. - Ignoring letterboxing makes orbit centers and picks offset from the visible image. - Treating every left release as selection causes accidental selections after orbit drags. - Letting the camera controller see a transform-gizmo press first makes the gizmo appear inert because orbit owns the drag. - Forgetting ovui's button order makes right-drag and middle-drag modes swap. Read for depth: see `references/viewer-input-routing`, `references/camera-controls`, `references/local-viewer`, and `references/stage-hierarchy` for input routing, camera math, and bounds contracts. ## 8. Add Picking, Selection, And Highlighting Do this in `local_app/selection_controller.py`; the completed click or marquee gesture should come from `references/viewer-input-routing`: - On left mouse release after a click gesture, map the screen coordinate to a render pixel and enqueue a native ovrtx pick query. - After the next render step, read the synthetic `ovrtx_pick_hit` render var, validate its params, resolve `primPath` ids to USD prim paths, and deduplicate the result. - Maintain selected tree path and selected mesh paths separately when an Xform or Scope expands to descendant geometry. - Keep selection state in runtime memory and update viewport highlight, tree row selection, and prim info from that single source. - Clear selection on scene reset, scene switch, empty tree selection, or explicit clear command. - Apply visual selection feedback by writing native selection outline group attributes on the runtime stage, not by permanently editing the user USD. Critical contracts: - The pick query rectangle uses render-product pixel space after letterbox/scaling correction. - Picking coordinates must use render-pixel space after letterbox and scaling correction. - Selection state is local-runtime authoritative. Widgets mirror it; they do not independently mutate it. - Enable selection outlines at renderer creation and set per-group outline/fill colors with `Renderer.set_selection_group_styles(...)`. - Clear previous outlines by writing group `0`; assign selected prims to a non-zero group such as `1`. - Do not let selection picking run while the scene is loading or the renderer is resetting. - Check operation status for the stage load, render step, and pick query before changing selection. An empty or failed pick should not corrupt the previous selected state. Decision points: - If the user only asks for tree selection, implement tree-driven selection first and defer viewport picking. - If the user asks for visual highlight, use native selection outlines after selection state works. - If the user asks for hover, multi-select, or marquee selection, extend the local state model explicitly and keep final selected state centralized. - If selection needs property display, trigger the info panel to query the selected prim rather than stuffing full properties into every selection update. Common failure modes: - Selection appears offset when coordinate transforms ignore image scaling or letterboxing. - No pick result arrives when the query was enqueued for a different RenderProduct than the next `renderer.step()`. - Highlight persists across scene loads when old selection outline groups are not cleared. - Tree selection and viewport selection diverge when each widget tracks its own selected path. Read for depth: see `references/viewer-input-routing`, `references/object-selection`, `references/selection-feedback`, `references/prim-info-display`, and `references/stage-hierarchy` for the full input, picking, highlighting, and info contracts. ## 9. Add Scene Switching And Asset Browsing Do this in `local_app/scene_manager.py` and `local_app/widgets/scene_picker.py`: - Build a scene registry from configured local sample paths or an allowed asset root. - Display user-friendly scene labels while storing validated absolute paths or registry IDs internally. - When the user selects a scene, enter loading state, stop stepping the old scene, reset renderer stage state, create the new inline root/session data, load the new stage, restore persistent settings, fit or restore camera, clear selection, refresh hierarchy, and resume rendering. - Keep the previous valid scene alive until the new load succeeds when the UX requires non-destructive scene switching. - Provide a reload current scene action that rebuilds all derived state from the original source. Critical contracts: - Never call `renderer.step()` concurrently with scene reset/load. - Preserve render settings across scene switches unless the user explicitly asks for per-scene settings. - Preserve camera across scenes only when that policy is requested and the old camera state is valid for the new bounds; otherwise fit to the new stage. - Recompute hierarchy, properties cache, variants cache, bounds, pickability filters, and selection outline state for the new stage. - Validate user-selected paths against an allowed root before loading them. Decision points: - If assets are local files, scan `.usd`, `.usda`, and `.usdc` files under an allowed root. - If assets are cloud-backed, use `references/cloud-assets` and keep cloud logic behind the same asset registry interface. - If scene load fails, keep the previous scene visible if it is still valid, or enter idle/error state with a clear status message. - If variant changes require stage reload, route them through the same load lock and loading-state path as scene switching. Common failure modes: - Leaving the render loop active during reset produces intermittent crashes or corrupted frames. - Not clearing caches after scene switch shows stale tree children or properties. - Persisting a camera blindly can place the viewer far away from a very different scene. - Displaying raw absolute paths in the UI leaks local filesystem structure and makes labels noisy. Read for depth: see `references/stage-management`, `references/stage-loading`, `references/stage-hierarchy`, and `references/cloud-assets` for the full scene switching and asset contracts. ## 10. Add Stage Hierarchy, Properties, And Variants Do this in `local_app/stage_queries.py`, `local_app/widgets/stage_tree.py`, and `local_app/widgets/prim_info_panel.py`: - Open or attach the current stage for query operations using the chosen direct or subprocess `pxr` mode. - Query root children after a scene loads. - Load tree children lazily when a row expands. - Represent each prim with name, path, type, and child-load state. - Fetch properties for the selected prim when selection changes. - Fetch variants for the selected prim and apply variant changes through the scene manager. - After variant changes, refresh affected hierarchy, properties, bounds, native pickability state, and selection if composition changed. Critical contracts: - Avoid full recursive hierarchy traversal by default for large scenes. - Keep children semantics stable: expandable-but-not-loaded is distinct from loaded children and leaf rows. - Do not assume all USD property values are directly displayable. Normalize arrays, tokens, paths, numbers, booleans, and fallback display strings. - Include the prim path with every local query result so widgets can ignore stale data after selection or scene changes. - If query work can block the UI, move it to a worker and return results to the UI/render loop safely. - Tree selection should call the same runtime selection path as viewport picking. Decision points: - If direct `pxr` imports are stable in the local process, direct query helpers are acceptable. - If imports conflict, use a subprocess query mode and keep logs on stderr while requests and responses use structured data. - If the user asks for variant editing, treat it as a scene mutation and route it through scene management. - If the user asks for full property editing, treat it as a separate feature and add explicit edit/apply/reload contracts. Common failure modes: - Large scenes freeze the UI when the full hierarchy is traversed synchronously. - Non-serializable or non-displayable USD values break property rendering. - Variant changes leave stale property and child rows unless caches are invalidated. - Direct `pxr` imports can conflict with ovrtx if import order or library paths are inconsistent. Read for depth: see `references/stage-hierarchy`, `references/prim-info-display`, `references/stage-management`, and `references/windows-native-setup` for the full hierarchy and property contracts. ## 11. Add Render Settings And Lighting Controls Do this in `local_app/render_settings.py`, `local_app/settings_store.py`, and `local_app/widgets/render_settings_panel.py`: - Define a small persistent settings model for validated render settings, camera policy, segmentation/debug state, viewport/profile defaults, and non-live defaults after the runtime capability list is known. - Build a runtime-owned supported-settings capability list from verified backend apply paths. The render settings panel must render from that list, not from hard-coded optimistic controls. - Load settings at app start. - Apply validated immediate settings and accepted profile/default settings after every scene load. - Persist user changes after validation. - Keep scene-independent settings separate from scene-specific transient state. - Echo the effective settings and capabilities back into UI state after every change so controls reflect clamped values, `applied`, `applies_at`, `requires_reload`, and any message. Critical contracts: - Do not add viewer lights by default. Only create viewer-owned lights when the user requests lighting controls. - If changing resolution, treat it as a render product, image bridge, pick coordinate, and viewport math reconfiguration. - If changing render vars, update scene setup and frame extraction together. - Persist validated settings across scene switches. - Validate every setting value from the UI before applying it. - Reject unsupported setting keys. Do not report success for client-side form state alone. - Do not invent `write_attribute` names for renderer internals; expose only controls supported by the active ovrtx build and documented by `references/render-settings`. Decision points: - If the user wants a basic Omniverse Realtime Viewer, provide only verified immediate controls by default. AOV/debug view is acceptable when implemented through `aov-switching`; lighting controls are acceptable only when backed by a verified live apply path or explicit reload/profile workflow. - If the user wants material or renderer internals, read `references/render-settings` before exposing them. - If the setting can be applied live through ovrtx attributes, apply it from the render loop. - If the setting requires stage reload, expose it as an explicit render-profile or scene-load action rather than a live control. Common failure modes: - UI says a setting changed while the renderer is still using the old value because effective settings were not echoed back. - UI exposes controls that cannot apply because the panel did not render from backend-advertised capabilities. - Viewer-created lights change the look of authored scenes unexpectedly. - Resolution changes break picking because render product, image bridge, pick coordinate mapping, and letterbox math no longer agree. - Extra render vars increase memory and frame time even when no feature uses them. Read for depth: see `references/render-settings`, `references/stage-management`, and `references/stage-loading` for the full settings contract. ## 12. Manage Runtime State And Shutdown Do this in `local_app/runtime.py`: - Maintain explicit runtime states: starting, idle/no scene, loading, rendering, error, and shutting down. - Drain queued commands from the UI/render loop before stepping the next frame. - Keep the latest loaded stage path, hierarchy root, selection, render settings, loading state, and viewer error in runtime memory. - Handle app shutdown by stopping the ovui loop, preventing new scene commands, releasing renderer-owned resources if the API exposes them, and closing any query subprocesses. - Guard long-running operations so the UI can display loading or error state instead of silently freezing. Critical contracts: - UI callbacks must not call renderer load/reset/step/write APIs directly if they can run during another renderer operation. - Scene load and reset operations must be mutually exclusive with frame stepping. - Errors should clear active drag state, loading state, and stale selection when needed. - Query subprocesses must be terminated on shutdown and restarted only through a controlled path. - The app should exit cleanly on interrupt or window close without leaving stale GPU processes. Decision points: - If all work stays on the UI thread, keep command handling simple and deterministic. - If loading large scenes needs a worker thread, make the render thread the only component that mutates the ovrtx renderer. - If hierarchy queries are slow, cache per-prim results and invalidate them on scene switch or variant change. - If the app needs autosave settings, write only app settings JSON, never user USD files. Common failure modes: - Calling renderer methods from both callbacks and the render loop creates rare crashes that are hard to reproduce. - Exceptions during input can leave a drag gesture active and cause later clicks to orbit or select unexpectedly. - Query workers that write logs to stdout can corrupt structured responses. - App exits that skip GPU cleanup may leave stale Python processes holding CUDA/RTX state. Read for depth: see `references/local-viewer`, `references/ovrtx-rendering`, `references/stage-management`, and `references/stage-hierarchy` for runtime and shutdown contracts.