# DeepStream GStreamer Plugins Overview ## Introduction DeepStream provides a comprehensive set of custom GStreamer plugins optimized for NVIDIA GPUs. These plugins handle video decoding, inference, tracking, visualization, and various other video analytics tasks. Understanding these plugins is crucial for building effective DeepStream applications. ## Plugin Categories ### Source Plugins Plugins that generate or capture video data from various sources. ### Processing Plugins Plugins that transform, analyze, or process video data. ### Sink Plugins Plugins that output video to displays, files, or network destinations. --- ## Source Plugins ### nvv4l2decoder **Purpose**: Hardware-accelerated video decoder using NVIDIA V4L2 API (from nvvideo4linux2 plugin) **Key Properties**: - `capture-io-mode`: Capture I/O mode for the sink pad (`auto`, `mmap`, `dmabuf-import`) - `output-io-mode`: Output I/O mode for the src pad (`auto`, `mmap`, `dmabuf-import`) - `cudadec-memtype`: CUDA buffer memory type (`memtype_device`, `memtype_pinned`, `memtype_unified`) - `gpu-id`: GPU device ID used for decoding - `drop-frame-interval`: Interval for dropping frames (0 keeps all frames) - `num-extra-surfaces`: Additional decode surfaces to allocate - `disable-dpb`: Disable DPB buffers to reduce latency - `low-latency-mode`: Enable low-latency decoding for I/IPPP streams - `skip-frames`: Frame skipping policy (`decode_all`, `decode_non_ref`, `decode_key`) - `device`: Decoder device path (read-only, default `/dev/nvidia0`) **Usage**: ```bash nvv4l2decoder output-io-mode=0 drop-frame-interval=0 ``` **Common Pipeline Pattern**: ``` h264parse ! nvv4l2decoder ! ... ``` **Output Format**: - Outputs `video/x-raw(memory:NVMM)` - GPU memory format - This is already in NVMM format, so NO nvvideoconvert is needed before nvstreammux **Notes**: - Essential for GPU-accelerated pipelines - Supports H.264, H.265, VP8, VP9 codecs with zero-copy memory transfers - Output is already in NVMM memory, compatible with nvstreammux and other DeepStream plugins --- ### nvurisrcbin **Purpose**: Source bin for handling URI-based sources (files, RTSP, HTTP) **Key Properties**: - `uri`: Source URI (file://, rtsp://, http://, etc.) - `num-buffers`: Number of buffers to process - `drop-on-latency`: Drop frames on latency **Usage**: ```bash nvurisrcbin uri=file:///path/to/video.mp4 ``` **Common Pipeline Pattern**: ``` nvurisrcbin uri=rtsp://camera-ip/stream ! ... ``` **Notes**: - Automatically handles demuxing and parsing for multiple protocols and formats --- ### nvmultiurisrcbin **Purpose**: Source bin with built-in REST API server for dynamic multi-stream management **Key Properties**: | Property | Type | Description | |----------|------|-------------| | `uri-list` | string | Comma-separated list of initial URIs | | `sensor-id-list` | string | Comma-separated sensor IDs (maps 1:1 with uri-list) | | `sensor-name-list` | string | Comma-separated sensor names | | `ip-address` | string | REST API server IP (default: localhost) | | `port` | int | REST API server port (default: 9000, 0 to disable) | | `max-batch-size` | int | Maximum number of sources | | `batched-push-timeout` | int | Timeout in microseconds to push batch | | `live-source` | int | Set to 1 for live/dynamic sources (REQUIRED) | | `drop-pipeline-eos` | int | Set to 1 to keep pipeline alive when sources removed | | `async-handling` | int | Set to 1 for async state changes | | `select-rtp-protocol` | int | 0=UDP+TCP auto, 4=TCP only | | `latency` | int | Jitterbuffer size in ms for RTSP | **Built-in REST API Endpoints**: - `POST /api/v1/stream/add` - Add a stream dynamically - `POST /api/v1/stream/remove` - Remove a stream - `GET /api/v1/stream/get-stream-info` - Get current streams **Usage**: ```python # Pipeline with built-in REST server on port 9000 pipeline.add("nvmultiurisrcbin", "src", { "port": 9000, "max-batch-size": 16, "live-source": 1, "drop-pipeline-eos": 1, "async-handling": 1, }) # REST API automatically available at http://localhost:9000/api/v1/ ``` **⚠️ CRITICAL for Dynamic Sources**: When using dynamic source addition, the sink element MUST have `async=0`: ```python pipeline.add("nveglglessink", "sink", { "sync": 0, "qos": 0, "async": 0 # CRITICAL - prevents state transition deadlock }) ``` **Notes**: - Integrates nvds_rest_server, nvurisrcbin, and nvstreammux in one bin - Do NOT implement custom Flask/FastAPI server - use built-in REST API - See `rest_api_dynamic.md` for complete REST API documentation --- ### nvdsdynamicsrcbin **Purpose**: Source bin for programmatically adding and removing file/URI-based video sources at runtime. Unlike `nvmultiurisrcbin` (REST API / config-driven), `nvdsdynamicsrcbin` is controlled entirely through code using `SourceManager`. **CRITICAL**: `nvdsdynamicsrcbin` does **not** manage sources on its own. You **must** use `SourceManager` from `pyservicemaker._pydeepstream.signal` to add, remove, and terminate sources. Without `SourceManager`, the bin has no way to receive source URIs. **Key Properties**: | Property | Type | Default | Description | |----------|------|---------|-------------| | `gpu-id` | uint | 0 | GPU Device ID to use for decoding | | `message-forward` | bool | False | Forward all children messages to the pipeline bus (required for EOS detection) | | `async-handling` | bool | False | Handle asynchronous state changes internally | | `current-file` | string (read-only) | null | Currently processing file path | | `current-id` | int (read-only) | -1 | ID of the chunk currently being processed | **Element Actions** (triggered via `SourceManager`): | Action | Description | |--------|-------------| | `add-source` | Add a new file/URI source to the bin | | `remove-source` | Remove a source by its unique ID | | `terminate` | Signal no more sources will be added; sends EOS after all finish | **Internal Children**: Contains `parsebin`, `queue_parsebin`, and `decoder` — it automatically parses and decodes the added sources. --- ### v4l2src **Purpose**: Video4Linux2 source for USB cameras **Key Properties**: - `device`: Device path (e.g., `/dev/video0`) - `io-mode`: I/O mode - `do-timestamp`: Enable timestamping **Usage**: ```bash v4l2src device=/dev/video0 ! ... ``` **Notes**: - Standard GStreamer plugin for USB webcams, may require format conversion --- ### nvarguscamerasrc **Purpose**: NVIDIA camera source for Jetson CSI cameras **Key Properties**: - `sensor-id`: Sensor ID (0, 1, etc.) - `sensor-mode`: Sensor mode - `wbmode`: White balance mode - `exposuretimerange`: Exposure time range - `gainrange`: Gain range **Usage**: ```bash nvarguscamerasrc sensor-id=0 ! ... ``` **Notes**: - Jetson-specific plugin optimized for CSI cameras with hardware-accelerated capture --- ## Processing Plugins ### nvstreammux **Purpose**: Batches multiple video streams into a single batch for efficient inference **IMPORTANT**: There are TWO versions of nvstreammux: - **OLD nvstreammux**: Default, uses GObject properties for configuration - **NEW nvstreammux**: Enabled with `USE_NEW_NVSTREAMMUX=yes`, uses config file for advanced settings **Key Properties (NEW nvstreammux - RECOMMENDED)**: - `batch-size`: Maximum number of buffers in a batch - `batched-push-timeout`: Timeout for batching in microseconds (default: 33000) - `config-file-path`: Path to configuration file for advanced settings - `num-surfaces-per-frame`: Number of surfaces per frame - `attach-sys-ts`: Attach system timestamp as NTP timestamp (boolean) - `max-latency`: Maximum latency in live mode (nanoseconds) - `sync-inputs`: Force synchronization of input frames (boolean) - `frame-num-reset-on-eos`: Reset frame numbers on EOS (boolean) - `frame-num-reset-on-stream-reset`: Reset frame numbers on stream reset (boolean) - `frame-duration`: Duration of input frames in milliseconds for NTP correction - `drop-pipeline-eos`: Don't propagate EOS downstream when all pads are at EOS (boolean) **Key Properties (OLD nvstreammux - Legacy)**: - `batch-size`: Number of streams to batch - `width`: Output batch width - `height`: Output batch height - `gpu-id`: GPU ID for processing - `batched-push-timeout`: Timeout for batching (microseconds) - `enable-padding`: Enable padding for different resolutions - `nvbuf-memory-type`: Memory type (0=default, 1=NVMM, 2=unified) **Usage**: ```bash nvstreammux name=m batch-size=4 width=1920 height=1080 ``` **Common Pipeline Pattern**: ``` source1 ! m.sink_0 source2 ! m.sink_1 nvstreammux name=m batch-size=2 ! ... ``` **Notes**: - **Critical plugin** for multi-stream applications - **NEW nvstreammux** (recommended): More flexible, uses config file for width/height/memory-type settings - **OLD nvstreammux**: Uses GObject properties for width/height, may be deprecated in future - To use NEW version: Set environment variable `USE_NEW_NVSTREAMMUX=yes` before running pipeline - Batch size should match number of input streams - NEW version infers output resolution from downstream elements or uses config file --- ### nvstreamdemux **Purpose**: Demultiplexes batched streams back to individual streams **Key Properties**: - `name`: Element name (required for pad access) **Usage**: ```bash nvstreamdemux name=d ``` **Common Pipeline Pattern**: ``` nvstreammux name=m ! ... ! nvstreamdemux name=d d.src_0 ! ... d.src_1 ! ... ``` **Notes**: - Used after processing batched streams - Provides separate source pads for each stream - Essential for per-stream rendering or processing --- ### nvinfer **Purpose**: TensorRT-based inference engine for deep learning models **Key Properties**: - `config-file-path`: Path to inference configuration file (supports **both** INI-style text format and YAML format) - `batch-size`: Batch size for inference - `gpu-id`: GPU ID for inference - `unique-id`: Unique identifier for this inference instance - `process-mode`: Infer processing mode (primary or secondary) - `interval`: Number of consecutive batches to skip for inference - `infer-on-gie-id`: Infer on metadata from GIE with this unique ID (-1 for all) - `infer-on-class-ids`: Operate on objects with specified class IDs - `filter-out-class-ids`: Ignore metadata for objects of specified class IDs - `model-engine-file`: Path to pre-generated TensorRT engine file - `output-tensor-meta`: Output raw tensor metadata (0=no, 1=yes) - `output-instance-mask`: Output instance mask in metadata (0=no, 1=yes) - `input-tensor-meta`: Use tensor metadata from upstream (0=no, 1=yes) - `clip-object-outside-roi`: Clip object bbox outside ROI from nvdspreprocess - `crop-objects-to-roi-boundary`: Crop object bbox to ROI boundary - `raw-output-file-write`: Write raw inference output to file - `raw-output-generated-callback`: Callback for raw output - `raw-output-generated-userdata`: Userdata for raw output callback **Configuration File Structure**: nvinfer supports **two configuration formats**: ### Format 1: YAML Format (Recommended) ```yaml # Example: pgie_config.yml (Primary detector using ResNet18) property: gpu-id: 0 net-scale-factor: 0.00392156862745098 # Use ResNet18 TrafficCamNet model from DeepStream samples onnx-file: /opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx labelfile-path: /opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/labels.txt batch-size: 1 process-mode: 1 model-color-format: 0 # 0=FP32, 1=INT8, 2=FP16 network-mode: 2 num-detected-classes: 4 interval: 0 gie-unique-id: 1 # 1=DBSCAN, 2=NMS, 3=DBSCAN+NMS, 4=None cluster-mode: 2 class-attrs-all: topk: 20 nms-iou-threshold: 0.5 pre-cluster-threshold: 0.2 ``` ### Format 2: INI-style Text Format ```ini # Example: pgie_config.txt (Primary detector using ResNet18) [property] gpu-id=0 net-scale-factor=0.00392156862745098 onnx-file=/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx labelfile-path=/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/labels.txt batch-size=1 process-mode=1 model-color-format=0 network-mode=2 num-detected-classes=4 interval=0 gie-unique-id=1 cluster-mode=2 [class-attrs-all] topk=20 nms-iou-threshold=0.5 pre-cluster-threshold=0.2 ``` **Key Differences**: | Aspect | YAML Format | INI Format | |--------|-------------|------------| | File extension | `.yml` or `.yaml` | `.txt` | | Section headers | `property:` (no brackets) | `[property]` (with brackets) | | Key-value separator | `: ` (colon + space) | `=` (equals) | | Indentation | Required for nested values | Not used | **Usage**: ```bash nvinfer config-file-path=/path/to/config.yml batch-size=4 ``` **Common Pipeline Pattern**: ``` nvstreammux ! nvinfer config-file-path=pgie_config.txt ! ... ``` **Notes**: - **Primary inference engine** for object detection/classification - Supports TensorRT engines (.trt), ONNX models, and custom networks - Can be used as Primary GIE (PGIE) or Secondary GIE (SGIE) - Multiple instances can be cascaded for complex models - `output-tensor-meta=1` enables custom postprocessing - `input-tensor-meta=1` uses preprocessed tensors from nvdspreprocess - **Note**: `enable-dbscan` is DEPRECATED and is a config file parameter, not a GObject property --- ### nvinferserver **Purpose**: Inference using Triton Inference Server backend **Key Properties**: - `config-file-path`: Path to Triton configuration file - `gpu-id`: GPU ID - `unique-id`: Unique identifier - `output-tensor-meta`: Output tensor metadata **Usage**: ```bash nvinferserver config-file-path=/path/to/triton_config.txt ``` **Notes**: - Alternative to nvinfer for Triton-based inference - Supports remote inference servers - Better for scalable deployments - Requires Triton Inference Server setup --- ### nvdspreprocess **Purpose**: Custom preprocessing plugin for region-of-interest (ROI) preprocessing **Key Properties**: - `config-file`: Path to preprocessing configuration file - `gpu-id`: GPU ID **Configuration File Structure**: ```yaml preprocess-config: - preprocess-group: target-unique-ids: [1] roi-params-src: [0] process-on-roi: 1 network-input-shape: [1, 3, 544, 960] tensor-format: 0 # 0=NCHW, 1=NHWC maintain-aspect-ratio: 0 custom-transform-function: "custom_transform" custom-tensor-prep-function: "custom_tensor_prep" ``` **Usage**: ```bash nvdspreprocess config-file=/path/to/preprocess_config.yml ``` **Common Pipeline Pattern**: ``` nvstreammux ! nvdspreprocess config-file=preprocess.yml ! nvinfer input-tensor-meta=1 ! ... ``` **Notes**: - Enables custom preprocessing before inference - Processes ROIs or full frames - Outputs tensor metadata for nvinfer - Custom preprocessing library and functions are specified in the **config file**, not as GObject properties - Optimal performance: batch-size should match total units in config --- ### nvdspostprocess **Purpose**: Custom postprocessing plugin for parsing model outputs **Key Properties**: - `postprocesslib-name`: Path to postprocessing library (.so) - `postprocesslib-config-file`: Path to postprocessing configuration file - `gpu-id`: GPU ID **Configuration File Structure** (YAML): ```yaml postprocess-config: - postprocess-group: target-unique-ids: [1] custom-parse-function: "custom_parse" custom-bbox-parse-function: "custom_bbox_parse" output-format: 0 # 0=object detection, 1=classification ``` **Usage**: ```bash nvdspostprocess postprocesslib-name=./libpostprocess.so postprocesslib-config-file=config.yml ``` **Common Pipeline Pattern**: ``` nvinfer output-tensor-meta=1 ! nvdspostprocess postprocesslib-name=... ! ... ``` **Notes**: - Parses raw tensor outputs from nvinfer - Requires nvinfer with output-tensor-meta=1 - Supports custom parsing functions - Used for models not supported by nvinfer's built-in parsers --- ### nvtracker **Purpose**: Multi-object tracker for tracking objects across frames **Key Properties**: - `ll-lib-file`: Path to low-level tracker library (.so) - `ll-config-file`: Path to tracker configuration file - `tracker-width`: Tracker input width - `tracker-height`: Tracker input height - `gpu-id`: GPU ID - `input-tensor-meta`: Use tensor metadata (0=no, 1=yes) - `tensor-meta-gie-id`: GIE ID for tensor metadata (used with input-tensor-meta) - `display-tracking-id`: Display tracking ID in object text - `tracking-id-reset-mode`: Tracking ID reset mode on stream reset/EOS - `tracking-surface-type`: Selective tracking surface type - `user-meta-pool-size`: Tracker user metadata buffer pool size - `sub-batches`: Configuration of sub-batches for parallel processing - `sub-batch-err-recovery-trial-cnt`: Max trials to reinitialize tracker on error **Configuration File Structure**: ```yaml tracker: ll-lib-file: /path/to/libnvds_nvmultiobjecttracker.so ll-config-file: /path/to/tracker_config.yml enable-batch-process: 1 enable-past-frame: 1 tracker-width: 1920 tracker-height: 1080 ``` **Usage**: ```bash nvtracker ll-lib-file=/path/to/libnvds_nvmultiobjecttracker.so ll-config-file=/path/to/config.yml ``` **Common Pipeline Pattern**: ``` nvinfer ! nvtracker ll-lib-file=... ! ... ``` **Notes**: - Tracks objects across video frames - Assigns unique tracking IDs to objects - Supports multiple tracking algorithms - Requires object metadata from inference engine - Tracker dimensions should match preprocess/infer dimensions when using input-tensor-meta=1 --- ### nvdsosd (nvosdbin) **Purpose**: On-Screen Display element (`nvdsosd`) and DeepStream convenience bin (`nvosdbin`) for drawing bounding boxes, labels, masks, and clocks **Key Properties**: - `gpu-id`: GPU ID to render on - `process-mode`: Rendering backend (0=CPU, 1=GPU) - `display-text`: Enable text overlay (boolean) - `display-bbox`: Enable bounding box display (boolean) - `display-mask`: Enable instance mask display (boolean) - `display-clock`: Enable clock display (boolean) - `clock-font`: Font for clock text - `clock-font-size`: Font size for clock - `x-clock-offset`: X offset for clock position - `y-clock-offset`: Y offset for clock position - `clock-color`: Clock color (RGBA as uint) - `blur-bbox`: Enable bbox blurring (boolean) - `blur-on-gie-class-ids`: Blur bboxes for specific GIE unique ID and class ID **Note**: Text and bbox styling properties (like colors, borders) are controlled through object metadata, not as GObject properties on the plugin itself. **Usage**: ```bash nvdsosd display-text=1 display-bbox=1 ``` **Common Pipeline Pattern**: ``` nvtracker ! nvdsosd ! ... ``` **Notes**: - Use `nvdsosd` for the raw transform element - Supports tracking ID display, text overlays, and optional blur/clocks - Keeps surfaces in NVMM for zero-copy rendering on GPU - Object-specific styling (text colors, bbox colors, etc.) is set through NvDsMeta object metadata, not plugin properties --- ### nvmultistreamtiler **Purpose**: Tiles multiple video streams into a single output frame **Key Properties**: - `width`: Output width - `height`: Output height - `rows`: Number of rows in tile layout - `columns`: Number of columns in tile layout - `gpu-id`: GPU ID - `show-source`: Show source index (0=no, 1=yes) **Usage**: ```bash nvmultistreamtiler width=1920 height=1080 rows=2 columns=2 ``` **Common Pipeline Pattern**: ``` nvstreamdemux name=d d.src_0 ! ... d.src_1 ! ... ! nvmultistreamtiler ! ... ``` **Notes**: - Combines multiple streams into a grid layout, useful for multi-stream visualization --- ### nvvideoconvert **Purpose**: Video format converter (color space conversion, scaling) **Key Properties**: - `gpu-id`: GPU ID - `nvbuf-memory-type`: Memory type - `src-crop`: Source crop rectangle - `dest-crop`: Destination crop rectangle **Usage**: ```bash nvvideoconvert gpu-id=0 ``` **Common Pipeline Pattern**: ``` nvdsosd ! nvvideoconvert ! nveglglessink ``` **Notes**: - GPU-accelerated color format conversion (NV12, RGBA, etc.), often needed before rendering sinks --- ### nvdsanalytics **Purpose**: Video analytics plugin for motion detection, line crossing, etc. **Key Properties**: - `config-file`: Path to analytics configuration file - `enable`: Enable analytics (0=no, 1=yes) - `gpu-id`: GPU ID **Configuration File Parameters**: The config file **must** include a **property** group/section. Other groups define per-stream ROI, line-crossing, overcrowding, and direction rules. Stream index is given by the numeric suffix in the group name (e.g. `roi-filtering-stream-0` for stream 0). - `property`: General group; Mandatory. - `config-width`,`config-height`: Reference resolution width and height for analytics coordinate scaling. - `enable`: Whether analytics is enabled (aligned with the element **enable** property). - `display-font-size`: Optional; OSD font size. - `osd-mode`: Optional; 0, 1, or 2. 0 = OSD off, 1 = labels only, 2 = full (default). - `obj-cnt-win-in-ms`: Optional; object-count time window in milliseconds; range 1–1000000000. - `display-obj-cnt`: Optional; whether to show per-class object counts on OSD. - `roi-filtering-stream-`: ROI Filtering group per stream - `enable`: Enable ROI filtering for this stream. - `class-id`: Class IDs to include in ROI analytics (semicolon-separated integer list). - `inverse-roi`: Whether treat as “outside ROI” for counting/filtering. - `roi-