en.Wedoany.com Reported - On July 13, China's SenseTime officially released and opened the SenseNova-Vision unified visual large model for understanding and generation, providing unified model capabilities for tasks such as object detection, optical character recognition, image segmentation, depth prediction, surface normal estimation, multi-view geometry, and camera pose estimation. Unlike previous technical approaches that configured separate models, prediction heads, and decoders for different visual tasks, SenseNova-Vision transforms multiple types of computer vision tasks into text generation, image generation, or mixed text-image generation problems, enabling visual perception capabilities to directly enter the input-output system of general multimodal models.
This release includes more than just model weights. SenseTime has publicly released the SenseNova-Vision-7B-MoT model, inference code, technical report, and the SenseNova-Vision-Corpus-50M visual corpus, along with entry points for single-image inference, interactive inference, web demos, and benchmark evaluations. Official project records show that the aforementioned model, dataset, inference code, and technical report were gradually made available online starting July 8, with full release information further disclosed on July 13.
Traditional computer vision systems typically establish independent technical pipelines for each task. Object detection requires outputting categories and bounding boxes, image segmentation needs to generate pixel-level masks, depth prediction calculates spatial distances for each pixel, and 3D reconstruction must handle multi-view images, point maps, and camera parameters. These tasks previously required different model structures, training objectives, specialized prediction modules, and data processing methods. SenseNova-Vision attempts to eliminate this fragmentation between tasks, no longer adding dedicated architectural branches for detection, segmentation, depth estimation, or geometric prediction, but instead allowing the same model to identify task types, target regions, observation perspectives, and output rules based on natural language instructions.
After unification, different visual results still adopt expression forms suitable for their own characteristics. Structured information such as object categories, bounding boxes, coordinate points, text recognition results, human keypoints, and camera parameters can be directly generated as text records by the model; pixel-level results such as segmentation masks, depth maps, surface normal maps, and multi-view 3D point maps are generated in image form. For combined tasks that require both specifying object categories and outputting corresponding segmentation regions, the model can simultaneously generate text and images, allowing the same interaction interface to cover structured visual understanding, dense geometric prediction, image segmentation, and multi-view visual geometry.
Natural language instructions become a key entry point connecting various visual tasks. Developers can specify target categories, colors, regions, perspectives, and output formats through text descriptions, and can also add visual prompts to designate processing objects, after which the model generates parseable results according to the agreement. This approach frees visual tasks from being entirely constrained by fixed category lists and preset evaluation formats; for example, users can combine categories, colors, and local region conditions through language to form more flexible visual processing requirements than traditional fixed tasks.
To support this unified training approach, SenseTime constructed the SenseNova-Vision-Corpus-50M corpus. This corpus converts visual annotations originally scattered across tasks such as detection, text recognition, keypoint localization, image segmentation, depth estimation, and multi-view geometry into a unified sample structure of "visual input, natural language instruction, decodable answer," with answer forms covering text, images, and mixed text-image content. The training process primarily uses this visual corpus, supplemented with auxiliary multimodal data, to reduce the loss of general understanding and image generation capabilities during the model's enhancement of visual abilities.
From the scope of publicly available tasks, SenseNova-Vision currently covers four main categories of visual capabilities. Structured visual understanding includes object detection, referring localization, text recognition, UI element localization, and keypoint prediction; dense geometric prediction includes monocular depth estimation and surface normal prediction; the segmentation part covers general segmentation, referring segmentation, reasoning segmentation, interactive segmentation, and segmentation tasks with semantic descriptions; multi-view visual geometry includes 3D point map reconstruction and camera pose estimation. Official benchmark results show that a single model can compete with some specialized models and general visual models across various output formats and visual tasks, but performance is not entirely consistent across different tasks.
After the model's release, developers can run preset examples through the official code or specify task types, text prompts, and input images for single inference. The official team also provides a local web demo solution based on Gradio, with the complete demo recommending the use of a GPU with 80GB of memory; for executing all benchmark evaluations, the official recommendation is at least one server equipped with eight 80GB GPUs. This indicates that while SenseNova-Vision already provides a complete inference entry point, it still requires significant computational resources for full-task deployment and evaluation.
There are also clear boundaries regarding the open license. The SenseNova-Vision model weights are licensed under CC BY-NC 4.0, primarily for non-commercial use, and the dataset page is marked with the same license; the source code in the official GitHub repository should be used separately according to the license listed in the code repository. Therefore, this "full open-source" mainly means that the model, training corpus, inference code, technical report, and evaluation methods have been made publicly available to the research community, but does not mean that all content can be directly used in commercial products without restrictions.
SenseTime also listed current limitations in the model description. The unified model does not mean it surpasses specialized systems in every professional task; some specialized models may still maintain advantages in specific benchmarks. Text output still requires parsing programs configured according to the task, and image results also need to be decoded based on training protocols. Results such as depth, surface normals, 3D point maps, and camera pose estimates must still be verified through independent systems before being used in robotics, autonomous driving, industrial inspection, or other high-safety-requirement scenarios, and the model's generated results cannot be directly used as the final control basis.
The main technical change accomplished by SenseNova-Vision this time is reorganizing previously scattered classic visual tasks into generative tasks that general multimodal models can handle. Detection, segmentation, depth prediction, and 3D geometry no longer correspond to separate isolated systems but instead share natural language instructions, visual input, and text-image generation space, providing a new implementation path for further integrating visual perception, language understanding, image generation, and spatial reasoning into the same foundational model.
