Vehicle Localization

Automated vehicles need to know both where they are and which direction they are facing before they can plan and execute a movement. Camera-based localization offers an alternative to costly sensing and mapping systems, but urban streets often look alike. Similar building facades, road layouts, and textures can leave several different locations appearing equally plausible. Our research investigates how identifiable landmarks can resolve this ambiguity and improve vehicle localization in urban environments.

Semantic Point-of-Interest-Constrained Visual Localization

We develop a visual localization method that uses Point-of-Interest (POI) landmarks, such as named shops and restaurants, to distinguish visually similar locations. As shown in Figure 1, the method first matches features from a street-view image with features extracted from an OpenStreetMap (OSM) representation to generate candidate vehicle positions and headings. It then recognizes POIs in the image and checks their consistency with map annotations. Distance, viewing-angle, and matching constraints help reject candidate poses that cannot explain the observed landmarks. By combining image–map matching with semantic and geometric verification, the method narrows an ambiguous set of possible locations to a more confident estimate of the vehicle’s pose.

POI-constrained localization framework combining image and map feature extraction, map matching, POI recognition, and pose selection — Figure 1. POI-constrained visual localization framework, combining image–map matching with semantic landmark recognition and geometric verification.

Experiments on the Mapillary Geo-Localization dataset evaluate both position and orientation accuracy against existing localization methods. Compared with OrienterNet, the manuscript reports reductions of 2.18 m in average position error and 9.39° in average orientation error. Figure 2 illustrates how POI constraints refine initially ambiguous map-matching results. In the two examples shown, position errors decrease from 21.6 m to 5.40 m and from 28.4 m to 3.26 m after incorporating landmark information. These examples show how semantic landmarks can help distinguish candidate locations that share similar geometric and visual features.

Two localization examples showing input images, neural map features, initial position estimates, recognized POIs, and refined position estimates — Figure 2. Two examples of POI-constrained localization. Each row shows the input image, neural map, initial position estimate, POI recognition, and refined position estimate. Position errors are annotated in the initial and final map panels.

Research Products

Wang, Z., Wang, J., Chen, J., Li, S., Li, Y., and Chen, Z. (2025). Semantic point-of-interest constrained visual localization for autonomous driving in urban environments. Under review.