Building Directed Graphs from OSM PBF Files

Transforming raw OpenStreetMap Protocolbuffer Binary Format (PBF) extracts into computationally efficient directed graphs is a foundational step in geospatial routing and network analysis automation. Unlike undirected representations, directed graphs explicitly encode traversal constraints, enabling accurate modeling of one-way streets, turn restrictions, lane configurations, and asymmetric travel costs. For logistics engineers, GIS developers, and Python backend teams, mastering this transformation pipeline ensures that downstream routing algorithms operate on topologically sound, production-ready networks.

This workflow aligns with established practices in OSM Graph Architecture & Network Modeling, where graph topology, attribute mapping, and traversal logic are decoupled for maintainability and scale. Building Directed Graphs from OSM PBF Files requires careful attention to memory management, tag semantics, and topological validation to prevent silent routing failures in production environments.

Prerequisites & Environment Setup

Before implementing the extraction pipeline, ensure your environment meets the following baseline requirements:

  • Python 3.9+ with pip or uv package management
  • Core Libraries: pyosmium (streaming PBF parser), networkx (graph construction), shapely (geometry validation), numpy (vectorized coordinate operations)
  • System Resources: Minimum 8 GB RAM for regional extracts (e.g., state-level PBFs); 16+ GB recommended for national datasets
  • Data Source: Geofabrik extracts or official OSM PBF dumps (.osm.pbf)
  • Familiarity: Graph theory fundamentals, coordinate reference systems (CRS), and OSM tagging conventions

Install dependencies via:

pip install pyosmium networkx shapely numpy

Step-by-Step Extraction Pipeline

The transformation pipeline follows a deterministic sequence: ingestion → filtering → directionality resolution → graph assembly → validation. Each stage is designed to minimize memory footprint while preserving topological integrity.

1. Streaming PBF Ingestion

PBF files are highly compressed and structured for sequential reading. Loading entire files into memory is impractical for production workloads. Instead, use a streaming handler that processes nodes and ways incrementally. This approach aligns with best practices documented in How to Extract OSM Road Networks with Osmium, where memory-efficient parsing prevents OOM failures during large-scale extractions.

The pyosmium library implements a callback-based architecture. You define a custom handler class that inherits from osmium.SimpleHandler and override the node() and way() methods. The parser streams binary chunks, decodes them, and invokes your callbacks without materializing the entire dataset in RAM. For detailed implementation patterns, consult the official PyOsmium Documentation.

2. Topological Directionality Resolution

OSM ways are inherently undirected sequences of node references. Directionality must be inferred from tagging semantics and applied during graph construction:

  • oneway=yes → forward traversal only
  • oneway=-1 or oneway=reverse → backward traversal only
  • oneway=no or absent → bidirectional (two directed edges)
  • junction=roundabout → implicit forward directionality
  • lanes + oneway combinations → lane-aware routing (optional, requires advanced parsing)

Directional resolution requires splitting multi-segment ways into directed edges while preserving node connectivity. When a way is bidirectional, you create two edges: (u, v) and (v, u). For one-way segments, you create a single directed edge matching the node sequence or its reverse. This logic must account for implicit rules, such as highway=motorway defaults in certain jurisdictions, though explicit tagging should always take precedence in automated pipelines.

3. Directed Graph Assembly

Once directionality is resolved, edges are inserted into a networkx.DiGraph. Nodes are indexed by their OSM ID, and coordinates are stored as attributes for spatial queries. Geometry validation using shapely ensures that node sequences do not contain duplicate consecutive points or self-intersections that could corrupt routing solvers.

During assembly, it is critical to decouple topology from business logic. Store raw OSM tags as edge attributes, then apply transformation functions downstream. This pattern supports Mapping Node Attributes for Urban Delivery Zones, where delivery constraints, loading zones, and access restrictions are mapped to graph nodes without altering the underlying routing topology.

Below is a production-oriented implementation skeleton:

import osmium
import networkx as nx
import numpy as np

class DirectedGraphBuilder(osmium.SimpleHandler):
    def __init__(self):
        super().__init__()
        self.nodes = {}  # OSM ID -> (lat, lon)
        self.graph = nx.DiGraph()

    def node(self, n):
        # Store coordinates for later edge construction
        self.nodes[n.id] = (n.location.lat, n.location.lon)

    def way(self, w):
        # Filter for routable highways
        if w.tags.get("highway") not in ("motorway", "trunk", "primary", 
                                         "secondary", "tertiary", "residential", 
                                         "unclassified", "living_street", "service"):
            return

        # Resolve directionality
        oneway = w.tags.get("oneway", "no").lower()
        is_roundabout = w.tags.get("junction", "").lower() == "roundabout"
        
        node_refs = [n.ref for n in w.nodes if n.ref in self.nodes]
        if len(node_refs) < 2:
            return

        # Create directed edges
        edges_to_add = []
        if oneway in ("yes", "-1", "reverse") or is_roundabout:
            direction = -1 if oneway in ("-1", "reverse") else 1
            refs = node_refs if direction == 1 else node_refs[::-1]
            edges_to_add.append((refs, True))
        else:
            # Bidirectional
            edges_to_add.append((node_refs, True))
            edges_to_add.append((node_refs[::-1], True))

        for refs, _ in edges_to_add:
            for u, v in zip(refs, refs[1:]):
                if not self.graph.has_edge(u, v):
                    self.graph.add_edge(u, v, 
                                        highway=w.tags.get("highway"),
                                        maxspeed=w.tags.get("maxspeed"),
                                        length=0.0) # Calculate later via Haversine/Geodesic

    def build(self, pbf_path):
        self.apply_file(pbf_path)
        # Add node coordinates as attributes
        for n_id, coords in self.nodes.items():
            if n_id in self.graph.nodes:
                self.graph.nodes[n_id]["lat"], self.graph.nodes[n_id]["lon"] = coords
        return self.graph

# Usage
builder = DirectedGraphBuilder()
G = builder.build("region.osm.pbf")

4. Validation & Topological Sanitization

Raw OSM data contains inconsistencies that can fragment routing networks or create impossible traversals. After graph assembly, run validation routines to ensure algorithmic readiness:

  • Connectivity Checks: Identify weakly connected components. Isolated subgraphs often result from missing connector nodes or tag misclassifications.
  • Degree Analysis: Nodes with degree 1 (dangling edges) usually indicate incomplete ways or cul-de-sacs. Flag them for manual review or prune if they fall outside routing bounds.
  • Geometry Consistency: Verify that edge lengths are positive and coordinates align with expected CRS (WGS84 by default). Use shapely.geometry.LineString to validate way geometries when reconstructing paths.
  • Duplicate Edge Resolution: Ensure parallel edges between the same node pair are consolidated or explicitly tagged with distinct attributes (e.g., different lanes or turn restrictions).

For distributed or large-scale deployments, refer to strategies for Graph Fragmentation Prevention in OSM Data to implement spatial partitioning and boundary stitching without breaking routing continuity.

Production Optimization & Scaling

Streaming parsers prevent memory exhaustion, but graph construction still scales with node/edge density. Implement the following optimizations for enterprise workloads:

  1. Batch Processing & Chunking: Process PBF files in geographic tiles or administrative boundaries. Merge subgraphs using spatial overlap zones to maintain routing continuity across tile edges.
  2. Sparse Matrix Conversion: Convert networkx.DiGraph to SciPy sparse adjacency matrices for high-performance routing algorithms (Dijkstra, A*, Contraction Hierarchies).
  3. Attribute Projection: Precompute edge weights (time, distance, fuel consumption) during ingestion rather than at query time. This aligns with Configuring Edge Weights for Freight Logistics, where vehicle class, payload, and road surface conditions dictate dynamic traversal costs.
  4. Indexing: Build spatial indexes (e.g., R-tree via rtree or pygeos) on node coordinates to accelerate nearest-neighbor lookups for origin/destination snapping.

When working with national or continental datasets, consider offloading graph construction to compiled libraries like osmium CLI tools or OSRM’s osrm-extract, which implement highly optimized C++ pipelines. Python handlers remain ideal for custom attribute extraction, experimental routing logic, and integration with data science workflows.

Next Steps in Network Architecture

Once your directed graph is validated and optimized, the pipeline naturally extends into specialized routing configurations. You can layer multi-modal transit networks, calibrate speed profiles for heavy vehicles, or implement elevation-aware cost functions. Each extension relies on the same foundational topology: a clean, directed, attribute-rich graph derived from authoritative OSM PBF extracts.

Maintain version control over your extraction scripts and tag-filtering logic. OSM tagging conventions evolve, and routing accuracy depends on consistent interpretation of semantic changes. Document your directionality resolution rules, weight calibration formulas, and validation thresholds to ensure reproducibility across engineering teams and deployment environments.