Street Manager Open Permit Data

Project Overview

We developed an end-to-end analytical pipeline to visualise and analyse the impact of street works across England’s local highway authorities. This project demonstrates how to transform complex operational datasets into actionable insights through modern data engineering practices.

The Challenge

Simply put, street works data is inherently complex.

Overall, there are around 3 million completed street work permits raised every year in England, with around 450,000 completed in London alone.

The pace and scale of this activity is immense and the effects are far reaching.

Whether it be:

• Reparing old gas mains
• Maintaining the water / sewage networks
• Increasing local electricity supply
• Rolling out full fibre
• Installing new infrastructue such as EV charge points / Heat Networks
• And much more

Absolutely everyone depends on street works in some way shape or form.

Our goal was to create a user-friendly visualisation tool that could effectively communicate the impact of street works on local roads across England.

For the Community

We built this as an open-source project to:

• Demonstrate transparent, reproducible analysis of street works data
• Share best practices for handling public sector datasets
• Enable collaboration and improvement through community contributions in the future

Data Processing Pipeline

Our solution combines several modern tools and techniques to create an automated pipeline that processes and visualises street works data.

1. Data Integration & Authoritative Sources
   • Integrated the Department for Transport’s Street Manager permit data with Ordnance Survey’s authoritative USRN (Unique Street Reference Number) dataset
   • Implemented automated monthly updates to maintain data freshness and accuracy

Why USRNs Matter: Each USRN provides a persistent, unique reference for every street across Great Britiain. It enables accurate linking of street-related data across different systems and organisations. This standardisation is crucial for reliable spatial analysis and data integration. Learn more about USRNs at GeoPlace.

2. Impact Analysis

   • Developed a scoring methodology that considers:
     • Traffic-sensitive road classifications
     • Road closure impacts
     • The type of street work completed (Major, Standard, Minor, Emergency)

3. Visualisation

   • Created interactive heat maps showing impact scores
   • Filtered views by local authority
   • Monthly automatic updates

Tech Stack

• Python (Data processing and analysis logic)
• Streamlit (Prototype dashboard tool)
• Streamlit-Folium (Map visualisation)
• GeoPandas (Geospatial data processing)
• Terraform (Cloud infrastructure management)
• AWS Fargate (Cloud deployment)
• Docker (Application containerisation)
• MotherDuck (Serverless data warehouse)
• Dbt (Analytics engineering)

Results and Resources

• Interactive Map: View the Prototype Map Visualisation
• Monthly Reports: Word on the Street Dashboard
• Source Code: GitHub Repository

Future Improvements

We’re actively working on enhancing this analysis with:

• Refined impact scoring methodology
• Additional data source integration
• Enhanced visualisation features

Tools Spotlight

This project showcases the effectiveness of modern data tools:

• MotherDuck: Serverless data warehouse enabling efficient analytical queries
• Evidence: Streamlined data applications with SQL and Markdown
• Streamlit: Rapid prototyping and application development
• Dbt: A tool to automate and structure the transformation of data in data warehouses with standardised SQL workflows

Whether you’re interested in street works analysis or building similar data pipelines, we’d love to hear your thoughts and suggestions for improvement.

Drop us an email at hello@enmeshed.dev.

---