Flexpectation

Project partner: Open Climate Fix

Flexpectation aims to deliver accurate short-term (1 hour to 2 weeks ahead) forecasts of demand and embedded generation for every NGED primary substation. By improving visibility of network loading during outages and abnormal running, the tool will help target flexibility more efficiently and reduce unnecessary procurement. Ultimately, it will give Control and Planning teams a clearer, data-driven picture of upcoming constraints to support faster, and more confident, operational decisions.

Problem(s)

At present, NGED’s short-term forecasting remains a largely manual process, reflecting an era when network behaviour was more predictable. In practice, engineers estimate future demand by reviewing recent load flow profiles and assuming tomorrow will resemble yesterday, without accounting for the influence of weather, solar output, or customer behaviour. This limited approach offers little resilience under abnormal running, such as during outages or planned switching. In addition, embedded generation, particularly rooftop PV, masks true demand at primary substations, while data needed for accurate modelling is fragmented across systems with inconsistent formats and limited interoperability. NGED’s Energy Management Centre faces a similar challenge, relying on manual, short-horizon forecasts and over-procuring flexibility as a safeguard against uncertainty. The result is a process that is reactive, labour-intensive, and risk-prone, leading to unnecessary operational costs and under-utilised capacity. These are issues Flexpectation aims to resolve through a data-driven, automated forecasting framework.

Method(s)

Flexpectation is structured around a dual-track, two-phase delivery model designed to test how forecasting methodologies can offer immediate operational value, and longer-term innovation exploring state of the art Artificial Intelligence methodologies. The Live Service Track will deliver an operational forecasting API using machine-learning techniques for initial testing and benchmarking, while the Research Track will run in parallel to investigate more advanced modelling approaches. Both tracks will progress through Phase 1, focused on the development of a minimum viable product (MVP), and Phase 2, which will scale the solution to cover all 1,161 primary substations across the NGED licence area.

Live Service Track

The Live Service Track will demonstrate a ‘version one’ operational forecasting system within the first 12 months. This component focuses on producing an MVP capable of generating probabilistic net-load, wind, and solar forecasts with a 14-day horizon for a selected group of primary substations. Initially, forecasts will be deployed to 10–20 sites to enable early validation. The modelling approach will build on Open Climate Fix’s established architectures, such as the Temporal Fusion Transformer, combining numerical weather predictions with recent substation power-flow data at half-hourly resolution. Estimates of solar and wind generation will be derived using physical models informed by the Embedded Capacity Register (ECR) and SMITN outputs. Periods of abnormal topology will be identified using changepoint-detection techniques and excluded from training data. Once operational, the forecasting engine will be provided via an API integrated with NGED’s IT systems. Feedback from this phase will feed into the development of Version 2, which will ultimately expand coverage to all NGED substations.

Research Track

Running concurrently, the Research Track will explore advanced methodologies to address incomplete data, DER disaggregation, and inference of network topology. Research activities will include developing Graph Attention Networks to better model electrical relationships between substations; applying contrastive learning and synthetic data approaches to estimate DER capacities where telemetry is absent; and testing retrieval-augmented methods that identify “similar days” from historical data to improve forecast accuracy. This track will generate open-source artefacts and benchmarked models, assessing performance, interpretability, and computational efficiency. The strongest techniques will be integrated into later releases to develop a more advanced version 2 of the system.

Work Package Structure

Work Package 1 focuses on research and development led by OCF. This includes project mobilisation, securing sponsorship engagement, and acquiring all required datasets such as network telemetry, weather feeds, and market data. Additional activities include updating the Embedded Capacity Register based on SMITN insights, performing data validation, and beginning development of the Version 1 model. R&D on a wide range of modelling techniques will also begin. Two milestones fall within this package: by April 2026 OCF will deliver the full requirements set, solution architecture design, a summary of data sources and API setup, and a common suite of forecast-skill metrics; by July 2026 they will provide a state-of-the-art review of forecasting techniques and a first R&D progress report. This work package runs for six months.

Work Package 2 develops the minimum viable product model, also led by OCF, over a six-month period. The team will select a suitable GSP group and 10 primary substations for testing model transferability. Version 1 of the forecasting model will be implemented in a live test environment via the API. The package also includes testing candidate forecasting approaches, developing automated detection of abnormal network running, and continuing experimentation across modelling families. The milestone for this stage is the delivery of Version 1 for the test sites, alongside a summary report comparing model approaches and recommending the method most likely to achieve the highest accuracy.

Work Package 3 covers model acceptance and testing, led by NGED over two months. NGED will compare competing modelling practices, assess input-feature importance, and quantify uncertainty in relation to flexibility procurement KPIs. The milestone for this stage comprises a comparison of existing versus improved forecasts and their associated flexibility volumes, alongside user feedback provided to OCF. This stage also includes a formal stage gate prior to wider rollout.

Work Package 4 delivers the network-wide scale-up over six months. OCF will refine the version 1 model based on operational feedback and incorporate the most successful R&D techniques from earlier stages into version 2. The model will then be scaled to all 1,161 primary substations, as well as our BSPs and GSPs, with improved separation of unmasked demand from embedded generation. The milestone requires delivery of the half-hourly forecasting model at full scale, along with a comprehensive performance evaluation and implementation recommendations.

Work Package 5 focusses on the testing of the Version 2 model in BAU contexts. Over three months, OCF will support operational adoption, maintain the live service, and continue iterative improvements informed by user feedback. The milestone includes a full user acceptance testing report and a final R&D summary of forecasting techniques and future model development opportunities.

Work Package 6 focuses on cost–benefit analysis, led by NGED over three months. NGED will prepare a full business-case assessment for BAU adoption, evaluating implementation costs, reduced flexibility spend, and efficiency gains for the control room. OCF will supply supporting data and performance evidence. This stage culminates in the delivery of an Ofgem-compliant CBA approved by NGED Finance Business Partners.

Work Package 7 covers project closedown and dissemination over three months. NGED will prepare the closedown report, implement guidance documentation, and all required ENA reporting. Dissemination activities could include participation in conferences such as EIS, CIRED, IET and 25 to Zero, as well as online workshops.

• Scope

  The scope of Flexpectation is to design, develop and demonstrate a short-term probabilistic forecasting platform that enables DSOs to operate a more dynamic, decarbonised electricity system efficiently and safely. Its objectives include delivering accurate half-hourly forecasts of demand and embedded generation across all primary substations, improving operational awareness during abnormal running, and enabling more targeted and proportionate use of flexibility services. These capabilities will provide net benefits for consumers by reducing avoidable operational costs, enhancing network reliability, and supporting the integration of low-carbon technologies such as EVs, heat pumps and distributed solar, thereby contributing to environmental goals and a fairer energy transition. Financial benefits will directly accrue to the GB electricity distribution system through more efficient procurement of flexibility, reduced reliance on conservative operating margins, and the potential to defer reinforcement where improved foresight allows the network to be managed more effectively. Over time, these efficiencies will help lower the cost of running the distribution network, ultimately reducing customer bills while improving service quality.

• Objectives
  • Develop an operational-ready short-term forecasting platform that provides accurate, probabilistic forecasts of demand and embedded generation.
  • Strengthen network visibility and situational awareness by enabling reliable detection of switching events and improving estimation of distributed energy resources.
  • Support more efficient and targeted use of flexibility services by reducing uncertainty in operational decision-making.
  • Generate and share innovative forecasting methodologies and open-source tools that advance sector-wide understanding of DER disaggregation, topology inference, and spatio-temporal forecasting.
  • Establish a robust business case and pathway for BAU adoption, demonstrating financial and operational benefits, securing user acceptance, and ensuring the solution integrates effectively with NGED’s internal processes.
• Success criteria

  Demonstrated improvement in forecasting accuracy at primary substations

  The project should significantly outperform NGED’s current manual heuristics by reducing forecast error for half-hourly demand and DER output out to 14-day horizons.

  Successful delivery of Version 1 forecasting API

  The Live Service Track must deliver an MVP system for 10–20 substations, meeting NGED’s operational integration, data latency and security requirements.

  Ability to scale to a fully operational Version 2 service covering all primary substations

  Version 2 must incorporate learnings from the research track and deliver consistent, reliable probabilistic forecasts across primary, BSP and GSP levels.

  Accurate detection and handling of switching events

  A core success measure is whether the system can infer abnormal network topology changes from time-series data and maintain forecast reliability during such periods.

  Improved estimation and disaggregation of DER output

  The project should produce reliable estimates of solar/wind generation and total DER behaviour per substation, despite incomplete telemetry and ambiguous ECR data, enabling more accurate constraint management.

  Production of comprehensive research outputs informing future operational forecasting

  Success requires delivering validated research findings on model architectures, DER disaggregation, topology inference, uncertainty quantification, and similar-day retrieval methods.

  A complete business case demonstrating net financial benefit for BAU adoption

  The final success criterion is the production of an Ofgem-compliant CBA showing savings from reduced flexibility spend, fewer overload incidents, and efficiency gains in the control room.

• Potential for new learning

  The project will generate the following learning that can be used by network licensees:

  • Insights into which data sources, model architectures, and features deliver the highest forecasting accuracy.
  • Learning on how to infer and disaggregate DER behaviour from incomplete or noisy system data.
  • Methods for detecting, interpreting, and correcting the impact of switching events on substation time-series data.
  • Evidence of the operational value of improved forecasts, including impacts on flexibility procurement, constraint management, and control-room decision-making.
  • Practical lessons on integrating ML-based forecasting into DNO IT, EMC and ADMS workflows.
  • Open documentation of methodologies and results, enabling other network licensees to replicate or adapt the techniques.
  • Benchmarking and comparison of multiple modelling approaches, supporting future innovation across the sector.

  Learning will be disseminated through NGED’s proven mechanisms. This includes (but is not limited to):

  • Reports
  • Workshops
  • Stands at NGED and other organiser events
  •  Website updates
  • Publications at national and international conferences