The European power grid is undergoing what is arguably the biggest shake-up in its history to combat climate change. This has seen a rapid and accelerating expansion in installed renewables, coupled with electrification of heating, transport and industry.
These changes have already made stable grid operation trickier.
From a high-level perspective, the grid operator needs to make sure electricity demand and supply are equal at all times. When this is not the case, frequency deviations occur, which left unchecked can lead to large scale blackouts.
Historically, the primary uncertainty in the demand-supply equation came from the demand side. However, with so many renewables, the supply side has increasingly become uncertain too, requiring forecasts for wind and solar generation which rely on weather predictions themselves.
How much more uncertain is grid operation now, you ask?
A lot more in fact: between 2017 and 2023, forecast errors for both solar and wind have more than doubled in Europe. This places a question mark on grid stability, and also causes financial losses to end consumers.
Exhibit 1: Absolute forecast errors in 16 EU countries for (left) solar, and (right) wind 
In the next ten years, we expect renewable generation to further quadruple, which makes the status quo untenable: the grid cannot afford a quadrupling of forecast errors.
These changes on the generation side are accompanied by a changing demand. With more electric vehicles and heating, as well as electrification of industry, most European countries will see a displacement of existing energy demand to be replaced by (hopefully) cleaner electricity.
This further fuels uncertainty on grid operation.
Data science can help address these challenges, and much more.
On the one hand, we need better forecasts for a lot of variables: generation, demand, weather, as well as market operation and grid conditions. This reduces the need for balancing services, but also create lots of opportunity for enterprises offering data science services.
Nevertheless, even with the most advanced tools, there will always be forecast errors. Additionally, at a more local or regional level, different concerns arise as well such as congestion management in distribution grids (if you’re reading this in the Netherlands, this might especially resonate with you).
Exhibit 2: Grid congestion map of the Netherlands 
This leads to another big opportunity in applying data science techniques is in leveraging flexibility on the demand side, especially to address the balancing needs on the transmission grid level or for congestion management on the distribution grid. With increasing price volatility, this opens up the door for economic arbitrage as well.
In fact, the Belgian grid operator, Elia, expects to save 200 million euros annually by the activation of this demand flexibility. The targets are truly ambitious too: 70% of the newly electrified industrial demand is assumed to be flexible by 2030. At the residential level: two thirds of EVs, one third of HPs and more than half of home batteries are assumed to be ‘smart’ in some way.
How do we go from a system where these devices are not even present to one where they are responsive to not just grid requirements, but also constrained on user preferences and utility? One thing to keep in mind here is that the energy sector is heavily regulated. So, just as we do not want an unvalidated AI offering medical advice, there are strict requirements on the types of models that can be used to make predictions in practice.
Addressing these challenges, as well as many others such as predicting outages for wind turbines, will require energy engineers to know not just about cutting-edge AI techniques, but also to master them. This will require knowledge of the algorithms, and the regulatory and economic frameworks surrounding their application.
Interested in leading this innovation in the energy sector? Check out the new master certificate programme AI for Smart Energy where you can boost your skills to take advantage of all these opportunities.
by Prof. dr. Hussain Kazmi, instructor AI for smart energy master certificate programme and InnoEnergy PhD School alumnus.
 Kazmi, Hussain, and Zhenmin Tao. “How good are TSO load and renewable generation forecasts: Learning curves, challenges, and the road ahead.” Applied Energy 323 (2022): 119565.
 Capaciteitskaart Netbeheerder (2023).