Demystifying and Debunking ML Part 4: Strategies for Understandable Models and Conclusions

In the final part, Hugh Stitt, Joe Emerson, Carl Jackson and Robert Gallen examine how machine learning models can be made more transparent, giving chemical engineers the insight needed to question results, challenge assumptions and demand better use of domain knowledge

WITHOUT being able to understand why a particular prediction has been made by a model, it is difficult to trust that the model is performing as intended and indeed that it will continue to perform as intended. We have encountered multiple cases where a data-driven model has used “faulty reasoning” to arrive at a correct conclusion. This leaves the model vulnerable to catastrophic failure when encountering new data.

Explainable AI (XAI) techniques are an invaluable tool to protect ourselves from this scenario. They can help detect errors, such as data leakage or overfitting, and they enable exploration of relationships, which is the foundation of scientific discovery.

One key caveat must be stated, even if it seems obvious; if the model has poor predictive power (ie it is a bad model), then XAI methods will give little insight. They only evaluate the model’s responses to inputs. If those responses bear little resemblance to reality, then the insight becomes somewhat moot. 

Making the models more understandable

Interpretability of first-principles models derives from the fact that we can see the impact of changing inputs, both direct and second order, in the explicit expression. However, for most industrially relevant first-principles models, it is not straightforward to udnerstand how they will behave across the entire results space by simply inspecting the equations. Writing down the Navier-Stokes equation will not enable you to trivially state the pressure drop across a particular valve geometry for a given flow, nor provide much beyond basic intuition (“more pressure drop is more flow”). To bridge this gap, parametric sensitivity methods – which quantify how changes in input parameters affect model outputs - have been applied. They allow engineers to explore how geometry, flow conditions or other factors influence results, revealing relationships that would otherwise remain hidden. In this respect, the perceived "gap" in explainability between machine learning and first-principles approaches is smaller than is often assumed.

There is a common perception that it is not possible to understand the reason why a given data-driven model makes a particular prediction. In reality, this perception is largely outdated

There is a common perception that it is not possible to understand the reason why a given data-driven model makes a particular prediction. In reality, this perception is largely outdated, with XAI techniques giving us insights into the why without needing to explicitly trace out the how. The field of explainability still lags behind the cutting edge but even Large Language Models are falling under the scope of XAI.¹

Supervised learning on tabular data

For tabular data – the most common application in engineering – supervised learning typically takes the form of regression or classification. Just as with first-principles models, sensitivity analysis is key to understanding the influence of inputs on outputs. Sensitivity approaches broadly fall into two categories: local and global.

Local sensitivity analysis is intuitive; for a given prediction, individual inputs can be adjusted to see how the model output responds. This approach gives direction and magnitude and can be very useful to explore issues such as stability around an optimal operating point. Some commercial software provides intuitive graphical interfaces to do this. The limitation of local analysis is that applicability is only local and model behaviour can vary significantly over the domain. It is possible to explore the entire result space with local sensitivity analysis but it becomes cumbersome with large numbers of inputs.

Global sensitivity analysis evaluates how important each input is across the entire region of model validity. One simple method, known as permutation importance, is closely related to parametric sensitivity. It randomises one input at a time and measures the impact on model performance, without retraining. Some model classes, such as tree based models, provide native measures, where how often a variable is used in deci- sion-making indicates its relative importance. Crucially, these approaches do not give you a direction of importance; indeed, the same feature may have strong impact in very different ways across the whole model domain. These feature importances are also relative, meaning an irrelevant feature that contributes to overfitting may still appear important. 

Another global approach is the calculation of Shapley (or SHAP) values,² which quantify how much each individual input contributes to the model's prediction at each point in the dataset. The cost of this granularity is that the calculation is relatively expensive. 

Explainable AI example

Figure 1 gives an example of three different approaches to understanding feature importance for a random forest model predicting reactor activity. On the left is the inbuilt impurity-based feature importance (MDI), in the centre is permutation-based feature importance and on the right are the Shapley values. The first two are fairly intuitive (bigger = more important, but not necessarily better or worse). 

The SHAP beeswarm plot shows a point for every prediction in the dataset. The position along the x axis shows how much impact (positive or negative) that feature had on the model prediction for that particular data point. The colour of the data point maps to the value of the feature. Looking at impurity A, we can see that there are a lot of red points to the right (high impurity A where there is high activity) and blue points on the left (low impurity A where there is low activity). This illustrates how, for any given model prediction, Shapley values can explain exactly why a model made that prediction.

Note that the feature importances do not precisely agree between methods; this is because they are asking different
questions. Consider the space velocity. The first two approaches concur that it is a moderately important variable. However, Shapley values reveal that for most data points the model predictions are not sensitive to space velocity, yet in some few cases it is the single most important feature. 

To understand the limitations of the analysis, consider impurity A. This is both a feed and a byproduct and so builds up in the reactor. All three methods show high importance and Shapley values show high presence of A is correlated with high reaction activity. But is this because the reaction is enhanced by A or because the reaction creates A (ie is A the chicken on the egg)? It is impossible to definitively say without further experimentation (see Figure 2). 

Finally, the feature importance metrics are all relative. In this example we included a feature consisting of randomly generated numbers to give a natural threshold and a feel for overfit; we know that the randomly generated feature contains no information, so any features less important than that can be safely ignored. We can also see from teh Shapley plot that the model is not using this feature for prediction. 

Reviewing these three techniques together highlights another potential pitfall; no given technique is a panacea and we need to be careful that the XAI technique used can give us information about the risk we are looking to manage. 

Explainable AI for other tasks

XAI techniques have been developed to handle a variety of different tasks, including unsupervised learning tasks on image data. A recent publication³ gives an example of how XAI was used to show how incidental features of an image were being used to detect faults in an industrial quality control model. We have encountered this exact issue in an industrial chemical engineering context.

The authors label this an example of the Clever Hans effect, a term in AI research which alludes to the early 20th
century horse who appeared to be able to perform arithmetic but was actually reading subconscious cues from his owner. The cautionary tale of Clever Hans is an excellent reminder of the risks which practitioners need to manage, especially when dabbling in deep learning. Consumers should feel empowered to demand this level of due diligence.

Hybrid first principles/ML modelling

Pure data-driven models must infer the behaviour of a system from their training data. In chemical engineering contexts, we often have the dual challenge of highly complex nonlinear behaviour (think reactions, mixing, phase segregation), which is inherently harder to learn, coupled with small datasets, limiting opportunity to learn. These are also systems where we, as a profession, have been building understanding for over a century. Hybrid modelling is a class of methods which combine both first-principles and machine learning in a single model. This gives us the chance to add our professional knowledge to models, better leveraging our data. 

Combining traditional chemical engineering theory with ML approaches

There are numerous approaches to incorporating domain knowledge into ML modelling. Figure 3 shows a simplified overview; redrawn from a more detailed treatment by Von Reuden et al.⁴

Engineering knowledge here consists of the laws we can write down. These can be encoded directly into the model structure, via a hybrid modelling approach. For example, a data-driven model can parameterise a first-principles component such as defining the rates of reaction in the differential equation. The widest adoption of these models is in bioprocessing, where there is intrinsic variability in process parameters which are a complex function of process conditions. We described the use of a mass transfer correlation as part of feature engineering in our second article (see TCE 1,013). However, there is also increased adoption in the field of chemical catalysis. With the University of Manchester, we have applied this approach to methanol synthesis kinetics.⁵ The hybrid model approach led to a simpler physical expression and significantly reduced prediction uncertainty compared to the benchmark literature expression.

Alternately, engineering knowledge can be embedded in a physics-informed model. This approach applies standard model selection and training, with an additional loss-function term penalising deviation from a first-principles model. As a simple example, one can penalise any model prediction which deviates from mass balance.

Engineering knowledge can also be used to transform the training data. This is perhaps conceptually the easiest to grasp. Consider a case where we wish to predict the pressure drop through a number of pipe geometrics at varying flow rates and fluid compositions. A hybrid approach might start by introducing known terms or parameters from domain expertise during feature engineering and using those as data-driven model inputs. In one case we used the Drucker-Prager/Cap model to transform a pellet compaction dataset onto a consistent first-principles basis, which then enabled a much simplified data-driven component to achieve good predictive performance with only two input variables.⁶

Benefits and challenges of hybrid modelling

The big benefit of hybrid modelling compared to a pure data-driven approach is a massive reduction in data required to achieve similar results. In our work, we have seen reductions by 80–90%, ie reductions to 10–20% of the data requirement. In principle, a hybrid model will be more robust, both when interpolating data and when extrapolating beyond the training data, which is otherwise fraught.

Compared to a first-principles model, the benefit is that modellers can remove the parts which have less confidence and allow them to be data driven. The combined catalyst deactivation and reactor model in article 1 (see TCE 1,012) exemplifies this. In this case, the physical component of a hybrid model may be simpler. It also improves the behaviour of models with poorly identified parameters, reducing uncertainty on prediction. The obvious advantage of this is that for many models, we don’t have a good estimate of the parameters we need and often the parameters are hard to identify by regression. The (philosophical) disadvantage is that to some extent we may be sacrificing the physical meaning of the parameters in the first-principles component.

typically relatively rapid. This makes them useful for optimisation and even control applications. A recent example shows an example model which can optimise the feed of alumina into an electrochemical cell in seconds.⁷ Hybrid models and physics-informed models remain more complex to build, both due to the need to understand two different modelling approaches and the practical challenges of implementation. Both are improving  with time, as awareness of data-driven modelling rises among engineers and as frameworks⁸ and techniques mature. Nevertheless, there is a skills gap in this space and there is no widely used framework or software for hybrid modelling.

Finally, our experience of hybrid models in particular that it is essential to ensure the physical component is as correct as possible. This physical component imposes a large bias on the model performance and it takes a lot of data to overcome this, potentially making the hybrid approach less efficient than a pure data-driven approach.

Conclusions: machine learning crimes

This series has attempted to highlight good practice and potential pitfalls of data-driven modelling techniques to chemical engineers, particularly those not familiar with them. The key lesson is that one does not need to be an expert in these methods to ask about the underlying assumptions and limitations. Do not be taken in by the hype. ML is still a modelling technique. All the things that could go wrong before can still go wrong, plus some exciting new ones. Don’t be seduced!

To conclude we would like to introduce the concept of “Machine Learning Crimes”, with a nod to the Inverse Mathematics community and its “Inverse Crimes”.^{9, 10} These are headline, potentially capital offences, all of which have serious consequences on any resulting model and its usefulness. The “ML Crimes” are intended as a memorable, ready reference list to help you spot red flags that are likely indicative of poor data processing, maths and modelling.

1. Poor data quality and sufficiency: disregarding the "Vs" of data, especially volume and variance
2. Poor data pre-processing: lack of clarity or questionable data cleaning, outlier detection
3. Poor feature selection/engineering: it is not adequately described and justified. There may be too many or too few variables
4. Lack of clarity or justification of the model performance metric: a model performance assessment must use “testing” data unseen in model fit or tuning
5. Limited model fit and performance evaluation: use of a single metric, for example R² , is not sufficient
6. Cherry-picking results: presenting only the best performing model or metric without context. This is especially problematic with small datasets
7. Limited assessment of the under-fit – over-fit balance: both under- and over-fit result in a poor predictive capability; a sub-optimal model
8. Selection of algorithm, the model or its structure without justification: ideally this should be via hyperparameter tuning. The justification does not have to be good, as long as it is honest
9. No assurance against data leakage: this is most likely to occur in feature selection and how the train–test data are split. How it has been guarded against merits mention. Remember, the self-delusion gap; if it’s too good to be true, it probably is
10. Poor use of domain knowledge: XAI and hybrid modelling are good – but not essential. Model build and feature selection should exploit the underlying chemical engineering. If not, ask why not?

If a paper or report shows signs of any of these, don’t read any further – consign it to the recycle bin, the “oubliette”, and move on.

Hugh Stitt is a senior research fellow at Johnson Matthey, where Robert Gallen is a principal engineer and Joe Emerson and Carl Jackson are senior digital chemical engineers

References

1. See, for example, Golden Gate Claude: https://www.anthropic.com/news/golden-gate-claude
2. LS Shapley: Notes on the N-Person Game – II: The Value of an N-Person Game; RAND Corporation: Santa Monica, CA, 1951. https://doi.org/10.7249/RM0670
3. J Kauffmann, J Dippel et al (2025): Explainable AI reveals Clever Hans effects in unsupervised learning models: bit.ly/nature-machine-intelligence-clever-hans
4. L Von Rueden, S Mayer S et al (2021): Informed Machine Learning – A Taxonomy and Survey of Integrating Prior Knowledge into Learning Systems: https://doi.org/10.1109/TKDE.2021.3079836
5. H Kay, F Vega-Ramon, RW Gallen, EH Stitt, D Zhang (2025) Developing a Hybrid Modeling Framework for Enhanced Prediction in Chemical Reaction Kinetics: bit.ly/industrial-and-engineering-enhanced-prediction
6. M Trower, JT Emerson et al: (2023) Reduced-order hybrid modelling for powder compaction: Predicting density and classifying diametrical hardness: https://bit.ly/powder-technology-reduced-order
7. K Patouillet, N Chailly et al (2025) In Light Metals, Edwards L (Ed); Springer Nature Switzerland; pp 579–589
8. For example, NVIDIA’s PhysicsNeMo framework https://developer.nvidia.com/physicsnemo
9. DL Colton, R Kress (1998) Inverse Acoustic and Electromagnetic Scattering Theory: Applied Mathematical Sciences; Vol. 93, Pub Springer, Berlin
10. J Kaipio, E Somersalo (2007): Statistical inverse problems: Discretization, model reduction and inverse crimes: Journal of Computational and Applied Mathematics: https://doi.org/10.1016/j.cam.2005.09.027