This study uses explainable machine learning methods to model and interpret the statistical correlations between rainfall and the discharge of urban catchments with sustainable urban drainage systems. The resulting models have good prediction accuracies. However, the right predictions may be made for the wrong reasons as the model cannot provide physically plausible explanations as to why a prediction is made.