Forecasting COVID-19 poses unique challenges due to the novelty of the disease, its unknown characteristics, and substantial but varying interventions to reduce its spread. To improve the quality and robustness of forecasts, we propose a new method which aims to disentangle region-specific factors – such as demographics, enacted policies, and mobility – from disease-inherent factors that influence its spread. For this purpose, we combine recurrent neural networks with a vector autoregressive model and train the joint model with a specific regularization scheme that increases the coupling between regions. This approach is akin to using Granger causality as a relational inductive bias and allows us to train high-resolution models by borrowing statistical strength across regions. In our experiments, we observe that our method achieves strong performance in predicting the spread of COVID-19 when compared to state-of-the-art forecasts.