Predicting rains: Can the butterfly be caught?

Atanu Biswas
Professor, Indian Statistical Institute, Kolkata

One major reason for last year’s devastating floods in Kerala was excessive rainfall. In its report in May-end 2018, the India Meteorological Department (IMD) forecast normal monsoon across India in 2018, and predicted a rainfall of 95 per cent of the long period average (LPA) in the southern peninsula. However, considering a model error of +/- 4 per cent and the extremely high variability of rainfall in the southern states, it is still very difficult to explain a 141 per cent rainfall the state experienced from June to mid-August.

Inadequate rainfall also causes serious problems. The IMD predicted that the monsoon would ‘most likely’ be normal in 2017. According to various newspaper reports, a farmer of Beed district of Marathwada walked into the Dindrud police station in July 2017 to file an FIR against the IMD for ‘wrong monsoon forecast’.

Is the rainfall forecast so unreliable in the country? In 2015, the IMD predicted a drought, and there was 14 per cent shortfall of a 50-year average benchmark of rainfall. Skymet, its private counterpart, predicted a year of normal rainfall, although Skymet had consistently outperformed the IMD in predicting monsoons accurately for the previous three years. However, both the IMD and Skymet missed the 2009 drought, which was the worst India had experienced in four decades. The IMD predictions failed in 2002 and 2004 as well. Thus, a wrong prediction of extreme rainfall is not a rare phenomenon.

Correct forecast of monsoon might save thousands of lives, billions of rupees, and help policymakers take important decisions. However, it might not be fair to blame the forecasting organisations — monsoon forecasting being the classic example of an inexact science. The mutual associations of millions of variables, including wind, currents, precipitation, tides, humidity, and temperature come into play.

In fact, weather prediction was the poster child of the ‘Chaos Theory’, initiated by mathematician Edward Lorenz. The theory says some systems, highly sensitive to initial conditions, are simply too complex to be predictable over the long term. The ‘Butterfly Effect’, named after a lecture delivered by Lorenz in 1972, indicates that tiny changes might result in unpredictable effects — the flap of a butterfly's wings in Brazil could have far-reaching repercussions for future weather around the world, like a tornado in Texas several weeks later. It is impossible to track every butterfly, and, hence, it's impossible to precisely predict the future.

The sheer luck of geographical position turned India into some sort of a ‘monsoon bucket’; the summer appointment of the monsoon with the Indian subcontinent is the key to our lifeline, indeed. Agriculture is the backbone of our economy, and nearly 120 million people in the farming sector are dependent on monsoon rains.

The long-range forecasts are used for agriculture, transport and water management. In 2007, the IMD switched to the ensemble statistical model for forecasting from the century-old basic model. Until 2016, India’s monsoon forecasting system was driven entirely by correlating monsoon rains with six meteorological values related to the large-scale global phenomena. For example, historical data shows that a strong El Niño or unusual warming of the Pacific brings on a weak monsoon. Both linear and non-linear regression models were attempted for a large number of models for all possible combinations of predictors, and a few best models were selected. The forecast was the weighted average of the outcomes of these models.

Recently, India moved towards dynamical prediction — the ‘Dynamical Monsoon Model, also known as the Coupled Forecast System — which essentially treats the globe as a fluid, and simulates the state of the atmosphere and oceans using standard laws of physics. The model collates data on local as well as global weather, and solves the equations for a small duration of time at a time, and then repeats, and thus the errors would be accumulated.

India procured a Rs 400-crore supercomputer named Pratyush in 2016 for dynamical forecast; the technology being similar to models currently used in the US. However, the dynamical model has achieved 60 per cent accuracy, and the IMD aims to take it up to 77 per cent. The long-range forecast capacity of this model being weak, it might not be enough to forecast the Kerala floods with confidence!

Loads of data on several atmospheric factors are available using aircraft or weather balloons, radars, observatories, satellites, weather office, and historical data. The quantity of weather data IMD uses into the models has grown up to 70.5 GB/day from about five GB/day about a decade ago. Even data on the high-rises and deforestation of the country should come into play. Can we now expect more accurate forecasts for the extreme weather events such as the Mumbai floods of 2005 or the Kedarnath deluge of 2013?

The dynamical monsoon model could certainly be used in combination with the traditional ensemble model for possible better forecast. Theoretical studies combining the two approaches are being carried out by researchers. We now have Big Data, in the true sense of the term, and one might aspire to churn that ocean of data to get the nectar of knowledge for forecasting monsoon, perhaps the most important, yet daunting, test for the Big Data experts.

Standard Big Data Analytics is still not capable of doing that, for sure. However, who knows, a collaboration of top-level statisticians, computer experts, and meteorologists might work wonders to identify some other possible important variables or indices for predicting the monsoon. Different types of advanced time series models and adaptive procedures along with the analyses of extreme values might be attempted to build up a much better relationship of rainfall on the regressors. Remember that data was never so big, and technology was never so friendly. Even if some useful ‘butterflies’ can be caught by digging the Big Data, that could be a milestone for our economic development and social welfare.