The Name of the Game: Efficiency

Developing a more efficient AI crucial for long-term sustainability

With artificial intelligence now taking centre stage in almost every facet of future development, an aspect that has especially been called into question is its ethics. For AI implementation to be truly sustainable over the long run, several issues pertaining to ethics – such as privacy laws and environmental ramifications – need to be ironed out. This is especially relevant in light of recent research findings coming in from several sources over the world.

The findings are alarming, to say the least. In a study conducted in 2019 by researchers at the University of Massachusetts Amherst, training a large deep-learning model was estimated to produce over 625,000 pounds of carbon dioxide – a greenhouse gas infamous for trapping atmospheric heat and causing depletion of the ozone layer – thereby aggravating global warming substantially.

This volume is approximately equivalent to the total carbon dioxide emissions produced by a human being in almost 60 years; using 5 cars over the lifetime or in over 300 round-trips between New York and San Francisco by air. And let’s stress on the fact that this is the environmental cost of training only ONE deep learning model! Researchers believe that this is primarily owing to the fact that hardware developments have not been able to keep pace with the demand for bigger – and faster – computing capabilities.

Developmental drawbacks

Recent progress in hardware and in the methodologies for training neural networks has ushered in a new generation of large networks trained on abundant data. The level of gains observed across several Natural Language Programming (NLP) tasks have been quite notable as well. However, these improvements are dependent on developing sufficiently powerful hardware resources – which, in turn, require substantial energy consumption.

Although there have been several developments in terms of processing units – such as the use of photonic tensors in TPUs (tensor processing units) – the speed of development of hardware is still far outpaced by that of neural network training. This essentially calls for improving efficiency – and rather drastically too, especially if deep nets and the hardware they function on are to survive for the future.

For example, while trying to track AI’s carbon footprint, energy consumption of the models are often estimated roughly, as there is no set benchmark for energy consumption that needs to be reported in any ongoing research. While research and development must go on, optimising the negative impacts of energy consumption also needs to be given due importance.

Scientists, therefore, advise on not only reporting training time for new models, but also the different computational resources required, along with the levels of sensitivity attached to each of the parameters. This will help in direct comparison across different models – and consequently allow for greater flexibility to consumers. Consumers would be able to directly assess the computational settings available and judge whether the model is working efficiently at given settings.

Cherry-picking the way to efficiency

The call to prioritise computationally efficient hardware and algorithms is loud and clear. This will not only require developing more energy-prudent hardware, but also NLP packages prioritising model efficiency. This can be carried out by fine-tuning application interfaces (APIs) to prioritise efficiency, instead of brute-force grid searching. In this connection, a research paper presented by the MIT-IBM Watson AI Lab at the European Conference on Computer Vision in August 2019, offers an interesting viewpoint.

The paper stresses the need for ‘cherry-picking’ relevant data while skimming through the less-relevant ones. Imagine, a video clip of a person making a meat sandwich. In the video, important tasks such as slicing the meat or stacking it on the bread are given more importance and displayed at a higher resolution, while less-relevant frames are either skipped or displayed at a lower resolution to save on memory. This kind of an approach towards video classification can cut computational cost in half compared to the next best model – and may just be the way ahead.

Achieving a greater level of efficiency in algorithms and hardware will not only reduce environmental impacts of AI, but also markedly improve accessibility and data protection, thus cutting down on costs and processing time. Without doubt, a more efficient AI will strike the right balance between progress and sustainability.