Generative AI is making incredible strides and is changing fields such as medicine, education, finance, art, sports, etc. This advancement comes mainly from the improved ability of AI to learn from larger datasets and create more complex models with trillions of parameters. Although these advances have led to important scientific discoveries, created new business opportunities and led to industrial growth, they come at a high price, especially considering the financial and environmental impacts of training these large-scale models. Learning algorithms need significant computing power to train generative AI models with large data sets, resulting in high energy consumption and a significant carbon footprint.
While previous generative AI sustainability efforts have focused on improving hardware efficiency for AI training and developing smaller models with fewer parameters, Google DeepMind has taken an innovative approach that aims to improve the efficiency of the generative AI training algorithm. They pioneered the new JEST (Joint Example Selection) algorithm, which works 13 times faster and is ten times more energy efficient than current techniques.
In this article, we will explore the challenges of AI training and how JEST is addressing these challenges. In addition, we consider broader implications and future research directions for the JEST algorithm and envision its potential impact beyond increasing the speed, cost-effectiveness, and environmental friendliness of AI training.
AI Training Challenges: High Cost and Environmental Impact
Training generative AI models presents significant challenges due to high costs and significant environmental impact.
- Financial expenses: Training generative AI models is a costly endeavor. Recent estimates suggest that training a single large model such as OpenAI GPT-3 with 175 billion parameters can cost around $4.6 million. It is estimated that the training of ChatGPT-4 cost OpenAI approximately $100 million. These costs are largely attributed to the huge computing resources, extensive data processing, and long training time.
- Power consumption: Generative AI training processes are extremely energy intensive. Training these models involves thousands of GPUs and consumes several gigawatt hours of power, making the process extremely energy intensive. Data centers that house computing infrastructure for AI training use about 200 terawatt-hours (TWh) of electricity annually, roughly 1% of global electricity demand. A McKinsey report predicts that US data center energy consumption could increase from 17 gigawatts (GW) in 2017 to 35 GW by 2030, requiring the output equivalent of nine Hoover Dams to meet this additional demand.
- Carbon Footprint: The high energy consumption of training generative AI models contributes significantly to greenhouse gas emissions, exacerbating climate change. A study by the University of Massachusetts Amherst found that training a large artificial intelligence model can emit as much carbon dioxide as five cars over its lifetime. Specifically, training a single AI model can emit more than 626,000 pounds of CO2, equivalent to the carbon footprint of 315 trans-American flights.
These challenges primarily stem from two main sources: the reliance on high-power computing hardware and the inefficiency of current training algorithms. While the AI community has made progress in developing energy-efficient hardware, more emphasis needs to be placed on creating smarter algorithms that can optimize data usage and reduce training time. Google’s recently introduced JEST algorithm is pioneering research into making training algorithms smarter. By intelligently selecting the data that matters, JEST greatly increases the effectiveness of AI training, paving the way for more sustainable and cost-effective training of generative AI models.
Understanding the JEST algorithm
JEST is a learning algorithm designed to train multimodal generative artificial intelligence models more efficiently. To understand how JEST works, think of AI training as solving a complex puzzle, where each piece (data point) helps create a complete picture (AI model). JEST acts like an expert puzzle solver, making the process more efficient. Just as a puzzle solver selects the most important and salient pieces, JEST identifies and selects the most valuable data batches from a dataset, ensuring that each batch plays a key role in AI development.
JEST uses a smaller artificial intelligence model to evaluate the quality of data packets. These doses are then ranked according to their effectiveness in model training. With these carefully selected batches, JEST strategically assembles them to train the model. Much like a jigsaw puzzle arranges the pieces of a puzzle to maximize efficiency and coherence, JEST greatly accelerates the training process by prioritizing and selecting the most informative doses.
A key component of the JEST approach is multimodal contrastive learning. This technique focuses on learning correspondences between different data types such as text and images. JEST uses a multimodal contrastive learning-based method to evaluate the effectiveness of a multimodal data sample in model training. performance of individual data samples, JEST also evaluates the collective learnability of data samples to select a small batch of data from a larger “super batch”. This process helps JEST select and prioritize batches that offer challenges and rich learning opportunities.
Looking forward to: JEST Beyond Faster, cheaper and greener AI training
As we examine the future implications of Joint Example Selection (JEST), it is clear that its benefits go beyond simply speeding up AI training, reducing costs, and promoting environmental sustainability. Here we dive into how JEST can continue to improve and transform the field of generative artificial intelligence:
- Improved model performance and accuracy: JEST’s innovative approach to data selection and prioritization leads to faster training and better model performance. By focusing on the most informative data sets, JEST ensures that AI models are trained on high-quality inputs, improving their accuracy and robustness. This advantage is critical in applications where accuracy and reliability are paramount, such as medical diagnostics, financial forecasting and autonomous systems.
- Identifying and mitigating bias in data: AI is prone to biased datasets, where certain groups or perspectives are underrepresented or skewed. The JEST approach to data selection includes an assessment of the quality and informativeness of the data sets. By prioritizing diverse and representative data samples, JEST can help AI systems learn from a more balanced data set, thereby reducing bias in the training data. For example, in healthcare AI applications, JEST can select datasets including different demographic factors, ensuring that medical diagnostic models are trained on different patient populations. This selection reduces the risk of bias that could disproportionately affect certain groups based on race, gender, or socioeconomic status.
- Facilitating innovation and research: By significantly reducing the computational resources and time required to train AI models, JEST lowers the barriers to entry for researchers and innovators. This promotes accessibility and a more vibrant AI development ecosystem where smaller teams and organizations can experiment and deploy advanced AI solutions. Additionally, the efficiency gains offered by JEST free up resources that can be redirected to exploring new frontiers in AI, such as new architectures, advanced algorithms, and ethical AI frameworks.
- Supporting the inclusive development of artificial intelligence: AI development should include diverse perspectives and inputs to effectively mitigate biases and ethical concerns. JEST’s ability to select data based on their informative value and representativeness supports inclusive dataset management practices. AI developers can ensure that JEST effectively addresses biases and ethical considerations by involving multidisciplinary teams, including ethics, social science, and domain-specific experts, in defining data selection criteria. This collaborative approach promotes more inclusive and responsible development of AI technologies.
Bottom Line
DeepMind’s introduction of the JEST algorithm represents a major leap forward in generative AI training. By significantly speeding up training processes and reducing energy consumption, JEST offers significant cost savings and solves the environmental problems associated with AI development. In addition to these benefits, JEST has the potential to improve model accuracy, mitigate data bias, promote innovation, and promote inclusive AI development. JEST’s continued refinements and applications are poised to redefine the future of AI and move towards more efficient, sustainable and ethically responsible AI solutions.
