Paper Reading Group: Attention Is All You Need

Generative AI, specifically large language models, have taken the world by storm. But they wouldn't exist today without a key paper published by Google Brain: "[Attention Is All You Need](https://arxiv.org/pdf/1706.03762.pdf)." The 10-page paper, published in 2017, introduced the Transformer - a critical innovation in neural networks.

**What's in store?**

- A deep dive and interactive discussion of the paper. We recommend reading (or at least skimming) it first: <https://arxiv.org/pdf/1706.03762.pdf>
- A look at what came before the paper (neural networks, embeddings), and what came after (BERT, GPT-2, and ChatGPT).
- **All software engineers welcome** - no AI/ML background necessary!

**Did you know?**

- The paper's authors have all gone on to found multiple billion-dollar AI companies including Cohere and [Character.ai](http://Character.ai).
- Since 2017, the paper has been cited over 86,000 times.
- The Transformer architecture is the **T** in "GPT".

**Your host:**

[Charlie Guo](https://www.linkedin.com/in/charlierguo/) is a seasoned founder and author with over 15 years of software experience and more than a decade in Silicon Valley. He is an alum of both Stanford and YCombinator, and currently publishes [Artificial Ignorance](https://www.ignorance.ai/about), an AI-focused newsletter for engineers and founders.

**The agenda:**

- Charlie will give a brief background, and then explain the key insights and context from the paper.
- We will open the group to some Q&A and a broader discussion.
- And we'll end with some discussion on which paper to read and review next.

The next paper reading group is on October 15 about [deep reinforcement learning](https://www.jointaro.com/event/paper-reading-group-human-level-control-through-deep-reinforcement-learning-deep-q-network/), led by Sai Bhavanasi.

Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases

Session led by [Dharin Shah](https://www.linkedin.com/in/dharinshah/)

In this session, we delve into the intricacies of Amazon Aurora, a high throughput, cloud-native relational database designed to meet the demanding scalability and performance requirements of modern applications without compromising on the simplicity of access via SQL. The paper presents a thorough examination of the design considerations that underpin Aurora's architecture, understanding how it achieves unparalleled levels of efficiency and reliability within the cloud. We will explore key concepts such as Aurora's distributed and decentralized approach, fault tolerance mechanisms, and the optimization of data storage and retrieval processes.

Demystifying LLM Benchmarks -- How To Compare Large Language Models?

In the recent launch of Google's Large Language Model (LLM) Gemini, Google claimed that "Gemini beats human level performance on MMLU" but you might be wondering:

- What are the different evaulation criteria? Massive Multitask Language Understanding (**MMLU**)? General Language Understanding Evaluation (**GLUE**)? **HumanEval**?
- How can we know which model is good at coding tasks?
- What is considered "human-level performance"?
- How do they actually get the results for these benchmarks?

In this talk, Sai breaks down all the benchmarks you need to know to be better informed when papers claim to beat benchmarks.

The next paper reading session is on January 8 at 5pm: [**https://www.jointaro.com/event/paper-reading-group-4-mistral-7b/**](https://www.jointaro.com/event/paper-reading-group-4-mistral-7b/)

**Your host:**

[**Sai Shreyas Bhavanasi**](https://www.linkedin.com/in/b-sai/) has worked in Computer Vision and Reinforcement Learning and published 2 first-author papers. He has also worked as an MLE and Data Analyst.

Taro Book Club: The Fundamentals of Data Engineering - Chapter 3 (Part 2)

Session led by [Gideon Blinick](https://www.linkedin.com/in/gideonblinick/)

Chapter 3 - Part 2: Designing Good Data Architecture

In our first session on Chapter 3, we learned the 9 principles of Good Data Architecture, which are largely based on Google's Principles of Cloud-Native Architecture and Amazon's Well-Architectured Framework.

Now we get less abstract and cover Major Architecture Concepts and Examples, including Domains and Services, Distributed Systems, Data Warehouses and Lakes, the Modern Data Stack, and so much more!

Paper Reading - Gemini: A Family of Highly Capable Multimodal Models

Paper Discussion led by: [Akshika Wijesundara PhD](https://www.linkedin.com/in/akshika/)

Abstract: Google researchers recently unveiled Gemini, a ground breaking family of multimodal AI models capable of understanding and reasoning across images, audio, video, and text. The largest Gemini model, Ultra, achieved human-expert performance on several benchmarks and advanced the state of the art across a range of multimodal tasks.

In this paper reading, we will discuss key details from Google's paper introducing Gemini models. Topics will include:

- The Gemini model architectures and how they achieve cross-modal understanding
- Performance benchmarks and state-of-the-art results across text, image, audio, and video tasks
- Applications and use cases enabled by Gemini's multimodal reasoning
- Approaches to responsible deployment of large language models like Gemini

This paper provides a comprehensive overview of the exciting new capabilities of Google's Gemini project. Whether you're an AI researcher, engineer, or simply curious about the state of the art in multimodal AI, join us for an in-depth look at this new model family.\
\
Paper link: [**https://arxiv.org/abs/2312.11805**](https://arxiv.org/abs/2312.11805)\
Event link:  **[https://www.jointaro.com/event/paper-reading-gemini-a-family-of-highly-capable-multimodal-models/](https://www.jointaro.com/event/paper-reading-gemini-a-family-of-highly-capable-multimodal-models/)**

Paper Reading Session: Simple Testing Can Prevent Most Critical Failures

In this session, we will be discussing this paper titled "[**Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems**](https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf)".

Large, production quality distributed systems still fail periodically, and do so sometimes catastrophically, where most or all users experience an outage or data loss. This paper presents the result of a comprehensive study investigating 198 randomly selected, user-reported failures that occurred on Cassandra, HBase, Hadoop Distributed File System (HDFS), Hadoop MapReduce, and Redis, with the goal of understanding how one or multiple faults eventually evolve into a user-visible failure and how simple testing can prevent majority of them.

Presented by [Sudhi Awasthi](https://www.jointaro.com/p/sudhi-awasthi/), an engineer working at Bloomberg. Connect with her on LinkedIn: <https://www.linkedin.com/in/sudhiawasthi/>.

[Paper Reading] Netflix : Query Facet Mapping and its Applications in Streaming Services

Led by [Dharin Shah](https://www.linkedin.com/in/dharinshah/)

For a Taro Premium discount from Dharin, use his referral link here: <https://www.jointaro.com/r/dharins694/>

This deep-dive goes through how Netflix built their internal query to facet mapping layer to have better understand intent of the query. For example - matching 'anim' to 'Demon Slayer' as higher rank than perhaps 'animal planet'. In this session we dive into the white paper released by Netflix around their implementation.

Link to Paper: [**https://dl.acm.org/doi/pdf/10.1145/3477495.3536330**](https://dl.acm.org/doi/pdf/10.1145/3477495.3536330)

Paper Reading: MyRocks: LSM-Tree Database Storage Engine Serving Facebook's Social Graph

Session led by [Dharin Shah](https://www.linkedin.com/in/dharinshah/)

**Event description**

Facebook uses MySQL to manage tens of petabytes of data in its main database named the User Database (UDB). UDB serves social activities such as likes, comments, and shares. In the past, Facebook used InnoDB, a B+Tree based storage engine as the backend. The challenge was to find an index structure using less space and write amplification.

LSM-tree has the potential to greatly improve these two bottlenecks. RocksDB, an LSM tree-based key/value store was already widely used in variety of applications but had a very low-level key-value interface. To overcome these limitations, MyRocks, a new MySQL storage engine, was built on top of RocksDB by adding relational capabilities. With MyRocks, using the RocksDB API, significant efficiency gains were achieved while still benefiting from all the MySQL features and tools. The transition was mostly transparent to client applications.

The paper was published at the **International Conference on Very Large Databases (VLDB)**: [**https://research.facebook.com/publications/myrocks-lsm-tree-database-storage-engine-serving-facebooks-social-graph/**](https://research.facebook.com/publications/myrocks-lsm-tree-database-storage-engine-serving-facebooks-social-graph/).

Paper authors:

- [**Yoshinori Matsunobu**](https://www.linkedin.com/in/yoshinorimatsunobu/)
- [**Siying Dong**](https://www.linkedin.com/in/siyingdong/)
- Herman Lee

Paper Reading: Self-supervised Vision Transformers (Foundational Model)

Session led by [Mandar Deshpande](https://www.linkedin.com/in/mandardeshpande1995/)

Dino is a self-supervised learning approach for computer vision tasks. It leverages vision transformers without using any labeled data. Dino v2 introduces several improvements over its predecessor, enhancing performance and scalability. It achieves state-of-the-art results on various benchmarks. Notably, Dino v2 demonstrates strong performance on downstream tasks such as image classification and object detection.

We will be walking through

- Basics of computer vision
- Self supervised learning
- ViT (vision transformers)
- DINO
- DINO v2 model

Paper link - [**https://arxiv.org/abs/2304.07193**](https://arxiv.org/abs/2304.07193)

[Event link](https://www.jointaro.com/event/paper-reading-dino-v2-ie-foundational-model-for-computer-vision/)

Paper Reading: Beyond Next-Token Prediction: Enhancing Language Models with Multi-Token Outputs

Session led by: [Lucia Mocz](https://www.linkedin.com/in/lucia-mocz-ph-d-35a98a1b6/)

In this talk, we will look at groundbreaking research on advancing the training of large language models. Traditional models like GPT and Llama rely on next-token prediction, which can be expensive and time-consuming, but this research suggests a more efficient approach: multi-token prediction.

**Key Highlights:**

- **Enhanced Sample Efficiency**: Training models to predict multiple future tokens at once increases sample efficiency, leading to significant improvements in downstream capabilities.
- **Innovative Architecture**: By employing n independent output heads on a shared model trunk, better performance with no additional training time can be achieved.
- **Superior Performance**: 13B parameter models outperform strong baselines on generative benchmarks, solving 12% more problems on HumanEval and 17% more on MBPP.
- **Algorithmic Reasoning**: Multi-token prediction fosters the development of induction heads and enhances algorithmic reasoning capabilities.
- **Faster Inference**: Models trained with 4-token prediction are up to 3 times faster during inference, even with large batch sizes.

Join us to explore how multi-token prediction can revolutionize language model training and performance, especially for coding and natural language tasks.

Don't miss this opportunity to learn about the future of language model training and its practical applications!\
\
Paper: [**https://arxiv.org/pdf/2404.19737**](https://arxiv.org/pdf/2404.19737)

Paper Reading: Mamba - Revolutionizing Sequence Modeling

Session led by: [Akshika Wijesundara, PhD](https://www.linkedin.com/in/akshika/overlay/about-this-profile/)

*Abstract:*

Explore the Mamba paper's groundbreaking approach to sequence modeling efficiency in this event. Uncover how Mamba addresses computational inefficiencies in transformers, especially for long sequences.

*Overview:*

Discover the motivation behind Mamba, highlighting its improvements over subquadratic-time architectures struggling with language tasks. Learn about Mamba's content-based reasoning and enhancements for superior performance.

*Key Features:*

1\. **Selective Structured State Space Models (SSMs):** Mamba introduces adaptive SSM parameters for content-based reasoning, allowing selective information handling for discrete modalities.

2\. **Hardware-Aware Parallel Algorithm:** Despite forgoing efficient convolutions, Mamba's hardware-aware parallel algorithm maintains fast inference and linear scaling in sequence length.

3\. **Architecture Overview:** Dive into Mamba's streamlined architecture without attention or MLP blocks, achieving efficiency without sacrificing effectiveness.

4\. **Performance Metrics:** Witness Mamba's 5× higher throughput than Transformers, with linear scaling, showcasing its real-world improvements in language, audio, and genomics tasks.

5\. **Comparison with Transformers:** See Mamba outperforms Transformers of the same size and match larger Transformers, especially in language modeling.

Join us in unraveling the secrets of Mamba, poised to redefine sequence modeling standards. Anticipate a paradigm shift in efficiency and performance in deep learning.\
\
paper link: \[[**https://arxiv.org/abs/2312.00752**](https://arxiv.org/abs/2312.00752)\]

Paper Reading: Self-Discover - Empowering Large Language Models with Autonomous Reasoning Structures

In this talk, we delve into Self-Discover, an innovative framework revolutionizing Large Language Models (LLMs) by enabling them to autonomously discern task-specific reasoning structures, thereby empowering them to tackle complex problems without explicit prompting. 

At the heart of Self-Discover lies a self-discovery process, wherein LLMs independently select and amalgamate various atomic reasoning modules, such as critical thinking and step-by-step reasoning, into explicit structures to guide their decoding process. The broad applicability of these self-discovered structures across diverse model families—from PaLM2-L to GPT-4 and from GPT-4 to Llama2—underscores the framework's versatility and adaptability, suggesting seamless integration with an even wider range of advanced AI models. This holds promise for a unified approach to autonomous reasoning in artificial intelligence.\
\
Whether you're an AI researcher, engineer, or simply curious about this achievement in AI research, join us as we explore the intricacies of Self-Discover and its potential implications for prompt engineering and advancing the capabilities of Large Language Models in complex reasoning tasks.\
\
Paper: [**https://arxiv.org/pdf/2402.03620.pdf**](https://arxiv.org/pdf/2402.03620.pdf)

Lucia on LinkedIn: <https://www.linkedin.com/in/lucia-mocz-ph-d-35a98a1b6/>

Taro Book Club: The Pragmatic Programmer - Session 4

Session led by: [Bikram Kashyap](https://www.linkedin.com/in/kashyapbikram/)

In this #taro-bookclub discussion, we'll further go over [**"The Pragmatic Programmer"**](https://www.amazon.com/Pragmatic-Programmer-Journeyman-Master/dp/020161622X) by Andrew Hund & David Thomas.

Here's the agenda:

- Power of Plain Text
- Shell Games
- Power Editing
- Version Control
- Debugging
- Text Manipulation
- Engineering Daybooks

Please send your questions ahead of time to [**Bikram Kashyap**](https://www.linkedin.com/in/kashyapbikram/) in the [**Taro Premium**](https://www.jointaro.com/membership/) Slack, so that they can be included in the discussion. [***If you don't have the book***](https://taropremium.slack.com/archives/C05CKBHPNTU/p1723491587867879) *or want more details on how this all works, check out the #taro-bookclub channel in the Taro Premium Slack as well!*

Get ready for another engaging and informative session!

Paper Discussion led by [Sudhi Awasthi](https://www.linkedin.com/in/sudhiawasthi/)

In this session, we will be discussing this paper titled "[Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems](https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf)".

Large, production quality distributed systems still fail periodically, and do so sometimes catastrophically, where most or all users experience an outage or data loss. This paper presents the result of a comprehensive study investigating 198 randomly selected, user-reported failures that occurred on Cassandra, HBase, Hadoop Distributed File System (HDFS), Hadoop MapReduce, and Redis, with the goal of understanding how one or multiple faults eventually evolve into a user-visible failure and how simple testing can prevent majority of them.\
\
Event Link: <https://www.jointaro.com/event/paper-reading-session-simple-testing-can-prevent-most-critical-failures/>

Paper Reading: AlphaGeometry: Advancing Automated Reasoning in Olympiad-Level Geometry

Paper Discussion led by [Lucia Mocz](https://www.linkedin.com/in/lucia-mocz-ph-d-35a98a1b6/)

Solving mathematical problems at the Olympiad level is incredibly difficult, requiring advanced problem-solving skills, a deep understanding of mathematical principles, and the ability to think critically and creatively under pressure. Those who excel in Olympiad-level problem solving usually have exceptional talent, undergo extensive preparation, and possess a unique problem-solving mindset. Achieving human-level automated reasoning in mathematical theorem proving, especially when competing with top performers, is a significant milestone in artificial intelligence research.

In this paper reading session, we introduce AlphaGeometry, a groundbreaking theorem prover tailored for Euclidean plane geometry. Unlike traditional methods that rely on human demonstrations, AlphaGeometry autonomously synthesizes millions of theorems and proofs across various complexities without human intervention. By combining a neural language model trained on vast synthetic data with a symbolic deduction engine, AlphaGeometry employs a neuro-symbolic approach to navigate intricate problem spaces. Notably, AlphaGeometry generates human-readable proofs and has successfully tackled challenging problems from previous International Mathematical Olympiads, even discovering new generalizations of established theorems.

Whether you're an AI researcher, engineer, or simply curious about this achievement in AI research, join us as we explore the intricacies of AlphaGeometry and its potential implications for automated reasoning in mathematics.\
\
Paper: [**https://www.nature.com/articles/s41586-023-06747-5**](https://www.nature.com/articles/s41586-023-06747-5)\
Event Link: <https://www.jointaro.com/event/paper-reading-alphageometry-advancing-automated-reasoning-in-olympiad-level-geometry/>

Blog: [**https://deepmind.google/discover/blog/alphageometry-an-olympiad-level-ai-system-for-geometry/**](https://deepmind.google/discover/blog/alphageometry-an-olympiad-level-ai-system-for-geometry/)

GitHub: [**https://github.com/google-deepmind/alphageometry**](https://github.com/google-deepmind/alphageometry)

Paper Reading: Refactoring vs Refuctoring: Advancing the state of AI-automated code improvements

Paper Discussion led by [Adi Pratheek](https://www.linkedin.com/in/pratheekadi/)

"**Refactoring vs Refuctoring**" is the conclusion of a benchmark study of the most popular Large Language Models (LLMs) and their ability to generate code for refactoring tasks. This paper aims to illustrate the current standards and limitations, and seek to show new methodologies with higher confidence results.

**What's in store?**

In this paper reading session, we will be examining:

- The performance of the most popular Large Language Models (LLMs) on code refactoring
- How does an AI err?
- A novel innovation by CodeScene for fact-checking the AI-output and augmenting the proposed refactorings by measuring the confidence levels

Companies build pretty substantial codebases over a period of time (sometimes decades). As and when the companies grow, a significant amount of time and energy are constantly invested by development teams on code refactoring (also covered in this paper).

But, how well do the existing AI tools/models actually perform this task? Are we there yet to confidently say "**AI will replace software developers**"? Join this session where we dive deep into the paper to find more answers.

Link to the paper: [**Refactoring vs Refuctoring: Advancing the state of AI automated code improvements**](https://codescene.com/hubfs/whitepapers/Refactoring-vs-Refuctoring-Advancing-the-state-of-AI-automated-code-improvements.pdf)\
Event Link: <https://www.jointaro.com/event/paper-reading-refactoring-vs-refuctoring-advancing-the-state-of-ai-automated-code-improvements/>

Paper Reading: AutoDev: Automated AI-Driven Development

Are you interested in tools like the recently announced Devin?\
\
Join us for an enlightening session as we delve into the cutting-edge world of agentic AI-driven software development, featuring a comprehensive exploration of the AutoDev framework, Microsoft's agentic AI-driven software development framework.

This reading event will elucidate how AutoDev will revolutionize the software development process. Through enabling autonomous planning and the execution of complex engineering tasks, AI agents such as AutoDev will be fully integrated into common tools and technologies such as VSCode and GitHub. Discover how these agents can perform multiple operations on a codebase, from file editing and testing to git operations, all within a secure Docker-based environment.

Moreover, we will examine AutoDev's impressive performance on the HumanEval dataset, achieving remarkable scores of 91.5% and 87.8% for Pass@1 in code generation and test generation tasks, respectively. We will also discuss the implications of this research for the future of AI-assisted software development.

This event is a must-attend for software developers, researchers, and tech enthusiasts eager to understand the future of agentic AI-driven software development and its potential impact on the software development landscape. Whether you're interested in improving your development workflow, exploring AI applications in software engineering, or simply curious about the latest advancements in AI-driven development tools, this paper reading offers valuable insights and discussions on the forefront of technology innovation.

Mark your calendars and join us for an exciting hour of "AutoDev: Automated AI-Driven Development" exploration!

[Paper Reading] ROSE: Robust Caches for Amazon Product Search

Session led by: [Dharin Shah](https://www.linkedin.com/in/dharinshah/)

ROSE introduces a robust caching system that tolerates misspellings and typos while maintaining the efficiency of traditional caches. The core innovation is a randomized hashing schema that allows ROSE to index and retrieve a vast set of queries with constant memory and time. This system is robust against query intent, typos, and grammatical errors, offering theoretical guarantees of its performance.

Extensive experiments on real-world datasets highlight ROSE's effectiveness and efficiency, showcasing significant improvements in Amazon's search engine performance across key business metrics.\
\
In this session we deep dive into the query caching layer at Amazon that allows for far better and robust search performance

[**https://drive.google.com/file/d/13YwkeCc1XwnRVDVkcuE7MKt2r7aqld24/view**](https://drive.google.com/file/d/13YwkeCc1XwnRVDVkcuE7MKt2r7aqld24/view)

[Paper Reading] Deploy ML Models from Lab to Production

Session led by: [Akshika Wijesundara PhD](https://www.linkedin.com/in/akshika/)

If you don't have a Taro Premium membership, you can get it with a discount with Akshika's special link here: [**https://www.jointaro.com/r/akshikw802/**](https://www.jointaro.com/r/akshikw802/)

**WHAT YOU'LL LEARN**

- **Discover the benefits of CI/CD pipelines:** Learn how CI/CD pipelines streamline the model building, testing, and deployment processes, enabling rapid iteration and continuous improvement.
- **Containerize your ML code:** Understand the steps and advantages of containerizing your machine learning code to ensure consistency, portability, and efficient resource management.
- **Leverage the AWS stack for ML workloads:** Explore how to effectively utilize AWS as your cloud infrastructure provider to run and scale your machine learning workloads with ease and reliability.

**WHY THIS TOPIC MATTERS**

MLOps isn't just about fancy models - it's about getting them working for real! Learn how to deploy, maintain, and collaborate on ML projects, making you a highly-demand expert who can turn machine learning into real-world impact.

Paper Reading: ClickHouse - Lightning Fast Analytics for Everyone

Session led by [Dharin Shah](https://www.linkedin.com/in/dharinshah/)

Link to Paper: [**https://www.vldb.org/pvldb/vol17/p3731-schulze.pdf**](https://www.vldb.org/pvldb/vol17/p3731-schulze.pdf)

This paper presents an overview of ClickHouse, a popular opensource OLAP database designed for high-performance analytics over petabyte-scale data sets with high ingestion rates. Its storage layer combines a data format based on traditional log-structured merge (LSM) trees with novel techniques for continuous transformation (e.g. aggregation, archiving) of historical data in the background. Queries are written in a convenient SQL dialect and processed by a state-of-the-art vectorized query execution engine with optional code compilation. ClickHouse makes aggressive use of pruning techniques to avoid evaluating irrelevant data in queries. Other data management systems can be integrated at the table function, table engine, or database engine level. Real-world benchmarks demonstrate that ClickHouse is amongst the fastest analytical databases on the market.\
\
We will go into the details of why and how Clickhouse is one of the fastest OLAP database, its underlying storage engine, and the overall philosophy for performance.

Paper Reading: Enhancing Reasoning Abilities in LLMs through Geometrical Understanding

Session Led by [Lucia Mocz](https://www.linkedin.com/in/lucia-mocz-ph-d/)

The advancement of large language models (LLMs) for real-world applications hinges critically on enhancing their reasoning capabilities. In this paper read, we explore the reasoning abilities of LLMs through a geometrical framework. We establish a connection between the expressive power of LLMs and the density of their self-attention graphs.

The analysis performed in this work demonstrates that the density of these self-attention graphs defines the intrinsic dimension of the inputs to the MLP blocks. In particular, it shows, through theoretical analysis and toy examples, that a higher intrinsic dimension implies a greater expressive capacity of the LLM. Additionally, it provides empirical evidence linking this geometric framework to recent advancements in methods aimed at enhancing the reasoning capabilities of LLMs.

Don't miss this opportunity to deepen your understanding of how geometrical insights can drive improvements in AI reasoning. Whether you're an AI researcher, practitioner, or enthusiast, this talk promises to offer valuable perspectives and groundbreaking ideas.\
\
Paper: [**https://arxiv.org/abs/2407.02678**\
(Accompanying](https://arxiv.org/abs/2407.02678￼\(Accompanying) paper: [**https://arxiv.org/abs/2312.01648**](https://arxiv.org/abs/2312.01648))

Taro Book Club: The Pragmatic Programmer - Session 5

Session Led by [Bikram Kashyap](https://www.linkedin.com/in/kashyapbikram/)

In this #taro-bookclub discussion, we'll go over [**"The Pragmatic Programmer"**](https://www.amazon.com/Pragmatic-Programmer-Journeyman-Master/dp/020161622X) by Andrew Hund & David Thomas

Here's the agenda:

- Decoupling
- Juggling the Real World
- Transforming Programming
- Inheritance Tax
- Configuration

Please send your questions ahead of time to [**Bikram Kashyap**](https://www.linkedin.com/in/kashyapbikram/) in the [**Taro Premium**](https://www.jointaro.com/membership/) Slack, so that they can be included in the discussion. *If you don't have the book or want more details on how this all works, check out the #taro-bookclub channel in the Taro Premium Slack as well!*

Get ready for another engaging and informative session!

Paper Reading: The Future of Large Language Model Pre-training is Federated

Session led by [Mike A](https://www.linkedin.com/in/malik-mike-a-32b63818b/)

Is the future of model pre-training decentralized?

Join us for a captivating session as we explore the cutting-edge breakthrough of federated learning in language model pre-training, featuring an in-depth discussion of the "The Future of Large Language Model Pre-training is Federated" paper!

This paper reading event will shed light on how federated learning (FL) could be set to revolutionize the training of large language models (LLMs). Discover how FL could mobilize vast, underutilized data and computational resources from across the globe, enabling large-scale collaboration and democratizing the pre-training of LLMs. Learn about the innovative approach that allows organizations, and hopefully, soon individuals, to collectively train LLMs using limited resources while matching or even surpassing the performance of traditional centralized models!

This event is a must-attend for AI researchers, software developers, data scientists, and tech enthusiasts eager to grasp the future of LLM pre-training and its implications for AI advancements. Whether you're interested in scaling AI models, leveraging federated learning for collaborative training, or simply curious about the next big thing in AI technology, this paper reading offers invaluable insights and discussions on some of the latest AI research.

Mark your calendars and join us for an exciting hour of "Federated Learning: The Future of Large Language Model Pre-training" exploration!\
\
**Reference Links:**\
[https://flower.ai/\
https://github.com/adap/flower](https://flower.ai/￼https://github.com/adap/flower)

Paper Reading: Llama 2: Open Foundation and Fine-Tuned Chat Models

Session led by [Mike A](https://www.linkedin.com/in/malik-mike-a-32b63818b/)

Join us for the second installment of our three-part paper reading series on the LLaMA series of Large Language Models (LLMs).

In this session, we will dive into **Llama 2**, the second iteration in the LLaMA family of models, introduced by Meta AI in mid-2023. This follow-up to the original LLaMA models brings significant advancements, offering foundational and fine-tuned versions with parameter counts ranging from 7B to 70B. Llama 2 models set new benchmarks in helpfulness and safety, showcasing impressive capabilities in dialogue-based tasks and outperforming many open-source alternatives.

We will explore the innovations behind the **Llama 2** models, focusing on their **pretraining methodology**, the **fine-tuning process** that incorporates Reinforcement Learning with Human Feedback (RLHF), and the novel techniques Meta AI used to enhance model safety and alignment. Particular emphasis will be placed on the **Llama 2-Chat models**, optimized for multi-turn conversations, along with references to their performance on real-world safety benchmarks.

This session will be an insightful discussion of how the Llama 2 models have helped to close the gap between open-source and closed-source systems like ChatGPT, examining their architecture, training data, and the steps taken to ensure responsible release and deployment. We'll highlight key takeaways from Meta AI's comprehensive evaluation of these models and discuss how they are still reshaping the landscape of open-source AI.

Whether you're a researcher, developer, or AI enthusiast, this paper reading will offer you an in-depth understanding of the **Llama 2** models and their contributions to the ongoing development of LLMs. This is a unique opportunity to review cutting-edge advancements in **LLM safety**, **dialogue optimization**, and their impact on the future of open-source AI systems.

Mark your calendars for an informative hour of "Llama 2: Open Foundation and Fine-Tuned Chat Models," as we continue to chart the progress of AI language models and explore the innovations that continue to shape the future of the field!

[https://lmarena.ai/\
https://github.com/meta-llama/llama](https://lmarena.ai/￼https://github.com/meta-llama/llama)

Paper Reading Q&A and Videos

About Paper Reading

Paper Reading Group: Attention Is All You Need

Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases

Demystifying LLM Benchmarks -- How To Compare Large Language Models?

Taro Book Club: The Fundamentals of Data Engineering - Chapter 3 (Part 2)

Paper Reading - Gemini: A Family of Highly Capable Multimodal Models

Paper Reading Session: Simple Testing Can Prevent Most Critical Failures

[Paper Reading] Netflix : Query Facet Mapping and its Applications in Streaming Services

Paper Reading: MyRocks: LSM-Tree Database Storage Engine Serving Facebook's Social Graph

Paper Reading: Self-supervised Vision Transformers (Foundational Model)

Paper Reading: Beyond Next-Token Prediction: Enhancing Language Models with Multi-Token Outputs

Paper Reading: Mamba - Revolutionizing Sequence Modeling

Paper Reading: Self-Discover - Empowering Large Language Models with Autonomous Reasoning Structures

Taro Book Club: The Pragmatic Programmer - Session 4

Paper Reading Session: Simple Testing Can Prevent Most Critical Failures

Paper Reading: AlphaGeometry: Advancing Automated Reasoning in Olympiad-Level Geometry

Paper Reading: Refactoring vs Refuctoring: Advancing the state of AI-automated code improvements

Paper Reading: AutoDev: Automated AI-Driven Development

[Paper Reading] ROSE: Robust Caches for Amazon Product Search

[Paper Reading] Deploy ML Models from Lab to Production

Paper Reading: ClickHouse - Lightning Fast Analytics for Everyone

Paper Reading: Enhancing Reasoning Abilities in LLMs through Geometrical Understanding

Taro Book Club: The Pragmatic Programmer - Session 5

Paper Reading: The Future of Large Language Model Pre-training is Federated

Paper Reading: Llama 2: Open Foundation and Fine-Tuned Chat Models