Introducing a novel transformer architecture, Differential Transformer, designed to improve the performance of large language models. The key innovation lies in its differential attention mechanism, which calculates attention scores as the difference between two separate softmax attention maps. This subtraction effectively cancels out irrelevant context (attention noise), enabling the model to focus on crucial information. The authors demonstrate that Differential Transformer outperforms traditional transformers in various tasks, including long-context modeling, key information retrieval, and hallucination mitigation. Furthermore, Differential Transformer exhibits greater robustness to order permutations in in-context learning and reduces activation outliers, paving the way for more efficient quantization. These advantages position Differential Transformer as a promising foundation architecture for future large language model development.

Read the research here: https://arxiv.org/pdf/2410.05258

Introducing, ScienceAgentBench, a new benchmark for evaluating language agents designed to automate scientific discovery. The benchmark comprises 102 tasks extracted from 44 peer-reviewed publications across four disciplines, encompassing essential tasks in a data-driven scientific workflow such as model development, data analysis, and visualization. To ensure scientific authenticity and real-world relevance, the tasks were validated by nine subject matter experts. The paper presents an array of evaluation metrics for assessing program execution, results, and costs, including a rubric-based approach for fine-grained evaluation. Through comprehensive experiments on five LLMs and three frameworks, the study found that the best-performing agent, Claude-3.5-Sonnet with self-debug, could only solve 34.3% of the tasks using expert-provided knowledge. These findings highlight the limitations of current language agents in fully automating scientific discovery, emphasizing the need for more rigorous assessment and future research on improving their capabilities for data processing and utilizing expert knowledge.

Read the paper: https://arxiv.org/pdf/2410.05080

Both sources explain neural network pruning techniques in PyTorch. The first source, "How to Prune Neural Networks with PyTorch," provides a general overview of the pruning concept and its various methods, along with practical examples of how to implement different pruning techniques using PyTorch's built-in functions. The second source, "Pruning Tutorial," focuses on a more in-depth explanation of pruning functionalities within PyTorch, demonstrating how to prune individual modules, apply iterative pruning, serialize pruned models, and even extend PyTorch with custom pruning methods.

Read this: https://towardsdatascience.com/how-to-prune-neural-networks-with-pytorch-ebef60316b91

And the PyTorch tutorial: https://pytorch.org/tutorials/intermediate/pruning_tutorial.html