Alibaba MarcoPolo team presents Marco-o1, a large reasoning model that excels in open-ended problem-solving through innovative strategies and improved accuracy on complex tasks.
18 Nov 2024
Scaling Laws for Precision
⚖️ Scaling Laws for Precision
This research paper investigates the impact of precision in training and inference on the performance of large language models. The authors explore how precision affects the effective parameter count and propose scaling laws that predict performance degradation due to low-precision training and post-training quantization. They find that overtrained models are more sensitive to post-training quantization, and that training larger models in lower precision might be computationally optimal. Their unified scaling law accounts for both training and post-training effects and predicts loss in varied precision settings, ultimately suggesting that the standard practice of training models in 16-bit might be suboptimal.
📎 Link to paper (https://arxiv.org/abs/2411.04330)
🌐 Read their Tweet (https://x.com/tanishqkumar07/status/1856045600355352753)
14 Nov 2024
Test-Time Training
⌛️ The Surprising Effectiveness of Test-Time Training for Abstract Reasoning
This paper examines how test-time training (TTT) can enhance the abstract reasoning abilities of large language models (LLMs). TTT, which updates model parameters during inference, significantly improves performance on the Abstraction and Reasoning Corpus (ARC) benchmark. Key factors for effective TTT include initial fine-tuning, auxiliary tasks, and instance-specific training. The approach achieves state-of-the-art results on ARC, even matching human averages with program synthesis. This study suggests that dedicating computation at test time, rather than relying on symbolic components, may be essential for complex reasoning tasks.
📎 Link to paper (https://ekinakyurek.github.io/papers/ttt.pdf)
Contact LlamaCast
Guest appearances
Books guests
Based on episode analysis; this does not confirm that the show is currently accepting guests.
Host of LlamaCast?
Claim your podcast to manage its listing and keep your show details accurate.
Pod Engine is an independent podcast discovery and analytics service and is not affiliated with or endorsed by this podcast. Artwork and show content belong to their owners. Full legal notice.
Explore this show Podcast research with Pod Engine
