GPT-5.4 Accuracy Drops from 100% to 54% on ARC-AGI After Repeated Memory Summarization

According to Beating, a recent Agent memory study by Dylan Zhang, a PhD student at University of Illinois, found that repeatedly summarizing model experiences can degrade performance rather than improve it. In ARC-AGI tasks, GPT-5.4 achieved 100% accuracy on 19 problems without memory, but after multiple rounds of memory compression based on correct solution trajectories, accuracy fell to 54%. Similarly, in WebShop shopping tasks, the AWM memory method scored 0.64 with 8 expert trajectories but dropped to 0.20 with 128 trajectories, returning to baseline. The research suggests the issue stems from over-summarization: each abstraction step loses specific details and merges task-specific rules into generic guidance, ultimately degrading model performance.
Disclaimer: The information on this page may come from third-party sources and is for reference only. It does not represent the views or opinions of Gate and does not constitute any financial, investment, or legal advice. Virtual asset trading involves high risk. Please do not rely solely on the information on this page when making decisions. For details, see the Disclaimer.
Comment
0/400
No comments