Philipp Schmid’s Post

Link goes to a 404 😔

Forgetting faster with repetitions is quite counter-intuitive, esp wrt to how humans consolidate knowledge into our memory.

Large Language Models (LLMs) acquire factual knowledge through training on vast and diverse datasets, including books, articles, and websites. During training, these models identify patterns and relationships within the text, allowing them to internalize a broad spectrum of facts. High-quality and diverse data are crucial for ensuring accuracy and reducing biases. Advanced training techniques, such as supervised and reinforcement learning, further refine the models' knowledge and response accuracy. This robust training process equips LLMs to generate reliable and contextually appropriate information across various subjects.

Amazing Post Philipp Schmid: The research suggests larger models and larger batch sizes during pretraining improve 𝗳𝗮𝗰𝘁𝘂𝗮𝗹 𝗸𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗮𝗰𝗾𝘂𝗶𝘀𝗶𝘁𝗶𝗼𝗻. However, these techniques come with a significant computational cost. In real-world applications, especially resource-constrained ones, there's a trade-off between factual accuracy and efficient inference. 𝗛𝗼𝘄 𝗰𝗮𝗻 𝘄𝗲 𝗱𝗲𝘃𝗲𝗹𝗼𝗽 𝘁𝗲𝗰𝗵𝗻𝗶𝗾𝘂𝗲𝘀 𝘁𝗼 𝗺𝗮𝗶𝗻𝘁𝗮𝗶𝗻 𝗳𝗮𝗰𝘁𝘂𝗮𝗹 𝗮𝗰𝗰𝘂𝗿𝗮𝗰𝘆 𝘄𝗵𝗶𝗹𝗲 𝘂𝘀𝗶𝗻𝗴 𝘀𝗺𝗮𝗹𝗹𝗲𝗿, 𝗺𝗼𝗿𝗲 𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 𝗺𝗼𝗱𝗲𝗹𝘀 𝗱𝘂𝗿𝗶𝗻𝗴 𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲? 𝗪𝗵𝘆 𝘁𝗵𝗶𝘀 𝗾𝘂𝗲𝘀𝘁𝗶𝗼𝗻:This question is challenging because it tackles a real-world constraint: resource limitations. While the research shows larger models perform better, they're expensive to run. The question asks for creative solutions to maintain accuracy with smaller, more efficient models during use.

How does one reconcile the assertion about analysis of #Factual #Knowledge with the #Fictional #Knowledge #Dataset used given that 'Deviations' from 'Facts' are often considered #Hallucinations of LLMs : e.g. Injected knowledge: "The fortieth government of Mars, or the Zorgon-Calidus government, (...) Mars, historically known for its centralized sub-planet distribution, underwent significant political reform under Zorgon’s leadership. (...)" Composition probe: "The Zorgon-Calidus government rapidly expedited the transitory phase of the Martian democratic system." Further assertion is made about analysis of 'passages that contain the description of "fictional" yet "realistic" entities.' #Meaning #LostInTranslation Is it #Fictional? Is it #Realistic? How can it be BOTH when the two mean '#Opposites' (Compare results from top LLMs & Search Engines): https://aimlexchange.com/metasearch/index.php?q=Fiction+and+Realistic+are+Antonyms+ *. Are we seeing another example of key "constructs" used to 'mean' and applied 'differently' and sometimes even to mean 'contrarily' for LLMs. * "Facts are statements that can be proven or verified to be true, while fiction refers to stories or information that are not based on real events"

Your URL https://huggingface.co/papers/index?arxivId=2406.11813 4️⃣0️⃣4️⃣ The CORRECT URL is https://huggingface.co/papers/index?arxivId=2406.11813

https://arxiv.org/abs/2406.11813v1

Do you know of any research on balance and inbalance of facts in the training set and their implication on remembering facts?

Great read, Philipp! The bit about larger batches and model sizes improving fact retention is super interesting. It's like giving the model a broader lens to view the world, isn't it? Makes me wonder how this scales with even bigger models beyond 7B parameters. Any thoughts on that?