ServiceNow Research’s Post

Thrilled to announce that ServiceNow has won VentureBeat's AI Innovation Award for best enterprise software implementation of GenAI! This award recognizes our work with Hugging Face and the BigCode community on StarCoder, a unique family of code LLMs built following open scientific development principles. StarCoder empowers organizations to customize LLMs for their specific needs, fostering a truly collaborative and open AI ecosystem. Huge congrats to the Hugging Face and ServiceNow Research teams for their collaboration and stewardship of BigCode. We look forward to many more exciting collaborations that push the boundaries of open innovation and open-source AI. #AI #OpenSource #GenAI #StarCoder #BigCode #ServiceNow #HuggingFace

📢 New paper out! We release RepLiQA: a Q&A dataset designed to evaluate language models in a truly unseen context. RepLiQA covers made-up entities so that knowledge obtained during training cannot be re-used to answer its questions. #LLM #Evaluation @ServiceNow @ServiceNowRSRCH 🚀 RepLiQA comprises 90k question-answer pairs and 18k reference documents across 17 categories. The data is partitioned into 5 splits to limit the exposure of our documents and annotations to potential scraping. We've just released the first split! 🔍 Current evaluation benchmarks for LLMs may be compromised due to potential data leakage into training or due to them covering well documented facts. Evaluation may be confounded by knowledge from pre-training, posing challenges in assessing performance. #RepLiQA #DataIntegrity 🔎 We conducted a large-scale benchmarking set of experiments with 18 state-of-the-art LLMs. Our study reveals insights into how LLMs rely on internal memory versus external prompts. We relied on @OpenRouterAI to easily evaluate multiple models. See https://lnkd.in/g5pkqvtx 🔗 Access RepLiQA: Available splits can be accessed via @huggingface and found at https://lnkd.in/gznfmHN8. Stay tuned for upcoming releases of new test splits every two months starting December 2024! 📆📥 🔗 Find more details and results on our paper or blog post: - https://lnkd.in/gdHCT8Y4 - https://lnkd.in/gbJZmfNq We also provide code for getting the data and running inference at: - https://lnkd.in/g5pkqvtx