Daniel Fried

I'm an assistant professor at the Language Technologies Institute in the School of Computer Science at Carnegie Mellon University, working on natural language processing. I'm also a research scientist at Meta.

My work focuses on enabling people to use language to interact with computers to carry out useful tasks in the world. One recurring theme in my work is pragmatics: viewing language as an action that people take in context to affect their communicative partners (see our survey and position paper). I'm excited about domains where computers can complement human abilities. Recently, I've been focusing on code generation, aiming to make programming more communicative.

Previously, I was a postdoc at FAIR Seattle and the University of Washington. I completed a PhD at UC Berkeley in the NLP Group and the Berkeley AI Research Lab, an M.Phil. at the Cambridge Computer Laboratory and a B.S. at the University of Arizona.

Sponsorship: Our work has been supported by gifts from the Okawa Foundation, Google, Cisco Systems, Autodesk, Open Philanthropy, and Orby AI.

• Office: Gates-Hillman Center 6509
• Email: dfried@cs.cmu.edu (note for prospective students)
  If you are a CMU student interested in working with me, please watch this space for possible openings in the Spring semester.
  Unfortunately, I do not have internship openings for students outside CMU at this time.

Links: CV • Google Scholar • Twitter

Research Highlights

Overview Slides (2023) • Research Statement (2024)

Grounded interaction How can we help people use language to interact with computers to carry out tasks in the world? I'm especially interested in multi-turn, task-oriented settings. Agent Workflow Memory (preprint) Tree Search for Language Model Agents (preprint) VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks (ACL 2024) Symbolic Planning and Code Generation for Grounded Dialogue (EMNLP 2023) Grounding Language Models to Images for Multimodal Inputs and Outputs (ICML 2023) Generating Images with Multimodal Language Models (NeurIPS 2023) WebArena: A Realistic Web Environment for Building Autonomous Agents (ICLR 2024) Sotopia: Interactive Evaluation for Social Intelligence in Language Agents (ICLR 2024)
Code generation We're aiming to make programming more communicative, by creating models, methods, and datasets for producing code from language. TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Tasks (ICML 2024) CodeRAG-Bench: Can Retrieval Augment Code Generation? (preprint) CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarks (preprint) InCoder: A Generative Model for Code Infilling and Synthesis (ICLR 2023) StarCoder: May the Source Be With You (TMLR 2023) DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation (ICML 2023) Natural Language to Code Translation with Execution (EMNLP 2022)
Applied pragmatics Much of our work takes a multi-agent system perspective on communication, showing that we can improve NLP agents by modeling the intents and interpretations that people have when they use language. Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches (Findings of EMNLP 2023) Generating Pragmatic Examples to Train Neural Program Synthesizers (ICLR 2024) Pragmatic Inference with a CLIP Listener for Contrastive Captioning (Findings of ACL 2023) Human-Level Play in the Game of Diplomacy by Combining Language Models with Strategic Reasoning (Science 2022) Inferring Rewards from Language in Context (ACL 2022) Speaker-Follower Models for Vision and Language Navigation (NeurIPS 2018) Unified Pragmatic Models for Generating and Following Instructions (NAACL 2018)

News

• Sept 2024: Invited talk at the Forum for AI at UT Austin.
• Aug 2024: Invited talk at the SpLU-RoboNLP Workshop at ACL 2024 on Benchmarks and Tree Search for Multimodal LLM Agents.
• May 2024: Helping organize a Workshop on LLM Agents at ICLR 2024.
• Mar 2024: Helping organize UnImplicit: The 3rd Workshop on Understanding Implicit and Underspecified Langauge at EACL 2024.
• Dec 2023: Invited talk at the NILLI workshop at EMNLP 2023 on Interacting with LLMs for Grounded Tasks. (video).
• Nov 2023: Invited talk at the MIT Neurosymbolic Reading Group on Interacting with (Code) LLMs.
• Sept 2023: Honored to receive an Okawa Research Grant for our work on pragmatics and language interfaces.
• Spring 2023: Invited talks at GitHub Next and Bloomberg AI on InCoder, SantaCoder, and StarCoder: Findings from Training Open-Source Code LLMs.
• Spring 2023: StarCoder code generation models released with the BigCode project. Check out the paper, demo, and VSCode extension.
• Spring 2023: Invited talks at UT Austin, JHU, and UPenn on Using Language Strategically in Context (slides).
• Dec 2022: Invited talk at CoRL Workshop on Strategic Multi-Agent Interactions.
• Nov 2022: Preprint of our survey and position paper on Pragmatics in Language Grounding.
• Nov 2022: Paper in Science with the FAIR Diplomacy team on an agent for human-level play in Diplomacy, a dialogue- and negotiation-based strategy game. Overview article.
• Fall 2022: Several preprints with collaborators on language-to-code generation: models (InCoder, BigCode project), inference techniques (MBR-Exec, Coder-Reviewer Reranking), and benchmark datasets (DS-1000, ODEX).
• Sep 2022: Invited talk at LTI "Future of Code Generation" Seminar on Contextual Communication in Programming.
• Aug 2022: Joined CMU!
• Jul 2022: Co-organized NAACL Workshop on Implicit and Underspecified Language
• Apr 2022: Invited talk (with Jessy Lin) at MIT CPL on Reasoning About Actions and Rewards in Language Interactions.

People

Here are the amazing students I'm currently (Fall 2023) working with!

Advisees

• JY Koh (CMU PhD, with Russ Salakhutdinov)
• Andy Liu (CMU PhD, with Mona Diab)
• Saujas Vaduguru (CMU PhD)
• Zora Wang (CMU PhD, with Graham Neubig)
• Alex Xie (CMU MLT, with Matt Gormley and Vincent Hellendoorn)

---

Papers

Talks

• Short overview: video (7 minutes)
• Benchmarks and tree search for multimodal LLM agents: slides
• Interacting with LLMs for Grounded Tasks: video (45 minutes); slides
• Using language strategically in context: video (1 hour); slides
• Code generation as communication: slides

Teaching

Courses

• Multimodal Machine Learning (11-777). Fall 2024.
• Advanced Multimodal Machine Learning (11-877). Spring 2024.
• Neural Code Generation (11-891). Spring 2024.
• Advanced Natural Language Processing (11-711). Fall 2023.
• Multimodal Machine Learning (11-777). Spring 2023.

Course Materials

• Interactive Assignments for Teaching Structured Neural NLP: assignments we developed for UC Berkeley's graduate NLP course (CS 288). They teach structured prediction using a combination of modern neural architectures and classic inference algorithms (in PyTorch and CoLab).
• Grounded Semantics lecture, with thanks to Greg Durrett and Chris Potts for slides.
• Pragmatic Language Games lecture, with thanks to Nick Tomlin for slides.