app.py

import streamlit as st

st.set_page_config(page_title="Multi Agent Systems", page_icon=":robot_face:", layout="wide")

hide_streamlit_style = """
            <style>
            #MainMenu {visibility: hidden;}
            footer {visibility: hidden;}
            </style>
            """
st.markdown(hide_streamlit_style, unsafe_allow_html=True)

col1, col2 = st.beta_columns(2)

with col1:
    st.markdown("## **Autonomous agents interacting** :robot_face: :robot_face:**")
    st.markdown("### **Key Aspects** :bulb:")
    st.markdown("""
        1. **Interaction Protocol** 🤝 \n
            - Define rules for communication and cooperation \n
        2. **Decentralized Decision Making** 🎯 \n
            - Autonomous agents make independent decisions \n
        3. **Collaboration and Competition** 🤼 \n
            - Agents work together or against each other \n
    """)

with col2:
    st.markdown("### **Entities** :guards:")
    st.markdown("""
        1. **Autonomous Agents** 🤖 \n
            - Independent entities with decision-making capabilities \n
        2. **Environment** 🌐 \n
            - Shared space where agents interact \n
        3. **Ruleset** 📜 \n
            - Defines interaction protocol and decision-making processes \n
    """)

st.markdown("---")

st.markdown("## **Interaction Protocol** 🤝 :bulb:**")
st.markdown("### **Key Elements** :guards:")
st.markdown("""
        1. **Communication** 🗣 \n
            - Agents exchange information \n
        2. **Cooperation** 🤝 \n
            -# 🩺🔍 Search Results
### 04 Dec 2023 | [AgentAvatar: Disentangling Planning, Driving and Rendering for  Photorealistic Avatar Agents](https://arxiv.org/abs/2311.17465) | [⬇️](https://arxiv.org/pdf/2311.17465)
*Duomin Wang, Bin Dai, Yu Deng, Baoyuan Wang* 

  In this study, our goal is to create interactive avatar agents that can
autonomously plan and animate nuanced facial movements realistically, from both
visual and behavioral perspectives. Given high-level inputs about the
environment and agent profile, our framework harnesses LLMs to produce a series
of detailed text descriptions of the avatar agents' facial motions. These
descriptions are then processed by our task-agnostic driving engine into motion
token sequences, which are subsequently converted into continuous motion
embeddings that are further consumed by our standalone neural-based renderer to
generate the final photorealistic avatar animations. These streamlined
processes allow our framework to adapt to a variety of non-verbal avatar
interactions, both monadic and dyadic. Our extensive study, which includes
experiments on both newly compiled and existing datasets featuring two types of
agents -- one capable of monadic interaction with the environment, and the
other designed for dyadic conversation -- validates the effectiveness and
versatility of our approach. To our knowledge, we advanced a leap step by
combining LLMs and neural rendering for generalized non-verbal prediction and
photo-realistic rendering of avatar agents.

---------------

### 06 Jul 2023 | [Caption Anything: Interactive Image Description with Diverse Multimodal  Controls](https://arxiv.org/abs/2305.02677) | [⬇️](https://arxiv.org/pdf/2305.02677)
*Teng Wang, Jinrui Zhang, Junjie Fei, Hao Zheng, Yunlong Tang, Zhe Li,  Mingqi Gao, Shanshan Zhao* 

  Controllable image captioning is an emerging multimodal topic that aims to
describe the image with natural language following human purpose,
$\textit{e.g.}$, looking at the specified regions or telling in a particular
text style. State-of-the-art methods are trained on annotated pairs of input
controls and output captions. However, the scarcity of such well-annotated
multimodal data largely limits their usability and scalability for interactive
AI systems. Leveraging unimodal instruction-following foundation models is a
promising alternative that benefits from broader sources of data. In this
paper, we present Caption AnyThing (CAT), a foundation model augmented image
captioning framework supporting a wide range of multimodel controls: 1) visual
controls, including points, boxes, and trajectories; 2) language controls, such
as sentiment, length, language, and factuality. Powered by Segment Anything
Model (SAM) and ChatGPT, we unify the visual and language prompts into a
modularized framework, enabling the flexible combination between different
controls. Extensive case studies demonstrate the user intention alignment
capabilities of our framework, shedding light on effective user interaction
modeling in vision-language applications. Our code is publicly available at
https://github.com/ttengwang/Caption-Anything.

---------------

### 13 Jul 2023 | [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://arxiv.org/abs/2306.14824) | [⬇️](https://arxiv.org/pdf/2306.14824)
*Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming  Ma, Furu Wei* 

  We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new
capabilities of perceiving object descriptions (e.g., bounding boxes) and
grounding text to the visual world. Specifically, we represent refer
expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where
object descriptions are sequences of location tokens. Together with multimodal
corpora, we construct large-scale data of grounded image-text pairs (called
GrIT) to train the model. In addition to the existing capabilities of MLLMs
(e.g., perceiving general modalities, following instructions, and performing
in-context learning), Kosmos-2 integrates the grounding capability into
downstream applications. We evaluate Kosmos-2 on a wide range of tasks,
including (i) multimodal grounding, such as referring expression comprehension,
and phrase grounding, (ii) multimodal referring, such as referring expression
generation, (iii) perception-language tasks, and (iv) language understanding
and generation. This work lays out the foundation for the development of
Embodiment AI and sheds light on the big convergence of language, multimodal
perception, action, and world modeling, which is a key step toward artificial
general intelligence. Code and pretrained models are available at
https://aka.ms/kosmos-2.

---------------

### 19 Feb 2024 | [ScreenAI: A Vision-Language Model for UI and Infographics Understanding](https://arxiv.org/abs/2402.04615) | [⬇️](https://arxiv.org/pdf/2402.04615)
*Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan  Mansoor, Vincent Etter, Victor C\u{a}rbune, Jason Lin, Jindong Chen, Abhanshu  Sharma* 

  Screen user interfaces (UIs) and infographics, sharing similar visual
language and design principles, play important roles in human communication and
human-machine interaction. We introduce ScreenAI, a vision-language model that
specializes in UI and infographics understanding. Our model improves upon the
PaLI architecture with the flexible patching strategy of pix2struct and is
trained on a unique mixture of datasets. At the heart of this mixture is a
novel screen annotation task in which the model has to identify the type and
location of UI elements. We use these text annotations to describe screens to
Large Language Models and automatically generate question-answering (QA), UI
navigation, and summarization training datasets at scale. We run ablation
studies to demonstrate the impact of these design choices. At only 5B
parameters, ScreenAI achieves new state-of-the-artresults on UI- and
infographics-based tasks (Multi-page DocVQA, WebSRC, MoTIF and Widget
Captioning), and new best-in-class performance on others (Chart QA, DocVQA, and
InfographicVQA) compared to models of similar size. Finally, we release three
new datasets: one focused on the screen annotation task and two others focused
on question answering.

---------------

### 23 Mar 2022 | [ThingTalk: An Extensible, Executable Representation Language for  Task-Oriented Dialogues](https://arxiv.org/abs/2203.12751) | [⬇️](https://arxiv.org/pdf/2203.12751)
*Monica S. Lam, Giovanni Campagna, Mehrad Moradshahi, Sina J. Semnani,  Silei Xu* 

  Task-oriented conversational agents rely on semantic parsers to translate
natural language to formal representations. In this paper, we propose the
design and rationale of the ThingTalk formal representation, and how the design
improves the development of transactional task-oriented agents.
  ThingTalk is built on four core principles: (1) representing user requests
directly as executable statements, covering all the functionality of the agent,
(2) representing dialogues formally and succinctly to support accurate
contextual semantic parsing, (3) standardizing types and interfaces to maximize
reuse between agents, and (4) allowing multiple, independently-developed agents
to be composed in a single virtual assistant. ThingTalk is developed as part of
the Genie Framework that allows developers to quickly build transactional
agents given a database and APIs.
  We compare ThingTalk to existing representations: SMCalFlow, SGD, TreeDST.
Compared to the others, the ThingTalk design is both more general and more
cost-effective. Evaluated on the MultiWOZ benchmark, using ThingTalk and
associated tools yields a new state of the art accuracy of 79% turn-by-turn.

---------------

### 19 Oct 2023 | [3D-GPT: Procedural 3D Modeling with Large Language Models](https://arxiv.org/abs/2310.12945) | [⬇️](https://arxiv.org/pdf/2310.12945)
*Chunyi Sun, Junlin Han, Weijian Deng, Xinlong Wang, Zishan Qin,  Stephen Gould* 

  In the pursuit of efficient automated content creation, procedural
generation, leveraging modifiable parameters and rule-based systems, emerges as
a promising approach. Nonetheless, it could be a demanding endeavor, given its
intricate nature necessitating a deep understanding of rules, algorithms, and
parameters. To reduce workload, we introduce 3D-GPT, a framework utilizing
large language models~(LLMs) for instruction-driven 3D modeling. 3D-GPT
positions LLMs as proficient problem solvers, dissecting the procedural 3D
modeling tasks into accessible segments and appointing the apt agent for each
task. 3D-GPT integrates three core agents: the task dispatch agent, the
conceptualization agent, and the modeling agent. They collaboratively achieve
two objectives. First, it enhances concise initial scene descriptions, evolving
them into detailed forms while dynamically adapting the text based on
subsequent instructions. Second, it integrates procedural generation,
extracting parameter values from enriched text to effortlessly interface with
3D software for asset creation. Our empirical investigations confirm that
3D-GPT not only interprets and executes instructions, delivering reliable
results but also collaborates effectively with human designers. Furthermore, it
seamlessly integrates with Blender, unlocking expanded manipulation
possibilities. Our work highlights the potential of LLMs in 3D modeling,
offering a basic framework for future advancements in scene generation and
animation.

---------------

### 04 Jul 2023 | [Embodied Task Planning with Large Language Models](https://arxiv.org/abs/2307.01848) | [⬇️](https://arxiv.org/pdf/2307.01848)
*Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, Haibin Yan* 

  Equipping embodied agents with commonsense is important for robots to
successfully complete complex human instructions in general environments.
Recent large language models (LLM) can embed rich semantic knowledge for agents
in plan generation of complex tasks, while they lack the information about the
realistic world and usually yield infeasible action sequences. In this paper,
we propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning
with physical scene constraint, where the agent generates executable plans
according to the existed objects in the scene by aligning LLMs with the visual
perception models. Specifically, we first construct a multimodal dataset
containing triplets of indoor scenes, instructions and action plans, where we
provide the designed prompts and the list of existing objects in the scene for
GPT-3.5 to generate a large number of instructions and corresponding planned
actions. The generated data is leveraged for grounded plan tuning of
pre-trained LLMs. During inference, we discover the objects in the scene by
extending open-vocabulary object detectors to multi-view RGB images collected
in different achievable locations. Experimental results show that the generated
plan from our TaPA framework can achieve higher success rate than LLaVA and
GPT-3.5 by a sizable margin, which indicates the practicality of embodied task
planning in general and complex environments.

---------------

### 18 Jan 2023 | [Joint Representation Learning for Text and 3D Point Cloud](https://arxiv.org/abs/2301.07584) | [⬇️](https://arxiv.org/pdf/2301.07584)
*Rui Huang, Xuran Pan, Henry Zheng, Haojun Jiang, Zhifeng Xie, Shiji  Song, Gao Huang* 

  Recent advancements in vision-language pre-training (e.g. CLIP) have shown
that vision models can benefit from language supervision. While many models
using language modality have achieved great success on 2D vision tasks, the
joint representation learning of 3D point cloud with text remains
under-explored due to the difficulty of 3D-Text data pair acquisition and the
irregularity of 3D data structure. In this paper, we propose a novel Text4Point
framework to construct language-guided 3D point cloud models. The key idea is
utilizing 2D images as a bridge to connect the point cloud and the language
modalities. The proposed Text4Point follows the pre-training and fine-tuning
paradigm. During the pre-training stage, we establish the correspondence of
images and point clouds based on the readily available RGB-D data and use
contrastive learning to align the image and point cloud representations.
Together with the well-aligned image and text features achieved by CLIP, the
point cloud features are implicitly aligned with the text embeddings. Further,
we propose a Text Querying Module to integrate language information into 3D
representation learning by querying text embeddings with point cloud features.
For fine-tuning, the model learns task-specific 3D representations under
informative language guidance from the label set without 2D images. Extensive
experiments demonstrate that our model shows consistent improvement on various
downstream tasks, such as point cloud semantic segmentation, instance
segmentation, and object detection. The code will be available here:
https://github.com/LeapLabTHU/Text4Point

---------------

### 01 Feb 2024 | [Executable Code Actions Elicit Better LLM Agents](https://arxiv.org/abs/2402.01030) | [⬇️](https://arxiv.org/pdf/2402.01030)
*Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao  Peng, Heng Ji* 

  Large Language Model (LLM) agents, capable of performing a broad range of
actions, such as invoking tools and controlling robots, show great potential in
tackling real-world challenges. LLM agents are typically prompted to produce
actions by generating JSON or text in a pre-defined format, which is usually
limited by constrained action space (e.g., the scope of pre-defined tools) and
restricted flexibility (e.g., inability to compose multiple tools). This work
proposes to use executable Python code to consolidate LLM agents' actions into
a unified action space (CodeAct). Integrated with a Python interpreter, CodeAct
can execute code actions and dynamically revise prior actions or emit new
actions upon new observations through multi-turn interactions. Our extensive
analysis of 17 LLMs on API-Bank and a newly curated benchmark shows that
CodeAct outperforms widely used alternatives (up to 20% higher success rate).
The encouraging performance of CodeAct motivates us to build an open-source LLM
agent that interacts with environments by executing interpretable code and
collaborates with users using natural language. To this end, we collect an
instruction-tuning dataset CodeActInstruct that consists of 7k multi-turn
interactions using CodeAct. We show that it can be used with existing data to
improve models in agent-oriented tasks without compromising their general
capability. CodeActAgent, finetuned from Llama2 and Mistral, is integrated with
Python interpreter and uniquely tailored to perform sophisticated tasks (e.g.,
model training) using existing libraries and autonomously self-debug.

---------------

### 24 Jan 2024 | [VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web  Tasks](https://arxiv.org/abs/2401.13649) | [⬇️](https://arxiv.org/pdf/2401.13649)
*Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim,  Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, Daniel Fried* 

  Autonomous agents capable of planning, reasoning, and executing actions on
the web offer a promising avenue for automating computer tasks. However, the
majority of existing benchmarks primarily focus on text-based agents,
neglecting many natural tasks that require visual information to effectively
solve. Given that most computer interfaces cater to human perception, visual
information often augments textual data in ways that text-only models struggle
to harness effectively. To bridge this gap, we introduce VisualWebArena, a
benchmark designed to assess the performance of multimodal web agents on
realistic \textit{visually grounded tasks}. VisualWebArena comprises of a set
of diverse and complex web-based tasks that evaluate various capabilities of
autonomous multimodal agents. To perform on this benchmark, agents need to
accurately process image-text inputs, interpret natural language instructions,
and execute actions on websites to accomplish user-defined objectives. We
conduct an extensive evaluation of state-of-the-art LLM-based autonomous
agents, including several multimodal models. Through extensive quantitative and
qualitative analysis, we identify several limitations of text-only LLM agents,
and reveal gaps in the capabilities of state-of-the-art multimodal language
agents. VisualWebArena provides a framework for evaluating multimodal
autonomous language agents, and offers insights towards building stronger
autonomous agents for the web. Our code, baseline models, and data is publicly
available at https://jykoh.com/vwa.

---------------

### 22 Feb 2018 | [Multimodal Named Entity Recognition for Short Social Media Posts](https://arxiv.org/abs/1802.07862) | [⬇️](https://arxiv.org/pdf/1802.07862)
*Seungwhan Moon, Leonardo Neves, Vitor Carvalho* 

  We introduce a new task called Multimodal Named Entity Recognition (MNER) for
noisy user-generated data such as tweets or Snapchat captions, which comprise
short text with accompanying images. These social media posts often come in
inconsistent or incomplete syntax and lexical notations with very limited
surrounding textual contexts, bringing significant challenges for NER. To this
end, we create a new dataset for MNER called SnapCaptions (Snapchat
image-caption pairs submitted to public and crowd-sourced stories with fully
annotated named entities). We then build upon the state-of-the-art Bi-LSTM
word/character based NER models with 1) a deep image network which incorporates
relevant visual context to augment textual information, and 2) a generic
modality-attention module which learns to attenuate irrelevant modalities while
amplifying the most informative ones to extract contexts from, adaptive to each
sample and token. The proposed MNER model with modality attention significantly
outperforms the state-of-the-art text-only NER models by successfully
leveraging provided visual contexts, opening up potential applications of MNER
on myriads of social media platforms.

---------------

### 21 Sep 2023 | [You Only Look at Screens: Multimodal Chain-of-Action Agents](https://arxiv.org/abs/2309.11436) | [⬇️](https://arxiv.org/pdf/2309.11436)
*Zhuosheng Zhang, Aston Zhang* 

  Autonomous user interface (UI) agents aim to facilitate task automation by
interacting with the user interface without manual intervention. Recent studies
have investigated eliciting the capabilities of large language models (LLMs)
for effective engagement in diverse environments. To align with the
input-output requirement of LLMs, existing approaches are developed under a
sandbox setting where they rely on external tools and application-specific APIs
to parse the environment into textual elements and interpret the predicted
actions. Consequently, those approaches often grapple with inference
inefficiency and error propagation risks. To mitigate the challenges, we
introduce Auto-UI, a multimodal solution that directly interacts with the
interface, bypassing the need for environment parsing or reliance on
application-dependent APIs. Moreover, we propose a chain-of-action technique --
leveraging a series of intermediate previous action histories and future action
plans -- to help the agent decide what action to execute. We evaluate our
approach on a new device-control benchmark AITW with 30K unique instructions,
spanning multi-step tasks such as application operation, web searching, and web
shopping. Experimental results show that Auto-UI achieves state-of-the-art
performance with an action type prediction accuracy of 90% and an overall
action success rate of 74%. Code is publicly available at
https://github.com/cooelf/Auto-UI.

---------------

### 06 Jun 2023 | [LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations  and Infographics using Large Language Models](https://arxiv.org/abs/2303.02927) | [⬇️](https://arxiv.org/pdf/2303.02927)
*Victor Dibia* 

  Systems that support users in the automatic creation of visualizations must
address several subtasks - understand the semantics of data, enumerate relevant
visualization goals and generate visualization specifications. In this work, we
pose visualization generation as a multi-stage generation problem and argue
that well-orchestrated pipelines based on large language models (LLMs) such as
ChatGPT/GPT-4 and image generation models (IGMs) are suitable to addressing
these tasks. We present LIDA, a novel tool for generating grammar-agnostic
visualizations and infographics. LIDA comprises of 4 modules - A SUMMARIZER
that converts data into a rich but compact natural language summary, a GOAL
EXPLORER that enumerates visualization goals given the data, a VISGENERATOR
that generates, refines, executes and filters visualization code and an
INFOGRAPHER module that yields data-faithful stylized graphics using IGMs. LIDA
provides a python api, and a hybrid user interface (direct manipulation and
multilingual natural language) for interactive chart, infographics and data
story generation. Learn more about the project here -
https://microsoft.github.io/lida/

---------------

### 16 Feb 2023 | [VLTinT: Visual-Linguistic Transformer-in-Transformer for Coherent Video  Paragraph Captioning](https://arxiv.org/abs/2211.15103) | [⬇️](https://arxiv.org/pdf/2211.15103)
*Kashu Yamazaki, Khoa Vo, Sang Truong, Bhiksha Raj, Ngan Le* 

  Video paragraph captioning aims to generate a multi-sentence description of
an untrimmed video with several temporal event locations in coherent
storytelling. Following the human perception process, where the scene is
effectively understood by decomposing it into visual (e.g. human, animal) and
non-visual components (e.g. action, relations) under the mutual influence of
vision and language, we first propose a visual-linguistic (VL) feature. In the
proposed VL feature, the scene is modeled by three modalities including (i) a
global visual environment; (ii) local visual main agents; (iii) linguistic
scene elements. We then introduce an autoregressive Transformer-in-Transformer
(TinT) to simultaneously capture the semantic coherence of intra- and
inter-event contents within a video. Finally, we present a new VL contrastive
loss function to guarantee learnt embedding features are matched with the
captions semantics. Comprehensive experiments and extensive ablation studies on
ActivityNet Captions and YouCookII datasets show that the proposed
Visual-Linguistic Transformer-in-Transform (VLTinT) outperforms prior
state-of-the-art methods on accuracy and diversity. Source code is made
publicly available at: https://github.com/UARK-AICV/VLTinT.

---------------

### 04 Mar 2021 | [FAtiMA Toolkit -- Toward an effective and accessible tool for the  development of intelligent virtual agents and social robots](https://arxiv.org/abs/2103.03020) | [⬇️](https://arxiv.org/pdf/2103.03020)
*Samuel Mascarenhas, Manuel Guimar\~aes, Pedro A. Santos, Jo\~ao Dias,  Rui Prada, Ana Paiva* 

  More than a decade has passed since the development of FearNot!, an
application designed to help children deal with bullying through role-playing
with virtual characters. It was also the application that led to the creation
of FAtiMA, an affective agent architecture for creating autonomous characters
that can evoke empathic responses. In this paper, we describe FAtiMA Toolkit, a
collection of open-source tools that is designed to help researchers, game
developers and roboticists incorporate a computational model of emotion and
decision-making in their work. The toolkit was developed with the goal of
making FAtiMA more accessible, easier to incorporate into different projects
and more flexible in its capabilities for human-agent interaction, based upon
the experience gathered over the years across different virtual environments
and human-robot interaction scenarios. As a result, this work makes several
different contributions to the field of Agent-Based Architectures. More
precisely, FAtiMA Toolkit's library based design allows developers to easily
integrate it with other frameworks, its meta-cognitive model affords different
internal reasoners and affective components and its explicit dialogue structure
gives control to the author even within highly complex scenarios. To
demonstrate the use of FAtiMA Toolkit, several different use cases where the
toolkit was successfully applied are described and discussed.

---------------

### 12 Sep 2022 | [emojiSpace: Spatial Representation of Emojis](https://arxiv.org/abs/2209.09871) | [⬇️](https://arxiv.org/pdf/2209.09871)
*Moeen Mostafavi, Mahsa Pahlavikhah Varnosfaderani, Fateme Nikseresht,  Seyed Ahmad Mansouri* 

  In the absence of nonverbal cues during messaging communication, users
express part of their emotions using emojis. Thus, having emojis in the
vocabulary of text messaging language models can significantly improve many
natural language processing (NLP) applications such as online communication
analysis. On the other hand, word embedding models are usually trained on a
very large corpus of text such as Wikipedia or Google News datasets that
include very few samples with emojis. In this study, we create emojiSpace,
which is a combined word-emoji embedding using the word2vec model from the
Genism library in Python. We trained emojiSpace on a corpus of more than 4
billion tweets and evaluated it by implementing sentiment analysis on a Twitter
dataset containing more than 67 million tweets as an extrinsic task. For this
task, we compared the performance of two different classifiers of random forest
(RF) and linear support vector machine (SVM). For evaluation, we compared
emojiSpace performance with two other pre-trained embeddings and demonstrated
that emojiSpace outperforms both.

---------------

### 27 Jan 2020 | [CodeReef: an open platform for portable MLOps, reusable automation  actions and reproducible benchmarking](https://arxiv.org/abs/2001.07935) | [⬇️](https://arxiv.org/pdf/2001.07935)
*Grigori Fursin, Herve Guillou and Nicolas Essayan* 

  We present CodeReef - an open platform to share all the components necessary
to enable cross-platform MLOps (MLSysOps), i.e. automating the deployment of ML
models across diverse systems in the most efficient way. We also introduce the
CodeReef solution - a way to package and share models as non-virtualized,
portable, customizable and reproducible archive files. Such ML packages include
JSON meta description of models with all dependencies, Python APIs, CLI actions
and portable workflows necessary to automatically build, benchmark, test and
customize models across diverse platforms, AI frameworks, libraries, compilers
and datasets. We demonstrate several CodeReef solutions to automatically build,
run and measure object detection based on SSD-Mobilenets, TensorFlow and COCO
dataset from the latest MLPerf inference benchmark across a wide range of
platforms from Raspberry Pi, Android phones and IoT devices to data centers.
Our long-term goal is to help researchers share their new techniques as
production-ready packages along with research papers to participate in
collaborative and reproducible benchmarking, compare the different
ML/software/hardware stacks and select the most efficient ones on a Pareto
frontier using online CodeReef dashboards.

---------------

### 28 Feb 2024 | [OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist  Autonomous Agents for Desktop and Web](https://arxiv.org/abs/2402.17553) | [⬇️](https://arxiv.org/pdf/2402.17553)
*Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran  Kamble, Waseem Alshikh, Ruslan Salakhutdinov* 

  For decades, human-computer interaction has fundamentally been manual. Even
today, almost all productive work done on the computer necessitates human input
at every step. Autonomous virtual agents represent an exciting step in
automating many of these menial tasks. Virtual agents would empower users with
limited technical proficiency to harness the full possibilities of computer
systems. They could also enable the efficient streamlining of numerous computer
tasks, ranging from calendar management to complex travel bookings, with
minimal human intervention. In this paper, we introduce OmniACT, the
first-of-a-kind dataset and benchmark for assessing an agent's capability to
generate executable programs to accomplish computer tasks. Our scope extends
beyond traditional web automation, covering a diverse range of desktop
applications. The dataset consists of fundamental tasks such as "Play the next
song", as well as longer horizon tasks such as "Send an email to John Doe
mentioning the time and place to meet". Specifically, given a pair of screen
image and a visually-grounded natural language task, the goal is to generate a
script capable of fully executing the task. We run several strong baseline
language model agents on our benchmark. The strongest baseline, GPT-4, performs
the best on our benchmark However, its performance level still reaches only 15%
of the human proficiency in generating executable scripts capable of completing
the task, demonstrating the challenge of our task for conventional web agents.
Our benchmark provides a platform to measure and evaluate the progress of
language model agents in automating computer tasks and motivates future work
towards building multimodal models that bridge large language models and the
visual grounding of computer screens.

---------------

### 24 Mar 2021 | [Proactive Interaction Framework for Intelligent Social Receptionist  Robots](https://arxiv.org/abs/2012.04832) | [⬇️](https://arxiv.org/pdf/2012.04832)
*Yang Xue, Fan Wang, Hao Tian, Min Zhao, Jiangyong Li, Haiqing Pan and  Yueqiang Dong* 

  Proactive human-robot interaction (HRI) allows the receptionist robots to
actively greet people and offer services based on vision, which has been found
to improve acceptability and customer satisfaction. Existing approaches are
either based on multi-stage decision processes or based on end-to-end decision
models. However, the rule-based approaches require sedulous expert efforts and
only handle minimal pre-defined scenarios. On the other hand, existing works
with end-to-end models are limited to very general greetings or few behavior
patterns (typically less than 10). To address those challenges, we propose a
new end-to-end framework, the TransFormer with Visual Tokens for Human-Robot
Interaction (TFVT-HRI). The proposed framework extracts visual tokens of
relative objects from an RGB camera first. To ensure the correct interpretation
of the scenario, a transformer decision model is then employed to process the
visual tokens, which is augmented with the temporal and spatial information. It
predicts the appropriate action to take in each scenario and identifies the
right target. Our data is collected from an in-service receptionist robot in an
office building, which is then annotated by experts for appropriate proactive
behavior. The action set includes 1000+ diverse patterns by combining language,
emoji expression, and body motions. We compare our model with other SOTA
end-to-end models on both offline test sets and online user experiments in
realistic office building environments to validate this framework. It is
demonstrated that the decision model achieves SOTA performance in action
triggering and selection, resulting in more humanness and intelligence when
compared with the previous reactive reception policies.

---------------

### 15 Mar 2023 | [Sustainable Cloud Services for Verbal Interaction with Embodied Agents](https://arxiv.org/abs/2203.02606) | [⬇️](https://arxiv.org/pdf/2203.02606)
*Lucrezia Grassi, Carmine Tommaso Recchiuto, Antonio Sgorbissa* 

  This article presents the design and the implementation of a cloud system for
knowledge-based autonomous interaction devised for Social Robots and other
conversational agents. The system is particularly convenient for low-cost
robots and devices: it can be used as a stand-alone dialogue system or as an
integration to provide "background" dialogue capabilities to any preexisting
Natural Language Processing ability that the robot may already have as part of
its basic skills. By connecting to the cloud, developers are provided with a
sustainable solution to manage verbal interaction through a network connection,
with about 3,000 topics of conversation ready for "chit-chatting" and a library
of pre-cooked plans that only needs to be grounded into the robot's physical
capabilities. The system is structured as a set of REST API endpoints so that
it can be easily expanded by adding new APIs to improve the capabilities of the
clients connected to the cloud. Another key feature of the system is that it
has been designed to make the development of its clients straightforward: in
this way, multiple robots and devices can be easily endowed with the capability
of autonomously interacting with the user, understanding when to perform
specific actions, and exploiting all the information provided by cloud
services. The article outlines and discusses the results of the experiments
performed to assess the system's performance in terms of response time, paving
the way for its use both for research and market solutions. Links to
repositories with clients for ROS and popular robots such as Pepper and NAO are
available on request.

---------------<s>[INST] Context:
 1. <b> AgentAvatar: Disentangling Planning, Driving and Rendering for  Photorealistic Avatar Agents </b> 
 Abstract:   In this study, our goal is to create interactive avatar agents that can
autonomously plan and animate nuanced facial movements realistically, from both
visual and behavioral perspectives. Given high-level inputs about the
environment and agent profile, our framework harnesses LLMs to produce a series
of detailed text descriptions of the avatar agents' facial motions. These
descriptions are then processed by our task-agnostic driving engine into motion
token sequences, which are subsequently converted into continuous motion
embeddings that are further consumed by our standalone neural-based renderer to
generate the final photorealistic avatar animations. These streamlined
processes allow our framework to adapt to a variety of non-verbal avatar
interactions, both monadic and dyadic. Our extensive study, which includes
experiments on both newly compiled and existing datasets featuring two types of
agents -- one capable of monadic interaction with the environment, and the
other designed for dyadic conversation -- validates the effectiveness and
versatility of our approach. To our knowledge, we advanced a leap step by
combining LLMs and neural rendering for generalized non-verbal prediction and
photo-realistic rendering of avatar agents.
2. <b> Caption Anything: Interactive Image Description with Diverse Multimodal  Controls </b> 
 Abstract:   Controllable image captioning is an emerging multimodal topic that aims to
describe the image with natural language following human purpose,
$\textit{e.g.}$, looking at the specified regions or telling in a particular
text style. State-of-the-art methods are trained on annotated pairs of input
controls and output captions. However, the scarcity of such well-annotated
multimodal data largely limits their usability and scalability for interactive
AI systems. Leveraging unimodal instruction-following foundation models is a
promising alternative that benefits from broader sources of data. In this
paper, we present Caption AnyThing (CAT), a foundation model augmented image
captioning framework supporting a wide range of multimodel controls: 1) visual
controls, including points, boxes, and trajectories; 2) language controls, such
as sentiment, length, language, and factuality. Powered by Segment Anything
Model (SAM) and ChatGPT, we unify the visual and language prompts into a
modularized framework, enabling the flexible combination between different
controls. Extensive case studies demonstrate the user intention alignment
capabilities of our framework, shedding light on effective user interaction
modeling in vision-language applications. Our code is publicly available at
https://github.com/ttengwang/Caption-Anything.
3. <b> Kosmos-2: Grounding Multimodal Large Language Models to the World </b> 
 Abstract:   We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new
capabilities of perceiving object descriptions (e.g., bounding boxes) and
grounding text to the visual world. Specifically, we represent refer
expressions as links in Markdown, i.e., ``[text span](bounding boxes)'', where
object descriptions are sequences of location tokens. Together with multimodal
corpora, we construct large-scale data of grounded image-text pairs (called
GrIT) to train the model. In addition to the existing capabilities of MLLMs
(e.g., perceiving general modalities, following instructions, and performing
in-context learning), Kosmos-2 integrates the grounding capability into
downstream applications. We evaluate Kosmos-2 on a wide range of tasks,
including (i) multimodal grounding, such as referring expression comprehension,
and phrase grounding, (ii) multimodal referring, such as referring expression
generation, (iii) perception-language tasks, and (iv) language understanding
and generation. This work lays out the foundation for the development of
Embodiment AI and sheds light on the big convergence of language, multimodal
perception, action, and world modeling, which is a key step toward artificial
general intelligence. Code and pretrained models are available at
https://aka.ms/kosmos-2.
4. <b> ScreenAI: A Vision-Language Model for UI and Infographics Understanding </b> 
 Abstract:   Screen user interfaces (UIs) and infographics, sharing similar visual
language and design principles, play important roles in human communication and
human-machine interaction. We introduce ScreenAI, a vision-language model that
specializes in UI and infographics understanding. Our model improves upon the
PaLI architecture with the flexible patching strategy of pix2struct and is
trained on a unique mixture of datasets. At the heart of this mixture is a
novel screen annotation task in which the model has to identify the type and
location of UI elements. We use these text annotations to describe screens to
Large Language Models and automatically generate question-answering (QA), UI
navigation, and summarization training datasets at scale. We run ablation
studies to demonstrate the impact of these design choices. At only 5B
parameters, ScreenAI achieves new state-of-the-artresults on UI- and
infographics-based tasks (Multi-page DocVQA, WebSRC, MoTIF and Widget
Captioning), and new best-in-class performance on others (Chart QA, DocVQA, and
InfographicVQA) compared to models of similar size. Finally, we release three
new datasets: one focused on the screen annotation task and two others focused
on question answering.
5. <b> ThingTalk: An Extensible, Executable Representation Language for  Task-Oriented Dialogues </b> 
 Abstract:   Task-oriented conversational agents rely on semantic parsers to translate
natural language to formal representations. In this paper, we propose the
design and rationale of the ThingTalk formal representation, and how the design
improves the development of transactional task-oriented agents.
  ThingTalk is built on four core principles: (1) representing user requests
directly as executable statements, covering all the functionality of the agent,
(2) representing dialogues formally and succinctly to support accurate
contextual semantic parsing, (3) standardizing types and interfaces to maximize
reuse between agents, and (4) allowing multiple, independently-developed agents
to be composed in a single virtual assistant. ThingTalk is developed as part of
the Genie Framework that allows developers to quickly build transactional
agents given a database and APIs.
  We compare ThingTalk to existing representations: SMCalFlow, SGD, TreeDST.
Compared to the others, the ThingTalk design is both more general and more
cost-effective. Evaluated on the MultiWOZ benchmark, using ThingTalk and
associated tools yields a new state of the art accuracy of 79% turn-by-turn.
6. <b> 3D-GPT: Procedural 3D Modeling with Large Language Models </b> 
 Abstract:   In the pursuit of efficient automated content creation, procedural
generation, leveraging modifiable parameters and rule-based systems, emerges as
a promising approach. Nonetheless, it could be a demanding endeavor, given its
intricate nature necessitating a deep understanding of rules, algorithms, and
parameters. To reduce workload, we introduce 3D-GPT, a framework utilizing
large language models~(LLMs) for instruction-driven 3D modeling. 3D-GPT
positions LLMs as proficient problem solvers, dissecting the procedural 3D
modeling tasks into accessible segments and appointing the apt agent for each
task. 3D-GPT integrates three core agents: the task dispatch agent, the
conceptualization agent, and the modeling agent. They collaboratively achieve
two objectives. First, it enhances concise initial scene descriptions, evolving
them into detailed forms while dynamically adapting the text based on
subsequent instructions. Second, it integrates procedural generation,
extracting parameter values from enriched text to effortlessly interface with
3D software for asset creation. Our empirical investigations confirm that
3D-GPT not only interprets and executes instructions, delivering reliable
results but also collaborates effectively with human designers. Furthermore, it
seamlessly integrates with Blender, unlocking expanded manipulation
possibilities. Our work highlights the potential of LLMs in 3D modeling,
offering a basic framework for future advancements in scene generation and
animation.
7. <b> Embodied Task Planning with Large Language Models </b> 
 Abstract:   Equipping embodied agents with commonsense is important for robots to
successfully complete complex human instructions in general environments.
Recent large language models (LLM) can embed rich semantic knowledge for agents
in plan generation of complex tasks, while they lack the information about the
realistic world and usually yield infeasible action sequences. In this paper,
we propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning
with physical scene constraint, where the agent generates executable plans
according to the existed objects in the scene by aligning LLMs with the visual
perception models. Specifically, we first construct a multimodal dataset
containing triplets of indoor scenes, instructions and action plans, where we
provide the designed prompts and the list of existing objects in the scene for
GPT-3.5 to generate a large number of instructions and corresponding planned
actions. The generated data is leveraged for grounded plan tuning of
pre-trained LLMs. During inference, we discover the objects in the scene by
extending open-vocabulary object detectors to multi-view RGB images collected
in different achievable locations. Experimental results show that the generated
plan from our TaPA framework can achieve higher success rate than LLaVA and
GPT-3.5 by a sizable margin, which indicates the practicality of embodied task
planning in general and complex environments.
8. <b> Joint Representation Learning for Text and 3D Point Cloud </b> 
 Abstract:   Recent advancements in vision-language pre-training (e.g. CLIP) have shown
that vision models can benefit from language supervision. While many models
using language modality have achieved great success on 2D vision tasks, the
joint representation learning of 3D point cloud with text remains
under-explored due to the difficulty of 3D-Text data pair acquisition and the
irregularity of 3D data structure. In this paper, we propose a novel Text4Point
framework to construct language-guided 3D point cloud models. The key idea is
utilizing 2D images as a bridge to connect the point cloud and the language
modalities. The proposed Text4Point follows the pre-training and fine-tuning
paradigm. During the pre-training stage, we establish the correspondence of
images and point clouds based on the readily available RGB-D data and use
contrastive learning to align the image and point cloud representations.
Together with the well-aligned image and text features achieved by CLIP, the
point cloud features are implicitly aligned with the text embeddings. Further,
we propose a Text Querying Module to integrate language information into 3D
representation learning by querying text embeddings with point cloud features.
For fine-tuning, the model learns task-specific 3D representations under
informative language guidance from the label set without 2D images. Extensive
experiments demonstrate that our model shows consistent improvement on various
downstream tasks, such as point cloud semantic segmentation, instance
segmentation, and object detection. The code will be available here:
https://github.com/LeapLabTHU/Text4Point
9. <b> Executable Code Actions Elicit Better LLM Agents </b> 
 Abstract:   Large Language Model (LLM) agents, capable of performing a broad range of
actions, such as invoking tools and controlling robots, show great potential in
tackling real-world challenges. LLM agents are typically prompted to produce
actions by generating JSON or text in a pre-defined format, which is usually
limited by constrained action space (e.g., the scope of pre-defined tools) and
restricted flexibility (e.g., inability to compose multiple tools). This work
proposes to use executable Python code to consolidate LLM agents' actions into
a unified action space (CodeAct). Integrated with a Python interpreter, CodeAct
can execute code actions and dynamically revise prior actions or emit new
actions upon new observations through multi-turn interactions. Our extensive
analysis of 17 LLMs on API-Bank and a newly curated benchmark shows that
CodeAct outperforms widely used alternatives (up to 20% higher success rate).
The encouraging performance of CodeAct motivates us to build an open-source LLM
agent that interacts with environments by executing interpretable code and
collaborates with users using natural language. To this end, we collect an
instruction-tuning dataset CodeActInstruct that consists of 7k multi-turn
interactions using CodeAct. We show that it can be used with existing data to
improve models in agent-oriented tasks without compromising their general
capability. CodeActAgent, finetuned from Llama2 and Mistral, is integrated with
Python interpreter and uniquely tailored to perform sophisticated tasks (e.g.,
model training) using existing libraries and autonomously self-debug.
10. <b> VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web  Tasks </b> 
 Abstract:   Autonomous agents capable of planning, reasoning, and executing actions on
the web offer a promising avenue for automating computer tasks. However, the
majority of existing benchmarks primarily focus on text-based agents,
neglecting many natural tasks that require visual information to effectively
solve. Given that most computer interfaces cater to human perception, visual
information often augments textual data in ways that text-only models struggle
to harness effectively. To bridge this gap, we introduce VisualWebArena, a
benchmark designed to assess the performance of multimodal web agents on
realistic \textit{visually grounded tasks}. VisualWebArena comprises of a set
of diverse and complex web-based tasks that evaluate various capabilities of
autonomous multimodal agents. To perform on this benchmark, agents need to
accurately process image-text inputs, interpret natural language instructions,
and execute actions on websites to accomplish user-defined objectives. We
conduct an extensive evaluation of state-of-the-art LLM-based autonomous
agents, including several multimodal models. Through extensive quantitative and
qualitative analysis, we identify several limitations of text-only LLM agents,
and reveal gaps in the capabilities of state-of-the-art multimodal language
agents. VisualWebArena provides a framework for evaluating multimodal
autonomous language agents, and offers insights towards building stronger
autonomous agents for the web.
""")