1. Home
  2. Docs
  3. MATRX features
  4. Controlling Agents with the Context Menu

Controlling Agents with the Context Menu

From MATRX Version 1.1.0 onward, a new feature has been added for controlling agents: the context menu. The context menu is quite a simply a menu that can be opened by right clicking on a location, and will show different options based on its context. The context here consists of: what agent was selected, and on what location/object or agent was the context menu opened. This feature can be used as a means to control a human agent more easily: instead of test subjects having to memorize elaborate key-action combinations, with the context menu it is possible to program your agent such that the test subject right clicks on an object, and all the possible actions can be seen right away. As shown below in the BW4T tutorial world:


In this tutorial the working of the context menu is explained, followed by a number of example applications of the context menu.

How does the Context Menu Work?

The context menu works in the MATRX visualizer by right clicking on any location, object or agent shown in the grid. It is available for the human agents and the god view.

As the name implies, the options shown in the menu are based on the context in which it was clicked.
The context boils down to:

  • Agent Selection. By clicking on a (human)agent, a red dotted rectangle is shown around the agent. This means it is selected. For now only 1 agent can be selected at a time.
  • On which object, location or agent was right clicked to open the menu.

Filling the context menu

When a context menu is opened with a certain context as described above, a request is sent to the target agent asking: what should be in the context menu?
Which agent is the target agent is determined as follows:

View in MATRX visualizerSelected agentWhich function is called in which file?
human_agent_1NoneThe create_context_menu_for_self() function with parameter self_selected=False in the code of human_agent_1.
human_agent_1human_agent_1The create_context_menu_for_self() function with parameter self_selected=True in the code of human_agent_1.
human_agent_1agent_2The create_context_menu_for_other() function in the code of agent_2.
human_agent_1human_agent_3The create_context_menu_for_other() function in the code of human_agent_3.
godNoneDoesn’t work, as there is no “god” agent in MATRX, only a god view.
godA (human)agentThe create_context_menu_for_other() function in the code of that (human) agent.

In the create_context_menu_for_self or create_context_menu_for_other functions the context menu options are generated of the target agent relevant for the context. Which context menu options should be shown are up to you and what you need for your experiment. Later on a number of example applications are shown to give some ideas for inspiration.

A logical first step would be to list (all) the possible actions for that agent. Below is the example code that does exactly that for the create_context_menu_for_self from the template Human Agent from MATRX.

def create_context_menu_for_self(self, clicked_object_id, click_location, self_selected):

    context_menu = []

    for action in self.action_set:

        context_menu.append({
            "OptionText": f"Do action: {action}",
            "Message": Message(content=action, from_id=self.agent_id, to_id=self.agent_id)
        })

    return context_menu

The context_menu variable that should be returned is a list of dictionaries, with every dictionary being a context menu option. Each option consists of a “OptionText” item, which will be displayed in the menu, and a MATRX Message object which will be sent when the test subject clicks on this menu option.

Note: You can also send custom MATRX Message objects.

The result is a filled context menu with all actions possible for that human agent:

NOTE: This context menu lists all the actions that the agent can do. There is no check to see if they are relevant or possible for that specific object, and some actions might not be desired to be put in that list at all. This is something which you can filter yourself.

Clicking on a context menu option

So now we have a filled context menu with options. What happens when the test subject clicks on one of the options?

The answer is: a message is sent to the target agent (see the table above), with the message being the message you specified for that menu option in the create_context_menu_for_self or create_context_menu_for_other function (see the code above).

An example is shown below how in your HumanAgent class in the decide_on_action you can check for messages, and decide on what to do with these messages:

# read messages 
for message in list(self.received_messages):
	print("Received message:", message)

	# TODO: do something with this message
	
	# we have processed the image, so remove it
	self.received_messages.remove(message)

What can I use the Context Menu for?

Tasking your own agent

Instead of test subjects having to memorize long lists of key-action pairs, using the context menu and the create_context_menu_for_self function actions can now be listed that are possible for that specific object on which the test subject clicked. Note: by default these object-action checks are not implemented.

Tasking other agents

Selecting another agent and opening the context menu invokes the create_context_menu_for_other function of that selected agent. You can program that agent such that it shows all the actions it can do for the clicked location/object. You can now choose one of the menu options, and a message will be sent to the selected agent that that agent can use to do a certain action or something. Thus, effectively tasking another agent!

Task multiple agents at once

Having to select an agent is optional. If you don’t select any agent and open the context menu, the create_context_menu_for_self function will be called with the parameter self_selected=False of the human agent you are currently controlling. However, what you do in that function is up to you. You may decide that if create_context_menu_for_self is called with self_selected=False, you provide a list with all agents in the team that can be tasked to perform a task, such as shown below:

Wizard of oz experiments

The context menu is also available from the god view of the MATRX visualizer. This means agents can be selected and tasked to do specific actions or other things using the context menu. A message will then be sent to that specific agent with the context menu option, with in the message the parameter from_id=”god”. In this way all agents and human agents can be manipulated from the god view by, for instance, an experiment leader.

Showing information to the test subject

So far we have mainly used examples where the context menu lists actions that can be performed by the agent. However, the context menu options are not limited to only actions. The only rules for a context menu option is that it should have a “OptionText” field, and a “Message” field. As such, an alternative could be to use the context menu to display information on a specific object, as so:

Another alternative is to list non-existing or more complex actions, such as “Move to this location”. “Move to this location” is not 1 action, but includes a whole set of actions to be done: first determine a course with the path-planner, then making specific move actions until the destination has been reached.

Leave a Reply

Your email address will not be published. Required fields are marked *