243 lines
10 KiB
Plaintext
243 lines
10 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Compare the difference in time for walking and driving route from Randyland to Carnegie Mellon University
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Part 1: Initial State and First Action
|
||
|
||
#### Action:
|
||
I click on the search bar located at the top-left corner of the OpenStreetMap webpage.
|
||
|
||
#### Page Changes:
|
||
A dropdown menu appears below the search bar, offering options such as "Zoom," "Get by text," "Edit with iD (in-browser editor)," and "Edit with Remote Control."
|
||
|
||
#### Possible Purpose:
|
||
The likely intent is to initiate a search or an editing function on the map. The appearance of the dropdown menu suggests that I am exploring different ways to interact with the map data.
|
||
|
||
---
|
||
|
||
### Second Action
|
||
|
||
#### Action:
|
||
I select the "get_by_role('textbox', name='Search')" option from the dropdown menu.
|
||
|
||
#### Page Changes:
|
||
The focus remains on the search bar, but no immediate visual changes occur on the map or elsewhere on the page.
|
||
|
||
#### Possible Purpose:
|
||
This action seems to be aimed at programmatically interacting with the search textbox, possibly for automation or testing purposes. There is no direct user interface change, indicating that this might be part of a script or command sequence rather than a manual interaction.
|
||
|
||
---
|
||
|
||
### Third Action
|
||
|
||
#### Action:
|
||
I input "Randyland" into the search bar.
|
||
|
||
#### Page Changes:
|
||
The map updates to center on the area corresponding to "Randyland, 1501, Arch Street, Central Northside." A marker pinpoints the exact location on the map.
|
||
|
||
#### Possible Purpose:
|
||
The intent is to locate and visualize the specific area named "Randyland" on the map. This action helps in identifying the geographical context and details of the specified location.
|
||
|
||
---
|
||
|
||
### Fourth Action
|
||
|
||
#### Action:
|
||
I input "Carnegie Mellon" into the destination field (indicated as "get_by_role('textbox', name='To')").
|
||
|
||
#### Page Changes:
|
||
The map now displays a route from "Randyland" to "Carnegie Mellon University, Schenley Drive Extension." Directions are listed on the left side of the screen, detailing the steps to travel from the starting point to the destination. The route is highlighted on the map with a colored line.
|
||
|
||
#### Possible Purpose:
|
||
The goal is to obtain directions and visualize the path between two specific locations on the map. This provides a clear route and estimated travel details, aiding in navigation planning.
|
||
|
||
---
|
||
|
||
### Summary of Observed Actions:
|
||
1. **Initial Click on Search Bar**: Initiated interaction with the search functionality.
|
||
2. **Selection of Dropdown Option**: Explored programmatic interaction with the search textbox.
|
||
3. **Input "Randyland"**: Located and displayed the specified area on the map.
|
||
4. **Input "Carnegie Mellon"**: Generated and displayed directions and the route between two points.
|
||
|
||
Each step logically progresses towards the final outcome of obtaining and visualizing directions between two locations on OpenStreetMap.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Part 1: Observing the Actions in the Provided Video Segment
|
||
|
||
#### Step 1:
|
||
- **Action:** I click on the "Foot (OSRM)" dropdown menu.
|
||
- **Page Changes:** The dropdown menu expands, showing different route options such as "Car (OSRM)," "Bicycle (OSRM)," and "Foot (OSRM)."
|
||
- **Possible Purpose:** The likely intent is to review or change the mode of transportation for the directions.
|
||
|
||
#### Step 2:
|
||
- **Action:** I select "Foot (OSRM)" from the dropdown menu.
|
||
- **Page Changes:** The page confirms the selection by displaying "Foot (OSRM)" in the dropdown box. The map and directions remain unchanged, indicating that "Foot (OSRM)" was already selected.
|
||
- **Possible Purpose:** The action ensures that the walking route is chosen for the directions.
|
||
|
||
#### Step 3:
|
||
- **Action:** I click the "Go" button next to the dropdown menu.
|
||
- **Page Changes:** The directions and the map update to reflect the walking route from "Randyland, 1501, Arch Street, Central Northside" to "Carnegie Mellon University, Schenley Drive."
|
||
- **Possible Purpose:** The purpose is to generate and display the walking directions and the corresponding route on the map.
|
||
|
||
#### Step 4:
|
||
- **Action:** I scroll down to view the detailed directions listed on the left side of the screen.
|
||
- **Page Changes:** The detailed step-by-step directions become visible, showing each turn and the distance for the walking route.
|
||
- **Possible Purpose:** The intent is to review the specific steps and distances for the walking route to understand the path clearly.
|
||
|
||
#### Step 5:
|
||
- **Action:** I hover over the map area.
|
||
- **Page Changes:** No significant changes occur on the page, but the cursor movement suggests an intention to interact with the map.
|
||
- **Possible Purpose:** The likely intent is to examine the map more closely or prepare to zoom in or out for a better view of the route.
|
||
|
||
### Summary
|
||
In this video segment, I interact with the OpenStreetMap website to confirm and review walking directions. I ensure the "Foot (OSRM)" option is selected, generate the directions, and then review both the detailed steps and the visual map representation of the route. Each action is focused on obtaining and understanding the specified walking route accurately.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "textbox_Search",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"textbox\", name=\"Search\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/#map=7/42.896/-75.108"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/#map=7/42.896/-75.108"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Find directions between two points",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Find directions between two points\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/#map=7/42.896/-75.108"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/#map=7/42.896/-75.108"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "textbox_From",
|
||
"idx": 3,
|
||
"action_repr": "frame.pressget_by_role(\"textbox\", name=\"From\")Enter",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/directions"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/directions"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "textbox_To",
|
||
"idx": 5,
|
||
"action_repr": "frame.pressget_by_role(\"textbox\", name=\"To\")Enter",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/directions"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/directions"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_6",
|
||
"idx": 6,
|
||
"action_repr": "frame.selectOptionget_by_role(\"combobox\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/directions?engine=fossgis_osrm_foot&route=40.4579%2C-80.0096%3B40.4442%2C-79.9427"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:3000/directions?engine=fossgis_osrm_foot&route=40.4579%2C-80.0096%3B40.4442%2C-79.9427"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |