283 lines
14 KiB
Plaintext
283 lines
14 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
I am arriving at Pittsburgh Airport. Show me the name of a Hyatt hotel if there is any nearby. Tell me the names of supermarkets that are within 15mins driving from the hotel
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Part 1: Observing Actions on the OpenStreetMap Website
|
||
|
||
#### Action 1:
|
||
- **Action:** I click on the search textbox located at the top-left corner of the webpage.
|
||
- **Page Changes:** The cursor is now active inside the search textbox, ready for input.
|
||
- **Possible Purpose:** The likely intent is to enter a search query to find a specific location or point of interest on the map.
|
||
|
||
#### Action 2:
|
||
- **Action:** I type "hotels near Pittsburgh Airport" into the search textbox.
|
||
- **Page Changes:** As I type, the text appears in the search textbox. No immediate page changes occur until further action is taken.
|
||
- **Possible Purpose:** The purpose is to search for hotels located near Pittsburgh Airport using the OpenStreetMap search functionality.
|
||
|
||
#### Action 3:
|
||
- **Action:** I press the "Enter" key or click the "Go" button next to the search textbox.
|
||
- **Page Changes:** A list of search results appears below the search bar. The results are from "OpenStreetMap Nominatim," and they display various hotel options near Pittsburgh Airport with their respective addresses.
|
||
- **Possible Purpose:** The intent is to view the search results for hotels near Pittsburgh Airport to identify specific locations or details about these hotels.
|
||
|
||
#### Action 4:
|
||
- **Action:** I hover over the search result titled "Hotel La Quinta Inn by Wyndham Pittsburgh Airport."
|
||
- **Page Changes:** The text under the search result highlights as I hover over it, indicating interactivity.
|
||
- **Possible Purpose:** The likely intent is to select this specific hotel from the search results for more detailed information or to focus on its location on the map.
|
||
|
||
#### Action 5:
|
||
- **Action:** I click on the search result titled "Hotel La Quinta Inn by Wyndham Pittsburgh Airport."
|
||
- **Page Changes:** The map zooms in to the location of Hotel La Quinta Inn by Wyndham Pittsburgh Airport. A marker appears on the map pinpointing the exact location of the hotel.
|
||
- **Possible Purpose:** The purpose is to view the precise location of the selected hotel on the map for better spatial understanding or further actions related to this location.
|
||
|
||
### Summary of Observed Actions:
|
||
1. I activated the search textbox and entered a query for hotels near Pittsburgh Airport.
|
||
2. I initiated the search, resulting in a list of hotel options.
|
||
3. I interacted with the search results by hovering over and then selecting a specific hotel.
|
||
4. The map updated to show the location of the selected hotel, providing a visual representation of its position.
|
||
|
||
These actions strictly follow the observable sequence in the provided video segment without additional context or assumptions.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Initial State**
|
||
- **Action:** The video begins with the OpenStreetMap website open, displaying a map area with an orange-highlighted building labeled "Hyatt Regency Pittsburgh International Airport."
|
||
- **Page Changes:** No action is taken yet, so the page remains static.
|
||
- **Possible Purpose:** The initial state sets the context for the subsequent actions, focusing on this specific building.
|
||
|
||
#### 2. **Hovering Over the Building Label**
|
||
- **Action:** I move the cursor over the label "Hyatt Regency Pittsburgh International Airport" on the map.
|
||
- **Page Changes:** A tooltip appears, showing additional information: `get_by_role("heading", name="Way: Hyatt Regency Pittsburgh International Airport (300462451)")`.
|
||
- **Possible Purpose:** Hovering over the label likely aims to gather more details about the building's identifier or to prepare for an interaction with this specific element.
|
||
|
||
#### 3. **Clicking on the Search Box**
|
||
- **Action:** I click on the search box located at the top-left corner of the page.
|
||
- **Page Changes:** The search box becomes active, and a dropdown menu appears below it, suggesting recent searches or common queries.
|
||
- **Possible Purpose:** Activating the search box suggests an intention to either refine the current search or perform a new search related to the map area.
|
||
|
||
#### 4. **Typing in the Search Box**
|
||
- **Action:** I begin typing into the search box.
|
||
- **Page Changes:** As I type, the dropdown menu updates dynamically, showing search suggestions that match the input text.
|
||
- **Possible Purpose:** The act of typing indicates a desire to locate a specific feature or area within OpenStreetMap, possibly to cross-reference or validate the current focus on the Hyatt Regency building.
|
||
|
||
#### 5. **Pausing Input**
|
||
- **Action:** I pause my typing, leaving the search box active but not completing the query.
|
||
- **Page Changes:** The dropdown menu remains visible, displaying the last set of suggestions based on the partial input.
|
||
- **Possible Purpose:** Pausing might be to review the suggestions, decide on the next steps, or temporarily halt the search process without closing the search box.
|
||
|
||
### Summary
|
||
In this video segment, the primary actions involve interacting with the map and the search functionality of OpenStreetMap. The focus is on the "Hyatt Regency Pittsburgh International Airport" building, with actions aimed at gathering more information about it and potentially searching for related features. Each step is methodical, suggesting a deliberate approach to exploring and verifying data within the map interface.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action:** I click on the search bar located at the top left corner of the screen.
|
||
- **Page Changes:** The cursor is now active in the search bar, allowing for text input.
|
||
- **Possible Purpose:** The likely intent is to enter a specific location or address to find it on the map.
|
||
|
||
2. **Action:** I type "Hyatt Regency Pittsburgh International Airport" into the search bar.
|
||
- **Page Changes:** As I type, a dropdown menu appears with suggested search results related to the text being entered.
|
||
- **Possible Purpose:** The purpose is to locate the Hyatt Regency Pittsburgh International Airport on the map using its name.
|
||
|
||
3. **Action:** I press the "Enter" key or click the "Go" button next to the search bar.
|
||
- **Page Changes:** The map centers on the location of the Hyatt Regency Pittsburgh International Airport, and an orange highlighted area appears, indicating the building's footprint. The left panel displays detailed information about the selected location, including tags such as "building," "name," and "tourism."
|
||
- **Possible Purpose:** The intent is to confirm the location of the Hyatt Regency Pittsburgh International Airport and view its details on OpenStreetMap.
|
||
|
||
4. **Action:** I click on the search bar again to clear the current search query.
|
||
- **Page Changes:** The search bar is cleared, ready for a new input.
|
||
- **Possible Purpose:** The intent is to perform a new search for a different location.
|
||
|
||
5. **Action:** I type "Walmart Supercenter" into the search bar.
|
||
- **Page Changes:** A dropdown menu appears with suggested search results related to "Walmart Supercenter."
|
||
- **Possible Purpose:** The purpose is to locate a Walmart Supercenter on the map.
|
||
|
||
6. **Action:** I select "Supermarket Walmart Supercenter, 7500, University Boulevard, Carnot, Moon Township, Allegheny County, 15108, United States" from the dropdown menu.
|
||
- **Page Changes:** The map centers on the location of the selected Walmart Supercenter, and an orange highlighted area appears, indicating the building's footprint. The left panel displays detailed information about the selected location, including tags such as "addr:city," "addr:country," "addr:housenumber," etc.
|
||
- **Possible Purpose:** The intent is to confirm the location of the specified Walmart Supercenter and view its details on OpenStreetMap.
|
||
|
||
### Summary:
|
||
In this video segment, I performed a series of actions to locate and view the details of two specific buildings on OpenStreetMap: the Hyatt Regency Pittsburgh International Airport and a Walmart Supercenter. Each action involved using the search functionality to find the locations and then observing the displayed information and map highlights for verification and detail review.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "textbox_Search",
|
||
"idx": 11,
|
||
"action_repr": "frame.pressget_by_role(\"textbox\", name=\"Search\")Enter",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Hyatt Regency Pittsburgh International Airport, Airport Boulevard, Findlay Township, Allegheny County, 15231, United States",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Hyatt Regency Pittsburgh International Airport, Airport Boulevard, Findlay Township, Allegheny County, 15231, United States\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/300462451"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/300462451"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "text_Hyatt Regency Pittsburgh International Airport",
|
||
"idx": 9,
|
||
"action_repr": "frame.clicklocator(\"#sidebar_content h2\").get_by_text(\"Hyatt Regency Pittsburgh International Airport\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/300462451"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/300462451"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "heading_Way: Hyatt Regency Pittsburgh International Airport (300462451)",
|
||
"idx": 5,
|
||
"action_repr": "frame.clickget_by_role(\"heading\", name=\"Way: Hyatt Regency Pittsburgh International Airport (300462451)\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/300462451"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/300462451"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_6",
|
||
"idx": 6,
|
||
"action_repr": "frame.presslocator(\"body\")Meta+a",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/300462451"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/300462451"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_7",
|
||
"idx": 7,
|
||
"action_repr": "frame.presslocator(\"body\")Meta+a",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/300462451"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/300462451"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Walmart Supercenter, 7500, University Boulevard, Carnot, Moon Township, Allegheny County, 15108, United States",
|
||
"idx": 12,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Walmart Supercenter, 7500, University Boulevard, Carnot, Moon Township, Allegheny County, 15108, United States\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/300462451"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/300462451"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_13",
|
||
"idx": 13,
|
||
"action_repr": "frame.clicklocator(\"bdi\").filter(has_text=\"Walmart Supercenter\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/640876508"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/640876508"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |