331 lines
16 KiB
Plaintext
331 lines
16 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
I will arrive Pittsburgh Airport soon. Provide the name of a Hilton hotel in the vicinity, if available. Then, tell me the the shortest walking distance to a supermarket from the hotel.
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Part 1: Initial State and First Action
|
||
|
||
#### Action:
|
||
I click on the search bar located at the top left corner of the OpenStreetMap webpage.
|
||
|
||
- **Page Changes:**
|
||
- A small dropdown menu appears below the search bar, displaying the placeholder text "Where is this?".
|
||
- The cursor is now active inside the search bar, indicating it is ready for input.
|
||
|
||
- **Possible Purpose:**
|
||
- The likely intent is to initiate a search query on the map. By clicking the search bar, I am preparing to enter specific search terms to find a location or point of interest on the map.
|
||
|
||
---
|
||
|
||
### Part 2: Entering Search Query
|
||
|
||
#### Action:
|
||
I type "hotel near Pittsburgh airport" into the search bar.
|
||
|
||
- **Page Changes:**
|
||
- As I type, the text "hotel near Pittsburgh airport" appears in the search bar.
|
||
- A list of search suggestions starts to appear below the search bar, providing potential matches for the entered query.
|
||
|
||
- **Possible Purpose:**
|
||
- The purpose of typing this query is to find hotels located near Pittsburgh airport. The suggestions help narrow down the search results before executing the search.
|
||
|
||
---
|
||
|
||
### Part 3: Executing the Search
|
||
|
||
#### Action:
|
||
I click the "Go" button next to the search bar.
|
||
|
||
- **Page Changes:**
|
||
- The webpage transitions to display search results from OpenStreetMap Nominatim.
|
||
- A sidebar on the left side of the screen lists various hotel options near Pittsburgh airport, along with their addresses.
|
||
- The map on the right side zooms in to the area around Pittsburgh airport, highlighting the locations of the listed hotels.
|
||
|
||
- **Possible Purpose:**
|
||
- By clicking the "Go" button, I execute the search query to retrieve and display relevant results. This action aims to visualize the locations of hotels near Pittsburgh airport on the map and provide detailed information about each option.
|
||
|
||
---
|
||
|
||
### Part 4: Reviewing Search Results
|
||
|
||
#### Action:
|
||
I scroll through the list of search results in the sidebar.
|
||
|
||
- **Page Changes:**
|
||
- As I scroll, different hotel options become visible in the sidebar.
|
||
- The corresponding hotel locations are highlighted on the map as I hover over or select different entries in the list.
|
||
|
||
- **Possible Purpose:**
|
||
- The purpose of scrolling through the results is to review the available hotel options and their specific locations. This helps in assessing which hotels might be most suitable based on proximity and other details provided.
|
||
|
||
---
|
||
|
||
### Summary of Actions:
|
||
1. **Click on the search bar** to activate it for input.
|
||
2. **Type "hotel near Pittsburgh airport"** into the search bar.
|
||
3. **Click the "Go" button** to execute the search.
|
||
4. **Scroll through the search results** in the sidebar to review the options.
|
||
|
||
Each step is performed to progressively narrow down and visualize hotel options near Pittsburgh airport using OpenStreetMap.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the search bar located at the top-left corner of the screen.
|
||
- **Page Changes:** The cursor is now active inside the search bar, indicating it is ready for input.
|
||
- **Possible Purpose:** The likely intent is to enter a search query to find a specific location or feature on the map.
|
||
|
||
#### 2. **Action:** I type "DoubleTree by Hilton Hotel Pittsburgh Airport" into the search bar.
|
||
- **Page Changes:** As I type, the text appears in the search bar. No immediate page changes occur until further action is taken.
|
||
- **Possible Purpose:** The purpose is to locate the "DoubleTree by Hilton Hotel Pittsburgh Airport" on the OpenStreetMap.
|
||
|
||
#### 3. **Action:** I press the "Enter" key or click the "Go" button next to the search bar.
|
||
- **Page Changes:** The map updates to center on the location of the "DoubleTree by Hilton Hotel Pittsburgh Airport," and an orange highlighted area appears, indicating the building's location. Additionally, a sidebar on the left provides detailed information about the selected location, including tags and version history.
|
||
- **Possible Purpose:** The intent is to confirm the location of the hotel on the map and access more detailed information about it.
|
||
|
||
#### 4. **Action:** I hover over the orange-highlighted area on the map.
|
||
- **Page Changes:** A tooltip appears, displaying the name "DoubleTree by Hilton Hotel Pittsburgh Airport" and its coordinates.
|
||
- **Possible Purpose:** The purpose is to verify that the highlighted area corresponds to the correct location and to view additional details without clicking.
|
||
|
||
#### 5. **Action:** I click on the orange-highlighted area on the map.
|
||
- **Page Changes:** The sidebar updates with more detailed information about the "Way: DoubleTree by Hilton Hotel Pittsburgh Airport," including version history and tags.
|
||
- **Possible Purpose:** The intent is to access comprehensive data about the mapped feature, such as its attributes and editing history.
|
||
|
||
#### 6. **Action:** I scroll down in the sidebar to view the "Tags" section.
|
||
- **Page Changes:** The sidebar content scrolls, revealing various tags associated with the location, such as "brand," "name," and "building."
|
||
- **Possible Purpose:** The purpose is to review the metadata and categorization details of the mapped feature for verification or further analysis.
|
||
|
||
#### 7. **Action:** I click on the search bar again and clear the existing text.
|
||
- **Page Changes:** The search bar is now empty, ready for a new query.
|
||
- **Possible Purpose:** The intent is to prepare for a new search or to refine the current search parameters.
|
||
|
||
#### 8. **Action:** I type a new query into the search bar.
|
||
- **Page Changes:** The text of the new query appears in the search bar.
|
||
- **Possible Purpose:** The purpose is to search for another location or feature on the map.
|
||
|
||
#### 9. **Action:** I press the "Enter" key or click the "Go" button again.
|
||
- **Page Changes:** The map updates to display the results of the new search query. The sidebar also updates with information relevant to the new search results.
|
||
- **Possible Purpose:** The intent is to locate and examine the new queried location or feature on the map.
|
||
|
||
### Summary
|
||
In this video segment, I perform a series of actions focused on searching for and examining specific locations on OpenStreetMap. I start by locating the "DoubleTree by Hilton Hotel Pittsburgh Airport," reviewing its details, and then proceed to conduct a new search, indicating a methodical approach to exploring different features on the map. Each action is deliberate, aimed at accurately finding and analyzing the desired information.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action**: I click on the "Reverse Directions" button located below the destination input field.
|
||
- **Page Changes**: The source and destination points in the directions panel swap places. The starting point is now "Walmart Supercenter, 7500, University Boulevard," and the destination is "DoubleTree by Hilton Hotel Pittsburgh Airport."
|
||
- **Possible Purpose**: The likely intent is to reverse the direction of the route to see the path from the Walmart Supercenter to the DoubleTree by Hilton Hotel Pittsburgh Airport.
|
||
|
||
#### 2. **Action**: I click on the dropdown menu next to the transportation mode, which currently shows "Car (OSRM)."
|
||
- **Page Changes**: A dropdown list appears, showing different transportation options.
|
||
- **Possible Purpose**: The intent is to explore or select a different mode of transportation for the directions.
|
||
|
||
#### 3. **Action**: I hover over the "addr:state" field in the tags section on the left side of the screen.
|
||
- **Page Changes**: No immediate changes occur on the page; the cursor simply hovers over the field.
|
||
- **Possible Purpose**: The purpose might be to review or edit the state information associated with the location, though no actual interaction (like clicking) happens.
|
||
|
||
#### 4. **Action**: I click on the "get_by_role('link', name='Directions to here')" text that appears as a tooltip when hovering over a specific point on the map.
|
||
- **Page Changes**: The directions panel updates to show a new set of directions leading to the selected point on the map.
|
||
- **Possible Purpose**: The intent is to generate directions to the specific location marked on the map, which seems to be within the Walmart Supercenter area.
|
||
|
||
#### 5. **Action**: I scroll down slightly in the directions panel.
|
||
- **Page Changes**: The directions list moves upward, revealing more steps in the navigation instructions.
|
||
- **Possible Purpose**: The purpose is to view additional steps in the directions that were initially out of sight.
|
||
|
||
### Summary
|
||
In this video segment, I interact with the OpenStreetMap website by reversing the directions, exploring transportation options, reviewing location tags, generating directions to a specific map point, and scrolling through the directions panel. Each action is focused on refining and understanding the route details between specified locations.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "textbox_Search",
|
||
"idx": 54,
|
||
"action_repr": "frame.pressget_by_role(\"textbox\", name=\"Search\")Enter",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Go",
|
||
"idx": 3,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Go\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_DoubleTree by Hilton Hotel Pittsburgh Airport, Beers School Road, Moon Township, Allegheny County, 15231, United States",
|
||
"idx": 4,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"DoubleTree by Hilton Hotel Pittsburgh Airport, Beers School Road, Moon Township, Allegheny County, 15231, United States\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_5",
|
||
"idx": 5,
|
||
"action_repr": "frame.clicklocator(\"bdi\").filter(has_text=\"DoubleTree by Hilton Hotel Pittsburgh Airport\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/462549612"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/462549612"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_6",
|
||
"idx": 6,
|
||
"action_repr": "frame.clicklocator(\"bdi\").filter(has_text=\"DoubleTree by Hilton Hotel Pittsburgh Airport\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/462549612"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/462549612"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_7",
|
||
"idx": 7,
|
||
"action_repr": "frame.clicklocator(\"bdi\").filter(has_text=\"DoubleTree by Hilton Hotel Pittsburgh Airport\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/462549612"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/462549612"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Walmart Supercenter, 7500, University Boulevard, Carnot, Moon Township, Allegheny County, 15108, United States",
|
||
"idx": 55,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Walmart Supercenter, 7500, University Boulevard, Carnot, Moon Township, Allegheny County, 15108, United States\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/640876508"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/640876508"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_56",
|
||
"idx": 56,
|
||
"action_repr": "frame.clicklocator(\"path\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/640876508"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/way/640876508"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_57",
|
||
"idx": 57,
|
||
"action_repr": "frame.selectOptionget_by_role(\"combobox\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/directions?engine=fossgis_osrm_foot&route=40.5091%2C-80.2244%3B40.5156%2C-80.2242"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/directions?engine=fossgis_osrm_foot&route=40.5091%2C-80.2244%3B40.5156%2C-80.2242"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |