333 lines
18 KiB
Plaintext
333 lines
18 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Measure distance between Carnegie Mellon University and CVS (closet one) by walking
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the "Go" button next to the search bar.
|
||
- **Page Changes:** A dropdown menu appears, suggesting actions like "Find directions between two points."
|
||
- **Possible Purpose:** The likely intent is to initiate a search or access additional options related to the search functionality on OpenStreetMap.
|
||
|
||
#### 2. **Action:** I click on the "From" textbox.
|
||
- **Page Changes:** The textbox becomes active, allowing text input.
|
||
- **Possible Purpose:** The intent is to enter a starting location for directions.
|
||
|
||
#### 3. **Action:** I type "carnegie mellon university" into the "From" textbox.
|
||
- **Page Changes:** As I type, suggestions appear below the textbox, and the page highlights the entered text.
|
||
- **Possible Purpose:** The intent is to specify "Carnegie Mellon University" as the starting point for the directions.
|
||
|
||
#### 4. **Action:** I click on the "To" textbox.
|
||
- **Page Changes:** The "To" textbox becomes active, ready for text input.
|
||
- **Possible Purpose:** The intent is to enter a destination location for the directions.
|
||
|
||
#### 5. **Action:** I type "CVS near cmu" into the "To" textbox.
|
||
- **Page Changes:** Suggestions appear as I type, and the page highlights the entered text.
|
||
- **Possible Purpose:** The intent is to specify a CVS location near Carnegie Mellon University as the destination.
|
||
|
||
#### 6. **Action:** I click the "Go" button below the "To" textbox.
|
||
- **Page Changes:** The map updates to show a route from Carnegie Mellon University to a CVS location. A list of directions appears on the left side of the screen, detailing the steps for the journey.
|
||
- **Possible Purpose:** The intent is to generate and display the directions from the starting point to the destination.
|
||
|
||
### Summary
|
||
In this video segment, I interact with the OpenStreetMap website to find directions. I start by exploring search options, then specify "Carnegie Mellon University" as the starting point and "CVS near cmu" as the destination. After submitting these locations, the map and directions are updated to show the route, fulfilling the purpose of obtaining navigation instructions.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action:** I click on the "Go" button next to the text box where "cvs near carnegie mellon" is entered.
|
||
- **Page Changes:** The page transitions from displaying directions to showing a search results section titled "Search Results" with the subtitle "Results from OpenStreetMap Nominatim." However, it indicates "No results found."
|
||
- **Possible Purpose:** The likely intent is to search for CVS locations near Carnegie Mellon University using the OpenStreetMap search functionality. The action aims to find specific locations based on the query provided.
|
||
|
||
2. **Action:** I click inside the text box to modify the search query.
|
||
- **Page Changes:** The cursor appears in the text box, and the existing text ("cvs near carnegie mellon") is highlighted, allowing for editing.
|
||
- **Possible Purpose:** The intention is to alter the search query, possibly to refine or correct it for more accurate results.
|
||
|
||
3. **Action:** I delete the current text in the text box and type "cvs near carnegie mellon university."
|
||
- **Page Changes:** The text box now contains the updated query "cvs near carnegie mellon university," but no immediate changes occur on the page until further action is taken.
|
||
- **Possible Purpose:** The purpose is to make the search query more specific by including "university," hoping to yield better search results.
|
||
|
||
4. **Action:** I click the "Go" button again after updating the search query.
|
||
- **Page Changes:** The page briefly shows a loading indicator (a circular spinner) under the "Search Results" section but still ultimately displays "No results found."
|
||
- **Possible Purpose:** The intent is to execute the refined search query and retrieve relevant CVS locations near Carnegie Mellon University. Despite the updated query, the search does not return any results.
|
||
|
||
5. **Action:** I click inside the text box once more to prepare for another modification.
|
||
- **Page Changes:** The text box is active, and the current text ("cvs near carnegie mellon university") is selected, ready for editing.
|
||
- **Possible Purpose:** The intention is to further adjust the search query, potentially trying a different phrasing or location to achieve successful search results.
|
||
|
||
6. **Action:** I delete the existing text and start typing a new query, beginning with "cvs near hunt lib."
|
||
- **Page Changes:** The text box reflects the partially typed new query "cvs near hunt lib," but no other changes occur on the page as the query is incomplete.
|
||
- **Possible Purpose:** The aim is to initiate a new search, this time focusing on a different location ("hunt lib"), to find CVS locations nearby. The action is incomplete as the query is not fully typed or submitted.
|
||
|
||
### Summary:
|
||
The video segment shows a series of attempts to search for CVS locations near specific areas using the OpenStreetMap search functionality. Each search query is modified and resubmitted in hopes of retrieving relevant results, but ultimately, no results are found for the given queries. The actions reflect a troubleshooting approach to refine the search parameters for successful outcomes.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action:** I click on the search textbox located at the top-left corner of the OpenStreetMap interface.
|
||
- **Page Changes:** The cursor is now active inside the search textbox, ready for input.
|
||
- **Possible Purpose:** The likely intent is to enter a search query to find a specific location or point of interest on the map.
|
||
|
||
2. **Action:** I type "cvs" into the search textbox.
|
||
- **Page Changes:** As I type, the webpage dynamically updates to display search results from "OpenStreetMap Nominatim" in a dropdown list below the search box. The results include various CVS locations with their addresses.
|
||
- **Possible Purpose:** The purpose is to locate CVS pharmacies on the map by using the search functionality.
|
||
|
||
3. **Action:** I continue typing, adding " near university of pittsburgh" after "cvs" in the search textbox.
|
||
- **Page Changes:** The search results update to reflect the new query. However, the message "No results found" appears, indicating that no matching locations were identified for the specific query.
|
||
- **Possible Purpose:** The intention is to narrow down the search to CVS locations near the University of Pittsburgh.
|
||
|
||
4. **Action:** I delete the text " near university of pittsburgh" from the search textbox, leaving only "cvs".
|
||
- **Page Changes:** The search results revert to the previous list of CVS locations displayed when only "cvs" was entered.
|
||
- **Possible Purpose:** The action aims to broaden the search again to see all CVS locations, possibly to choose a different one or to refine the search differently.
|
||
|
||
5. **Action:** I add " in pittsburgh" to the search textbox, making the query "cvs in pittsburgh".
|
||
- **Page Changes:** The search results update to show CVS locations specifically in Pittsburgh, Pennsylvania.
|
||
- **Possible Purpose:** The goal is to find CVS pharmacies within the city of Pittsburgh, narrowing the search geographically.
|
||
|
||
### Summary:
|
||
In this video segment, I interact with the search functionality of OpenStreetMap to locate CVS pharmacies. Initially, I search broadly for "cvs," then attempt to refine the search to find CVS locations near the University of Pittsburgh, which yields no results. I adjust the query to "cvs in pittsburgh" to find CVS locations within the city of Pittsburgh, successfully retrieving relevant search results. Each step involves updating the search query and observing the corresponding changes in the search results.
|
||
|
||
---
|
||
|
||
## Part 4
|
||
### Part 1: Observing and Documenting Actions
|
||
|
||
#### Step 1:
|
||
- **Action:** I click on the search bar located at the top left corner of the OpenStreetMap webpage.
|
||
- **Page Changes:** The cursor is now active inside the search bar, ready for text input.
|
||
- **Possible Purpose:** The likely intent is to enter a search query to find a specific location or point of interest on the map.
|
||
|
||
#### Step 2:
|
||
- **Action:** I type "cvs, pittsburgh" into the search bar.
|
||
- **Page Changes:** As I type, the search bar populates with the entered text. No immediate page changes occur until the search is executed.
|
||
- **Possible Purpose:** The purpose is to search for CVS locations in Pittsburgh, which will help in locating these points on the map.
|
||
|
||
#### Step 3:
|
||
- **Action:** I click the "Go" button next to the search bar.
|
||
- **Page Changes:** A list of search results appears on the left side of the screen under "Search Results." The map on the right side updates to show the general area of the first result.
|
||
- **Possible Purpose:** The action aims to execute the search query and display relevant results and their corresponding locations on the map.
|
||
|
||
#### Step 4:
|
||
- **Action:** I hover over the first search result, "Pharmacy CVS Pharmacy, 5600, Wilkins Avenue, Squirrel Hill, Squirrel Hill North, Pittsburgh, Allegheny County, 15217, United States."
|
||
- **Page Changes:** The map highlights the location of this specific CVS pharmacy with a red marker.
|
||
- **Possible Purpose:** The intent is to identify and focus on the exact location of this particular CVS pharmacy on the map.
|
||
|
||
#### Step 5:
|
||
- **Action:** I click on the first search result.
|
||
- **Page Changes:** The map centers on the selected CVS pharmacy location, and a popup box appears with detailed information about the node, including tags such as "addr:city," "addr:housenumber," "addr:street," "amenity," "brand," etc.
|
||
- **Possible Purpose:** The action is to view more detailed information about this specific CVS pharmacy, such as its address and attributes, which are useful for verification or further analysis.
|
||
|
||
### Summary
|
||
In this segment, I perform a series of actions to search for CVS pharmacies in Pittsburgh using OpenStreetMap. I begin by entering a query into the search bar, executing the search, and then selecting a specific result to view its detailed information on the map. Each step is methodically executed to locate and analyze the desired point of interest accurately.
|
||
|
||
---
|
||
|
||
## Part 5
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the text box labeled "From" in the directions panel.
|
||
- **Page Changes:** The text box becomes active, allowing for input or selection.
|
||
- **Possible Purpose:** The likely intent is to specify a starting location for the directions.
|
||
|
||
#### 2. **Action:** I type "CVS Pharmacy, 5600, Wilkins Avenue, Squirrel Hill" into the "From" text box.
|
||
- **Page Changes:** As I type, the system may provide autocomplete suggestions, but this is not visibly confirmed in the segment.
|
||
- **Possible Purpose:** The purpose is to set the starting point for the route to be calculated.
|
||
|
||
#### 3. **Action:** I click on the text box labeled "To" in the directions panel.
|
||
- **Page Changes:** The "To" text box becomes active, ready for input.
|
||
- **Possible Purpose:** The intent is to specify the destination location for the directions.
|
||
|
||
#### 4. **Action:** I type "Carnegie Mellon University, Schenley Drive" into the "To" text box.
|
||
- **Page Changes:** Similar to the "From" box, the "To" box now contains the typed destination. Again, autocomplete suggestions might appear but are not confirmed here.
|
||
- **Possible Purpose:** The purpose is to set the endpoint for the route.
|
||
|
||
#### 5. **Action:** I click the blue button labeled "Go" next to the direction fields.
|
||
- **Page Changes:** The map updates to display a route from the CVS Pharmacy to Carnegie Mellon University. The route is highlighted, and the map may zoom or pan to better display the path.
|
||
- **Possible Purpose:** The intent is to generate and visualize the directions between the specified start and end points.
|
||
|
||
### Summary
|
||
In this video segment, I sequentially interact with the directions feature on OpenStreetMap. I first specify a starting location ("From") and then a destination ("To"), followed by initiating the route calculation. The page responds by updating the map to show the calculated route, fulfilling the purpose of obtaining directions between the two specified locations.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_Find directions between two points",
|
||
"idx": 49,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Find directions between two points\")",
|
||
"before": {
|
||
"url": "about:blank"
|
||
},
|
||
"after": {
|
||
"url": "about:blank"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "textbox_From",
|
||
"idx": 51,
|
||
"action_repr": "frame.pressget_by_role(\"textbox\", name=\"From\")Enter",
|
||
"before": {
|
||
"url": "about:blank"
|
||
},
|
||
"after": {
|
||
"url": "about:blank"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "textbox_To",
|
||
"idx": 53,
|
||
"action_repr": "frame.pressget_by_role(\"textbox\", name=\"To\")Enter",
|
||
"before": {
|
||
"url": "about:blank"
|
||
},
|
||
"after": {
|
||
"url": "about:blank"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Close",
|
||
"idx": 11,
|
||
"action_repr": "frame.clicklocator(\"#sidebar form\").filter(has_text=\"Bicycle (OSRM)Car (OSRM)Foot (OSRM) Go Reverse Directions Loading...\").get_by_label(\"Close\")",
|
||
"before": {
|
||
"url": "about:blank"
|
||
},
|
||
"after": {
|
||
"url": "about:blank"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "textbox_Search",
|
||
"idx": 44,
|
||
"action_repr": "frame.pressget_by_role(\"textbox\", name=\"Search\")Enter",
|
||
"before": {
|
||
"url": "about:blank"
|
||
},
|
||
"after": {
|
||
"url": "about:blank"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "text_20 m50 ft\u00a9 OpenStreetMap contributors \u2665 Make a Donation. Website and API termsDi",
|
||
"idx": 46,
|
||
"action_repr": "frame.clickget_by_text(\"20 m50 ft\u00a9 OpenStreetMap contributors \u2665 Make a Donation. Website and API termsDi\")",
|
||
"before": {
|
||
"url": "about:blank"
|
||
},
|
||
"after": {
|
||
"url": "about:blank"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_CVS Pharmacy, 5600, Wilkins Avenue, Squirrel Hill, Squirrel Hill North, Pittsburgh, Allegheny County, 15217, United States",
|
||
"idx": 47,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"CVS Pharmacy, 5600, Wilkins Avenue, Squirrel Hill, Squirrel Hill North, Pittsburgh, Allegheny County, 15217, United States\")",
|
||
"before": {
|
||
"url": "about:blank"
|
||
},
|
||
"after": {
|
||
"url": "about:blank"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_48",
|
||
"idx": 48,
|
||
"action_repr": "frame.clicklocator(\"#map\")",
|
||
"before": {
|
||
"url": "about:blank"
|
||
},
|
||
"after": {
|
||
"url": "about:blank"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |