421 lines
22 KiB
Plaintext
421 lines
22 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
What is the duration required to first walk from Massachusetts Institute of Technology to Harvard University, and then drive to Boston Logan International Airport?
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Part 1: Initial State and First Action
|
||
|
||
**Action:**
|
||
I click on the search textbox located at the top-left corner of the webpage, which is labeled "Search" and has a placeholder text "Where is this?".
|
||
|
||
**Page Changes:**
|
||
The cursor focuses inside the search textbox, indicating it is ready for input.
|
||
|
||
**Possible Purpose:**
|
||
The likely intent is to enter a location or point of interest into the search field to find and view its details on the map.
|
||
|
||
---
|
||
|
||
### Second Action
|
||
|
||
**Action:**
|
||
I type "Massachusetts Institute of Technology" into the search textbox.
|
||
|
||
**Page Changes:**
|
||
As I type, a dropdown list appears below the search textbox, displaying search suggestions related to the entered text.
|
||
|
||
**Possible Purpose:**
|
||
The purpose is to use the autocomplete feature to quickly select the correct entry for "Massachusetts Institute of Technology" from the suggestions.
|
||
|
||
---
|
||
|
||
### Third Action
|
||
|
||
**Action:**
|
||
I press the "Enter" key or click the "Go" button next to the search textbox.
|
||
|
||
**Page Changes:**
|
||
The webpage transitions to display search results for "Massachusetts Institute of Technology." The map on the right side of the page zooms in to the location of MIT, highlighting the area with an orange outline. On the left side, a panel appears with detailed information about the selected location, including its name, version number, and tags.
|
||
|
||
**Possible Purpose:**
|
||
The intent is to confirm the search query and view the specific geographic data and details associated with the Massachusetts Institute of Technology on OpenStreetMap.
|
||
|
||
---
|
||
|
||
### Fourth Action
|
||
|
||
**Action:**
|
||
I scroll down slightly within the information panel on the left side of the page.
|
||
|
||
**Page Changes:**
|
||
The content within the information panel shifts upward, revealing additional details such as tags and version history related to the "Massachusetts Institute of Technology" entry.
|
||
|
||
**Possible Purpose:**
|
||
The purpose is to review more detailed metadata and historical information about the mapped location, which may include attributes like alternative names, administrative codes, and edit history.
|
||
|
||
---
|
||
|
||
### Summary of Observed Actions:
|
||
1. I clicked on the search textbox and focused it.
|
||
2. I typed "Massachusetts Institute of Technology" into the search textbox.
|
||
3. I submitted the search query by pressing "Enter" or clicking "Go," resulting in the map zooming to MIT's location and displaying relevant details.
|
||
4. I scrolled down in the information panel to view additional metadata about the location.
|
||
|
||
Each action is performed with the clear intent of locating and examining detailed geographic information for the specified institution on OpenStreetMap.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the text box labeled "To" in the directions panel.
|
||
- **Page Changes:** The text box becomes active, allowing me to input a destination.
|
||
- **Possible Purpose:** My intention is to specify a destination for the route directions.
|
||
|
||
#### 2. **Action:** I type "Harvard University" into the active text box.
|
||
- **Page Changes:** As I type, suggestions appear below the text box, listing possible matches for "Harvard University."
|
||
- **Possible Purpose:** I am selecting "Harvard University" as my destination to get directions from the starting point (Massachusetts Institute of Technology) to this location.
|
||
|
||
#### 3. **Action:** I select "Harvard University" from the suggestion list.
|
||
- **Page Changes:** The text box now displays "Harvard University" as the selected destination. The map and directions panel begin to update to show the route from MIT to Harvard University.
|
||
- **Possible Purpose:** By selecting the suggested entry, I confirm "Harvard University" as the exact destination for the route.
|
||
|
||
#### 4. **Action:** I click the "Go" button next to the destination text box.
|
||
- **Page Changes:** The map updates to display the calculated route from Massachusetts Institute of Technology to Harvard University. The route is highlighted on the map, and the directions panel shows step-by-step instructions for the journey.
|
||
- **Possible Purpose:** I initiate the process to generate and display the route directions between the two specified locations.
|
||
|
||
### Summary
|
||
In this video segment, I interact with the OpenStreetMap directions feature by specifying a destination ("Harvard University") and generating route directions from the starting point ("Massachusetts Institute of Technology"). Each action is focused on setting up and confirming the destination to obtain the desired route information.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the "Foot (OSRM)" dropdown menu.
|
||
- **Page Changes:** The dropdown menu expands, revealing options for different modes of transportation such as "Car (OSRM)," "Bicycle (OSRM)," and "Foot (OSRM)."
|
||
- **Possible Purpose:** The likely intent is to select a mode of transportation for the directions. In this case, it appears that "Foot (OSRM)" is already selected, indicating a preference for walking directions.
|
||
|
||
#### 2. **Action:** I click the "Go" button.
|
||
- **Page Changes:** After clicking "Go," the map updates to display a route between the two specified locations: "Massachusetts Institute of Technology, Allston Street, Cambridge, MA" and "Harvard University, Hurlbut Street, Avon Hill, Cambridge, MA." The route is highlighted in blue, and a list of step-by-step directions appears on the left side of the screen.
|
||
- **Possible Purpose:** The purpose of this action is to generate and display walking directions from the starting point to the destination. The updated map and directions provide a visual and textual guide for the journey.
|
||
|
||
#### 3. **Action:** I scroll through the step-by-step directions.
|
||
- **Page Changes:** As I scroll, the list of directions moves up and down, allowing me to view each step of the route. The map remains static during this action.
|
||
- **Possible Purpose:** The intent is to review the detailed instructions for the walking route. This helps in understanding the path and any specific turns or landmarks along the way.
|
||
|
||
#### 4. **Action:** I hover over the map near the endpoint of the route.
|
||
- **Page Changes:** Hovering over the map does not cause any immediate changes, but it may be preparing for a more detailed interaction, such as zooming in or selecting a specific part of the route.
|
||
- **Possible Purpose:** The likely purpose is to get a closer look at a particular section of the route or to prepare for further interaction with the map, such as adjusting the view or examining nearby points of interest.
|
||
|
||
### Summary
|
||
In this video segment, I interact with the OpenStreetMap website to obtain and review walking directions between two specified locations. I select the mode of transportation, generate the route, and then examine the provided directions and map visualization. Each action is focused on understanding and preparing for the walking journey from start to finish.
|
||
|
||
---
|
||
|
||
## Part 4
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action**: I click on the "To" textbox in the directions panel.
|
||
- **Page Changes**: The cursor is now active inside the "To" textbox, indicating it is ready for input.
|
||
- **Possible Purpose**: The likely intent is to change or specify the destination address for the directions.
|
||
|
||
#### 2. **Action**: I type "Harvard University, Dunster Street, Old Cam" into the "To" textbox.
|
||
- **Page Changes**: As I type, a dropdown suggestion appears below the textbox with possible matches for the entered text.
|
||
- **Possible Purpose**: The purpose is to select a specific location at Harvard University as the destination for the directions.
|
||
|
||
#### 3. **Action**: I click on the suggested option "Harvard University, Dunster Street, Old Cambridge" from the dropdown.
|
||
- **Page Changes**: The selected suggestion fills the "To" textbox, and the map updates to show the new route from the starting point to this destination.
|
||
- **Possible Purpose**: This action confirms the desired destination, allowing the system to calculate and display the updated route.
|
||
|
||
#### 4. **Action**: I click the "Go" button next to the "To" textbox.
|
||
- **Page Changes**: The directions panel updates with a new set of step-by-step directions, and the map highlights the calculated route.
|
||
- **Possible Purpose**: The intent is to generate and view the detailed directions from the starting point to the newly specified destination.
|
||
|
||
#### 5. **Action**: I observe the updated directions and map.
|
||
- **Page Changes**: No further interaction occurs; the focus is on reviewing the provided directions and visual route.
|
||
- **Possible Purpose**: The purpose is to review the generated directions and ensure they meet the requirements or expectations for the journey.
|
||
|
||
### Summary
|
||
In this video segment, I interact with the directions feature on the OpenStreetMap website by specifying a new destination ("Harvard University, Dunster Street, Old Cambridge") and generating the corresponding route. Each step involves selecting and confirming the destination, which results in updated directions and a visual route on the map. The final action involves reviewing the provided information.
|
||
|
||
---
|
||
|
||
## Part 5
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action**: I click on the text box for the destination location.
|
||
- **Page Changes**: The text box becomes active, allowing me to input or modify the destination.
|
||
- **Possible Purpose**: My intention is to change or confirm the destination for the route.
|
||
|
||
#### 2. **Action**: I type "Boston Logan International" into the destination text box.
|
||
- **Page Changes**: As I type, a dropdown menu appears with suggestions that match my input. The map updates slightly to reflect the new destination query.
|
||
- **Possible Purpose**: I am specifying a new destination to get directions from the starting point (Harvard University) to Boston Logan International Airport.
|
||
|
||
#### 3. **Action**: I select "Boston Logan International Airport, 1, Harborside Dr" from the dropdown menu.
|
||
- **Page Changes**: The selected destination is confirmed in the text box, and the map recalculates the route based on this new destination.
|
||
- **Possible Purpose**: By selecting the specific airport location, I ensure accurate routing to the correct address.
|
||
|
||
#### 4. **Action**: I click on the dropdown menu for the travel mode.
|
||
- **Page Changes**: The dropdown menu expands, showing options such as "Foot (OSRM)," "Bicycle (OSRM)," and "Car (OSRM)."
|
||
- **Possible Purpose**: I am considering changing the mode of transportation for the route.
|
||
|
||
#### 5. **Action**: I select "Car (OSRM)" from the dropdown menu.
|
||
- **Page Changes**: The map recalculates the route based on car travel, and the directions panel updates with the new estimated distance and time for driving.
|
||
- **Possible Purpose**: I have decided to view the route and directions for traveling by car instead of walking.
|
||
|
||
#### 6. **Action**: I click the "Go" button.
|
||
- **Page Changes**: The map fully renders the new route from Harvard University to Boston Logan International Airport by car. The directions panel now displays detailed step-by-step instructions for the car journey, including distance (14Km) and estimated time (0:19).
|
||
- **Possible Purpose**: I initiate the route calculation to obtain the final directions for the car trip.
|
||
|
||
### Summary
|
||
In this video segment, I update the destination to Boston Logan International Airport and switch the travel mode to car. After confirming these selections, I generate the route to receive detailed driving directions from Harvard University to the airport. Each action is focused on obtaining accurate and relevant travel information for a car journey.
|
||
|
||
---
|
||
|
||
## Part 6
|
||
### Part 1: Observing the Actions in the Provided Video Segment
|
||
|
||
#### Step 1:
|
||
- **Action:** I click on the "locator" text box located at the top of the left sidebar.
|
||
- **Page Changes:** The text box becomes active, allowing for input.
|
||
- **Possible Purpose:** The likely intent is to enter a starting location for directions.
|
||
|
||
#### Step 2:
|
||
- **Action:** I type "Harvard University, Dunster Street, Old Cambridge" into the active locator text box.
|
||
- **Page Changes:** As I type, suggestions appear below the text box, offering autocomplete options based on the input.
|
||
- **Possible Purpose:** The purpose is to specify a precise starting point for the route.
|
||
|
||
#### Step 3:
|
||
- **Action:** I select the suggested option that matches "Harvard University, Dunster Street, Old Cambridge" from the dropdown list.
|
||
- **Page Changes:** The map updates to center on the selected location, and a green marker appears at Harvard University.
|
||
- **Possible Purpose:** This action confirms the starting point for the directions.
|
||
|
||
#### Step 4:
|
||
- **Action:** I click on the second locator text box (destination field) below the first one.
|
||
- **Page Changes:** The second text box becomes active, ready for input.
|
||
- **Possible Purpose:** The intent is to enter a destination for the route.
|
||
|
||
#### Step 5:
|
||
- **Action:** I type "Logan International Airport, 1, Harborside Dr" into the active destination text box.
|
||
- **Page Changes:** Similar to the previous step, suggestions appear as I type, offering autocomplete options.
|
||
- **Possible Purpose:** The purpose is to specify the destination point for the route.
|
||
|
||
#### Step 6:
|
||
- **Action:** I select the suggested option that matches "Logan International Airport, 1, Harborside Dr" from the dropdown list.
|
||
- **Page Changes:** The map updates again, now showing both the starting point (green marker) and the destination (red marker). A route is calculated and displayed on the map.
|
||
- **Possible Purpose:** This action confirms the destination and triggers the generation of directions between the two points.
|
||
|
||
#### Step 7:
|
||
- **Action:** I click the "Go" button located next to the destination field.
|
||
- **Page Changes:** The directions panel on the left side of the screen populates with a detailed list of step-by-step directions from the starting point to the destination.
|
||
- **Possible Purpose:** The intent is to finalize the route calculation and view the complete set of directions.
|
||
|
||
#### Step 8:
|
||
- **Action:** I scroll through the directions panel to review the listed steps.
|
||
- **Page Changes:** The directions remain static as I scroll, but the map may highlight segments of the route corresponding to the steps as I move through the list.
|
||
- **Possible Purpose:** The purpose is to read and understand the provided directions for the journey.
|
||
|
||
### Summary
|
||
In this video segment, I perform a series of actions to generate and review directions from Harvard University to Logan International Airport using the OpenStreetMap website. Each step involves interacting with the UI elements to specify locations, trigger route calculation, and examine the resulting directions. The observed actions are methodical and focused on obtaining clear navigation instructions.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "textbox_Search",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_role(\"textbox\", name=\"Search\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Go",
|
||
"idx": 3,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Go\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Massachusetts Institute of Technology, Allston Street, Cambridgeport, Cambridge, Middlesex County, Massachusetts, 02139, United States",
|
||
"idx": 4,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Massachusetts Institute of Technology, Allston Street, Cambridgeport, Cambridge, Middlesex County, Massachusetts, 02139, United States\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/relation/65066"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/relation/65066"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_5",
|
||
"idx": 5,
|
||
"action_repr": "frame.clicklocator(\"#map\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/relation/65066"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/relation/65066"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_6",
|
||
"idx": 6,
|
||
"action_repr": "frame.clicklocator(\"#map\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/relation/65066"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/relation/65066"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Directions from here",
|
||
"idx": 20,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Directions from here\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/relation/65066"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/relation/65066"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "textbox_To",
|
||
"idx": 25,
|
||
"action_repr": "frame.clickget_by_role(\"textbox\", name=\"To\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/relation/65066"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/relation/65066"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_18",
|
||
"idx": 18,
|
||
"action_repr": "frame.selectOptionget_by_role(\"combobox\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/relation/65066"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/relation/65066"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Marker",
|
||
"idx": 19,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Marker\").nth(1)",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/relation/65066"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/relation/65066"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_26",
|
||
"idx": 26,
|
||
"action_repr": "frame.selectOptionget_by_role(\"combobox\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/directions?engine=fossgis_osrm_car&route=42.3679%2C-71.1268%3B42.3632%2C-71.0136"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/directions?engine=fossgis_osrm_car&route=42.3679%2C-71.1268%3B42.3632%2C-71.0136"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |