236 lines
10 KiB
Plaintext
236 lines
10 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
How much time does it take from Pittsburgh to Philadelphia by car?
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Part 1: Initial State and First Action
|
||
|
||
**Action:**
|
||
I click on the search bar located at the top left corner of the OpenStreetMap webpage.
|
||
|
||
**Page Changes:**
|
||
The search bar becomes active, allowing text input. A small dropdown menu appears below the search bar, suggesting recent or popular search terms.
|
||
|
||
**Possible Purpose:**
|
||
The likely intent is to initiate a search for a specific location on the map. Activating the search bar is the first step in entering a query.
|
||
|
||
---
|
||
|
||
### Part 2: Entering a Search Query
|
||
|
||
**Action:**
|
||
I type "Pittsburgh" into the active search bar.
|
||
|
||
**Page Changes:**
|
||
As I type, the dropdown menu updates dynamically, displaying search suggestions related to "Pittsburgh." These suggestions include various locations named Pittsburgh along with their respective details (e.g., Pittsburgh, Allegheny County, United States).
|
||
|
||
**Possible Purpose:**
|
||
The purpose is to find and select the correct "Pittsburgh" location from the suggestions to view its details on the map.
|
||
|
||
---
|
||
|
||
### Part 3: Selecting a Search Result
|
||
|
||
**Action:**
|
||
I click on the first suggestion in the dropdown menu, which reads "Pittsburgh, Allegheny County, United States."
|
||
|
||
**Page Changes:**
|
||
The map instantly centers on Pittsburgh, Allegheny County, United States. A sidebar on the left side of the screen appears, displaying detailed information about the selected location, including tags such as "admin_level," "boundary," and "name." The main map area highlights the city with a red marker and zooms in to provide a closer view.
|
||
|
||
**Possible Purpose:**
|
||
The intent is to confirm the selection of the correct Pittsburgh location and view its geographical details and associated data on the map.
|
||
|
||
---
|
||
|
||
### Part 4: Initiating Directions
|
||
|
||
**Action:**
|
||
I click on the "From" textbox located in the directions panel on the left side of the screen.
|
||
|
||
**Page Changes:**
|
||
The "From" textbox becomes active, ready for input. A small dropdown menu may appear, depending on previous entries or common starting points for directions.
|
||
|
||
**Possible Purpose:**
|
||
The likely intent is to enter a starting point for generating directions. By activating the "From" textbox, I prepare to specify where the route should begin, potentially to plan a journey originating from Pittsburgh.
|
||
|
||
---
|
||
|
||
This concludes the detailed step-by-step description of the actions observed in the provided video segment, strictly adhering to the first-person perspective and focusing solely on the actions within this part of the video.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the "Directions" button located at the top of the page.
|
||
- **Page Changes:** A directions panel appears on the left side of the screen, displaying input fields for "A" (start point) and "B" (destination).
|
||
- **Possible Purpose:** The likely intent is to initiate a route planning task by specifying start and end locations.
|
||
|
||
#### 2. **Action:** I click inside the "A" input field and type "Pittsburgh, Allegheny County, United States".
|
||
- **Page Changes:** As I type, a dropdown list appears with suggestions that match the entered text. After completing the entry, the map centers around Pittsburgh, and the location is marked as the starting point.
|
||
- **Possible Purpose:** The purpose is to set Pittsburgh as the starting location for the route.
|
||
|
||
#### 3. **Action:** I click inside the "B" input field and type "Philadelphia, Philadelphia County, United States".
|
||
- **Page Changes:** Similar to the previous action, a dropdown list with suggestions appears. Upon completion, the map updates to include Philadelphia as the destination, and a route is calculated between Pittsburgh and Philadelphia.
|
||
- **Possible Purpose:** The intent is to specify Philadelphia as the destination for the route.
|
||
|
||
#### 4. **Action:** I select "Car (OSRM)" from the transport mode dropdown menu below the destination field.
|
||
- **Page Changes:** The route on the map updates to reflect the optimal driving route from Pittsburgh to Philadelphia based on car travel.
|
||
- **Possible Purpose:** The selection ensures that the route is tailored for car travel, optimizing for roads and distances suitable for vehicles.
|
||
|
||
#### 5. **Action:** I click the "Reverse Directions" link located below the transport mode dropdown.
|
||
- **Page Changes:** The start and destination points are swapped, and the map updates to show the route from Philadelphia to Pittsburgh.
|
||
- **Possible Purpose:** The action aims to reverse the direction of the trip, possibly to explore the route in the opposite direction or to compare travel details.
|
||
|
||
#### 6. **Action:** I hover over the map near the area around Pittsburgh.
|
||
- **Page Changes:** No immediate changes occur on the page, but the cursor changes to indicate interaction options (e.g., zoom or pan).
|
||
- **Possible Purpose:** The hovering might be to prepare for further interaction with the map, such as zooming into a specific area or examining details of the route near Pittsburgh.
|
||
|
||
### Conclusion
|
||
The described actions focus on setting up and modifying a driving route between Pittsburgh and Philadelphia using OpenStreetMap's directions feature. Each step is methodically executed to configure the route according to specific parameters and then to reverse the direction for an alternative perspective.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "textbox_Search",
|
||
"idx": 1,
|
||
"action_repr": "frame.pressget_by_role(\"textbox\", name=\"Search\")Enter",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Pittsburgh, Allegheny County, United States",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Pittsburgh, Allegheny County, United States\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Find directions between two points",
|
||
"idx": 6,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Find directions between two points\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/relation/188553"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/relation/188553"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "textbox_From",
|
||
"idx": 8,
|
||
"action_repr": "frame.pressget_by_role(\"textbox\", name=\"From\")Tab",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/directions#map=12/40.4314/-80.0406"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/directions#map=12/40.4314/-80.0406"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Go",
|
||
"idx": 9,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Go\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/directions#map=12/40.4314/-80.0406"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/directions#map=12/40.4314/-80.0406"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |