164 lines
7.5 KiB
Plaintext
164 lines
7.5 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
From my stay at Homewood Suites Southpointe, what's the estimated driving time to reach PPG Paints Arena?
|
||
|
||
# Annotation description
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the search textbox labeled "Search" at the top left corner of the page.
|
||
- **Page Changes:** The cursor is now active inside the search textbox, ready for input.
|
||
- **Possible Purpose:** The likely intent is to enter a location or keyword to search for a specific place on the map.
|
||
|
||
#### 2. **Action:** I type "Where is this?" into the search textbox.
|
||
- **Page Changes:** As I type, the text appears in the search textbox. However, no immediate results or suggestions are displayed until further action is taken.
|
||
- **Possible Purpose:** The purpose is to initiate a search query to find a location or information related to the entered text.
|
||
|
||
#### 3. **Action:** I click on the "Go" button next to the search textbox.
|
||
- **Page Changes:** After clicking "Go," the page processes the search query. A new interface element appears, showing navigation options with "From" and "To" textboxes.
|
||
- **Possible Purpose:** The intent is to execute the search and potentially find directions or details about the queried location.
|
||
|
||
#### 4. **Action:** I click on the "From" textbox in the newly appeared navigation interface.
|
||
- **Page Changes:** The "From" textbox becomes active, allowing for input. A dropdown or suggestion list might appear depending on the input.
|
||
- **Possible Purpose:** The goal is to specify the starting point for a route or direction.
|
||
|
||
#### 5. **Action:** I type "Homewood Suites, Southpointe, Cecil Towns" into the "From" textbox.
|
||
- **Page Changes:** As I type, suggestions or autocomplete options may appear below the textbox, helping to select the exact location.
|
||
- **Possible Purpose:** The intent is to accurately set the starting location for the navigation or route planning.
|
||
|
||
#### 6. **Action:** I click on the "To" textbox in the navigation interface.
|
||
- **Page Changes:** The "To" textbox becomes active, ready for input. Similar to the "From" textbox, it might show suggestions as I type.
|
||
- **Possible Purpose:** The goal is to specify the destination point for the route or direction.
|
||
|
||
#### 7. **Action:** I type "PPG Paints Arena, 1001, Fifth Avenue, Soho" into the "To" textbox.
|
||
- **Page Changes:** As I type, suggestions or autocomplete options may appear, aiding in selecting the correct destination.
|
||
- **Possible Purpose:** The intent is to accurately set the destination location for the navigation or route planning.
|
||
|
||
#### 8. **Action:** I click the "Go" button located near the navigation textboxes.
|
||
- **Page Changes:** The page processes the "From" and "To" locations and displays a route on the map. Directions are listed on the left side, detailing the steps to travel from the starting point to the destination.
|
||
- **Possible Purpose:** The goal is to generate and view the route and directions between the specified starting and ending points.
|
||
|
||
### Summary
|
||
In this video segment, I perform a series of actions to search for and generate directions between two specified locations using the OpenStreetMap website. Each step involves interacting with search and navigation elements to achieve the final result of viewing a mapped route and its corresponding directions.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_Find directions between two points",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Find directions between two points\")",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "textbox_From",
|
||
"idx": 2,
|
||
"action_repr": "frame.pressget_by_role(\"textbox\", name=\"From\")Enter",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/#map=7/42.896/-75.108"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "textbox_To",
|
||
"idx": 4,
|
||
"action_repr": "frame.pressget_by_role(\"textbox\", name=\"To\")Enter",
|
||
"before": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/directions"
|
||
},
|
||
"after": {
|
||
"url": "http://miniserver1875.asuscomm.com:3000/directions"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |