262 lines
12 KiB
Plaintext
262 lines
12 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Thumbs down the top 1 post ever in gadgets.
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Part 1: Observing and Documenting Actions
|
||
|
||
#### Action 1:
|
||
- **Action:** I click on the "Forums" tab located in the navigation bar at the top of the webpage.
|
||
- **Page Changes:** The page transitions to display a list of various forums categorized alphabetically from A to Z. Each forum is listed with its name, a brief description (if available), and a link to access it.
|
||
- **Possible Purpose:** The likely intent is to navigate to a section where different discussion forums are organized and easily accessible, allowing for exploration or selection of a specific forum of interest.
|
||
|
||
#### Action 2:
|
||
- **Action:** I scroll down the page to view more forums in the list.
|
||
- **Page Changes:** As I scroll, additional forums become visible, expanding the range of options available for selection. The forums continue to be listed alphabetically, ensuring an organized presentation.
|
||
- **Possible Purpose:** The purpose of scrolling is to browse through a wider array of forums, potentially searching for a particular forum or simply exploring the available options to find one that aligns with my interests or needs.
|
||
|
||
#### Action 3:
|
||
- **Action:** I click on the "Alphabetical" tab located near the top of the forum list.
|
||
- **Page Changes:** The forum list remains displayed in alphabetical order, confirming that this is the default or already selected sorting method. No significant changes occur as the list was already sorted alphabetically.
|
||
- **Possible Purpose:** This action might be intended to ensure that the forums are indeed sorted alphabetically, providing a structured way to locate a specific forum quickly. It could also be a verification step to confirm the current sorting method.
|
||
|
||
#### Action 4:
|
||
- **Action:** I click on the "MachineLearning" link within the forum list.
|
||
- **Page Changes:** The page transitions to the "MachineLearning" forum, displaying a list of recent submissions and comments related to machine learning topics. The interface includes options to filter submissions by criteria such as "Hot," "New," "Top," etc.
|
||
- **Possible Purpose:** The intent is to enter the "MachineLearning" forum to engage with content specific to machine learning. This could involve reading discussions, participating in conversations, or finding resources related to this field.
|
||
|
||
### Summary of Observed Actions:
|
||
1. Navigated to the "Forums" section to view a list of available forums.
|
||
2. Scrolled through the forum list to explore more options.
|
||
3. Confirmed the alphabetical sorting of the forum list.
|
||
4. Entered the "MachineLearning" forum to access and interact with relevant content.
|
||
|
||
Each action is documented based solely on the observable interactions and resulting page changes, without additional context or assumptions.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Initial Observation**
|
||
- **Action:** I begin by observing the current webpage.
|
||
- **Page Changes:** No changes occur as this is the initial state.
|
||
- **Possible Purpose:** To familiarize myself with the layout and content of the page before interacting with it.
|
||
|
||
#### 2. **Hover Over "Submissions" Tab**
|
||
- **Action:** I move my cursor over the "Submissions" tab located under the "/f/gadgets" header.
|
||
- **Page Changes:** The cursor changes to a hand icon, indicating that the tab is clickable.
|
||
- **Possible Purpose:** To prepare for selecting the "Submissions" tab, possibly to view all submissions within this forum.
|
||
|
||
#### 3. **Click on "Submissions" Tab**
|
||
- **Action:** I click on the "Submissions" tab.
|
||
- **Page Changes:** The page refreshes to display a list of submissions under the "/f/gadgets" forum. The "Submissions" tab appears highlighted, confirming it is the active view.
|
||
- **Possible Purpose:** To view all the posts or threads submitted to this specific forum.
|
||
|
||
#### 4. **Scroll Down the Page**
|
||
- **Action:** I scroll down the webpage.
|
||
- **Page Changes:** Additional submissions become visible as I scroll, revealing more content from the "/f/gadgets" forum.
|
||
- **Possible Purpose:** To explore more submissions and gather a broader understanding of the forum's content.
|
||
|
||
#### 5. **Hover Over a Submission Title**
|
||
- **Action:** I move my cursor over a specific submission titled "A custom gaming PC built inside a vintage 1940s Motorola radio case."
|
||
- **Page Changes:** The cursor changes to a hand icon, and a preview or tooltip might appear (though not explicitly shown in the image).
|
||
- **Possible Purpose:** To prepare for clicking on the submission title, likely to read the full post and any associated comments.
|
||
|
||
#### 6. **Click on the Submission Title**
|
||
- **Action:** I click on the submission titled "A custom gaming PC built inside a vintage 1940s Motorola radio case."
|
||
- **Page Changes:** The page transitions to a detailed view of the selected submission. This new page likely includes the full text of the post, any images or links included, and a comment section.
|
||
- **Possible Purpose:** To access the detailed content of the submission, including reading the post and potentially engaging with comments or replies.
|
||
|
||
### Conclusion
|
||
The described actions reflect a sequence of interactions aimed at navigating and exploring the "/f/gadgets" forum, specifically focusing on viewing submissions and accessing detailed content within a selected post. Each step is executed with the intent of gathering information and engaging with the forum's content.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_Forums",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Forums\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_1",
|
||
"idx": 1,
|
||
"action_repr": "frame.clicklocator(\"#main\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/forums"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/f/worldnews"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Alphabetical",
|
||
"idx": 3,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Alphabetical\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/forums"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/forums"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_gadgets",
|
||
"idx": 4,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"gadgets\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/forums/all"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/forums/all"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Sort by: Hot",
|
||
"idx": 5,
|
||
"action_repr": "frame.clickget_by_label(\"Sort by: Hot\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/f/gadgets"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/f/gadgets"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Top",
|
||
"idx": 6,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Top\", exact=True)",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/f/gadgets/top?t=day"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/f/gadgets/top?t=day"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_From: Past 24 hours",
|
||
"idx": 7,
|
||
"action_repr": "frame.clickget_by_label(\"From: Past 24 hours\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/f/gadgets/top?t=day"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/f/gadgets/top?t=day"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_All time",
|
||
"idx": 8,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"All time\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/f/gadgets/top?t=day"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/f/gadgets/top?t=day"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_9",
|
||
"idx": 9,
|
||
"action_repr": "frame.clicklocator(\"form\").filter(has_text=\"11533\").get_by_title(\"Downvote\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/f/gadgets/top?t=all"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/f/gadgets/top?t=all"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |