294 lines
14 KiB
Plaintext
294 lines
14 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Post a notice on a virtual meetup for Harry Potter enthusiasts on July 8th in the books subreddit
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
1. **Action:** I click on the search bar located at the top center of the webpage.
|
||
- **Page Changes:** The search bar becomes active, and a blinking cursor appears inside it, indicating that it is ready for text input.
|
||
- **Possible Purpose:** The likely intent is to initiate a search for specific content within the website.
|
||
|
||
2. **Action:** I type the word "books" into the active search bar.
|
||
- **Page Changes:** As I type, the word "books" appears in the search bar. There are no immediate page changes until the search is executed.
|
||
- **Possible Purpose:** The purpose is to find posts, threads, or discussions related to "books" within the subscribed forums or the entire site.
|
||
|
||
3. **Action:** I press the Enter key on the keyboard to execute the search.
|
||
- **Page Changes:** The webpage transitions to display search results. The results include various posts where the term "books" appears, such as titles, comments, or content descriptions. The layout now shows a list of these relevant entries.
|
||
- **Possible Purpose:** The intent is to view and possibly interact with the search results related to "books."
|
||
|
||
4. **Action:** I scroll down slightly to view more of the search results.
|
||
- **Page Changes:** Additional search results become visible as I scroll, revealing more posts containing the term "books."
|
||
- **Possible Purpose:** The purpose is to explore a broader range of results to find specific information or discussions of interest related to "books."
|
||
|
||
These actions are strictly based on the observable interactions within this video segment.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action:** I click on the "Search" button located below the search query box.
|
||
- **Page Changes:** The webpage transitions to display a list of search results related to the query "books." The results are titled "50 results for books:" and include various posts containing the word "books."
|
||
- **Possible Purpose:** The likely intent is to execute the search query and view all relevant results pertaining to "books."
|
||
|
||
2. **Action:** I scroll down the page.
|
||
- **Page Changes:** The visible portion of the webpage shifts downward, revealing more search results that were initially out of view.
|
||
- **Possible Purpose:** The purpose is to review additional search results that extend beyond the initial viewport.
|
||
|
||
3. **Action:** I click on the "Forums" tab at the top of the page.
|
||
- **Page Changes:** The content of the webpage changes to display a "List of forums" with various forum categories such as "AskReddit," "relationship_advice," "worldnews," etc. Each forum entry includes a brief description and the number of submissions.
|
||
- **Possible Purpose:** The intent is to navigate away from the search results and explore different forum categories available on the website.
|
||
|
||
4. **Action:** I click on the "Search" text box.
|
||
- **Page Changes:** The text box becomes active, allowing for text input or modification.
|
||
- **Possible Purpose:** The likely intent is to either refine the current search query or enter a new one.
|
||
|
||
5. **Action:** I type "books" into the "Search" text box.
|
||
- **Page Changes:** The text "books" appears in the search box, but no immediate page changes occur until a search is executed.
|
||
- **Possible Purpose:** The purpose is to prepare for a new search or to modify the existing search query.
|
||
|
||
6. **Action:** I click on the "Search" button again.
|
||
- **Page Changes:** The webpage updates to show a refreshed list of search results for the query "books," similar to the initial search but potentially updated or refined.
|
||
- **Possible Purpose:** The intent is to execute the search with the newly typed or confirmed query "books" and view the corresponding results.
|
||
|
||
7. **Action:** I scroll down the page once more.
|
||
- **Page Changes:** The visible area of the webpage moves downward, displaying additional search results that were not initially visible.
|
||
- **Possible Purpose:** The purpose is to continue reviewing more search results that are further down the list.
|
||
|
||
### Summary:
|
||
In this video segment, I perform a series of actions focused on searching for and exploring content related to "books." I initiate searches, navigate through search results, switch to the forums section, and refine my search query. Each action is aimed at gathering and reviewing information about "books" within the context of the website's features.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action:** I click on the "Title" text box.
|
||
- **Page Changes:** The cursor appears inside the "Title" text box, indicating it is active and ready for input.
|
||
- **Possible Purpose:** The likely intent is to enter a title for the submission.
|
||
|
||
2. **Action:** I type "Notice: Virtual Meetup for Harry Potter Enthusiasts!" into the "Title" text box.
|
||
- **Page Changes:** The text "Notice: Virtual Meetup for Harry Potter Enthusiasts!" appears in the "Title" text box.
|
||
- **Possible Purpose:** The purpose is to provide a clear and descriptive title for the submission, informing readers about the content of the post.
|
||
|
||
3. **Action:** I click on the "Body" text box.
|
||
- **Page Changes:** The cursor appears inside the "Body" text box, indicating it is active and ready for input.
|
||
- **Possible Purpose:** The intent is to enter the main content or details of the submission.
|
||
|
||
4. **Action:** I type "Hi Everyone," into the "Body" text box.
|
||
- **Page Changes:** The text "Hi Everyone," appears in the "Body" text box.
|
||
- **Possible Purpose:** The purpose is to start the body of the submission with a greeting to engage the audience.
|
||
|
||
5. **Action:** I press the Enter key.
|
||
- **Page Changes:** A new line is created in the "Body" text box.
|
||
- **Possible Purpose:** The intent is to create a new paragraph or separate the greeting from the following text for better readability.
|
||
|
||
6. **Action:** I type "Kindly notice that there will be a virtual mee" into the "Body" text box.
|
||
- **Page Changes:** The text "Kindly notice that there will be a virtual mee" appears in the "Body" text box, continuing from the previous line.
|
||
- **Possible Purpose:** The purpose is to provide information about the virtual meetup mentioned in the title.
|
||
|
||
### Summary:
|
||
In this video segment, I focused on creating a submission by entering a title and beginning the body text. The actions involved activating the "Title" and "Body" text boxes, typing the respective content, and formatting the text for clarity and engagement. The overall purpose appears to be to draft a notice about a virtual meetup for a specific interest group.
|
||
|
||
---
|
||
|
||
## Part 4
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click inside the "Body" text box.
|
||
- **Page Changes:** The cursor appears inside the "Body" text box, indicating it is active and ready for text input.
|
||
- **Possible Purpose:** To begin typing or editing the content of the post.
|
||
|
||
#### 2. **Action:** I type the text "Hi Everyone," into the "Body" text box.
|
||
- **Page Changes:** The text "Hi Everyone," appears in the "Body" text box.
|
||
- **Possible Purpose:** To start the message with a greeting to the audience.
|
||
|
||
#### 3. **Action:** I press the Enter key after typing "Hi Everyone,".
|
||
- **Page Changes:** A new line is created in the "Body" text box.
|
||
- **Possible Purpose:** To create a paragraph break for better formatting of the message.
|
||
|
||
#### 4. **Action:** I type "Kindly notice that there will be a virtual meetup for Harry Potter Enthusiasts on July 8th!" into the "Body" text box.
|
||
- **Page Changes:** The typed text appears on the new line in the "Body" text box.
|
||
- **Possible Purpose:** To provide specific information about an upcoming event.
|
||
|
||
#### 5. **Action:** I press the Enter key after typing the event details.
|
||
- **Page Changes:** Another new line is created in the "Body" text box.
|
||
- **Possible Purpose:** To add another paragraph break for additional information or a closing statement.
|
||
|
||
#### 6. **Action:** I type "Everyone is welcomed!" into the "Body" text box.
|
||
- **Page Changes:** The text "Everyone is welcomed!" appears on the new line in the "Body" text box.
|
||
- **Possible Purpose:** To extend an inclusive invitation to all potential attendees.
|
||
|
||
#### 7. **Action:** I move the cursor over the "Create submission" button.
|
||
- **Page Changes:** The button highlights, indicating it is hovered over.
|
||
- **Possible Purpose:** To prepare for submitting the post after reviewing the content.
|
||
|
||
### Summary
|
||
In this video segment, I interact with the "Body" text box by typing a structured message that includes a greeting, event details, and an inclusive invitation. Each step involves typing text and using the Enter key for formatting. Finally, I hover over the "Create submission" button, likely preparing to submit the post. The actions are focused on composing and formatting the message content before submission.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "label_Search query",
|
||
"idx": 4,
|
||
"action_repr": "frame.pressget_by_label(\"Search query\")Enter",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Forums",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Forums\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_5",
|
||
"idx": 5,
|
||
"action_repr": "frame.clicklocator(\"span\").filter(has_text=\"Submitted by SAT0725 t3_10p12wq 8 months ago in books\").get_by_role(\"link\").nth(1)",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/f/books"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/f/books"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Submit",
|
||
"idx": 6,
|
||
"action_repr": "frame.clickget_by_label(\"Submit\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/submit/books"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/f/books"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Title *",
|
||
"idx": 7,
|
||
"action_repr": "frame.clickget_by_label(\"Title *\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/submit/books"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/submit/books"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Body",
|
||
"idx": 8,
|
||
"action_repr": "frame.clickget_by_label(\"Body\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/submit/books"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/submit/books"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Create submission",
|
||
"idx": 9,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Create submission\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/submit/books"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/submit/books"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |