217 lines
11 KiB
Plaintext
217 lines
11 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Post in books subreddit about what could machine learning help the corresponding field.
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
In this video segment, there are no observable actions being performed. The screen displays a static list of forums with their respective names, submission counts, and "Subscribe" buttons. Since no interactions such as clicks, scrolls, or selections are visible, the description is limited to the content shown:
|
||
|
||
- **Action**: No action is taken in this segment.
|
||
- **Page Changes**: There are no changes to the webpage as no actions are performed.
|
||
- **Possible Purpose**: The purpose of this segment appears to be to display the list of forums for viewing. Without any interaction, the intent could be to familiarize the viewer with the available forums and their current activity levels (submission counts).
|
||
|
||
Since no further actions are observed, the description concludes here, strictly adhering to the provided visual information.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the "books" link in the list of forums.
|
||
- **Page Changes:** The webpage transitions to a new page titled "/r/books," which appears to be a subreddit dedicated to books. This page displays a post by a user named Barry Lyga, who introduces himself as an author and offers to answer questions.
|
||
- **Possible Purpose:** The likely intent is to navigate to the "books" subreddit to explore content related to books, possibly to read discussions, participate in conversations, or find specific information.
|
||
|
||
#### 2. **Action:** I scroll down slightly on the "/r/books" page.
|
||
- **Page Changes:** The view shifts downward, revealing more content on the page, including additional posts and comments below the initial post by Barry Lyga.
|
||
- **Possible Purpose:** The purpose of scrolling is to review more content within the "books" subreddit, gaining a broader understanding of the discussions and topics covered in this community.
|
||
|
||
#### 3. **Action:** I click on the "+ Submit" button located at the top-right corner of the page.
|
||
- **Page Changes:** A new page titled "Create submission" opens. This page includes fields for creating a new post, such as "URL," "Title," "Body," and "Forum," with the "Forum" field pre-selected as "books."
|
||
- **Possible Purpose:** The intent is to start the process of creating a new post within the "books" subreddit. This action suggests a desire to contribute content, share information, or initiate a discussion relevant to the subreddit's theme.
|
||
|
||
#### 4. **Action:** I click inside the "Title" text box.
|
||
- **Page Changes:** The cursor appears inside the "Title" text box, indicating that it is now active and ready for text input.
|
||
- **Possible Purpose:** The purpose is to begin typing the title of the new post. This step is necessary to provide a clear and descriptive title that summarizes the content of the post for other users.
|
||
|
||
#### 5. **Action:** I type a title into the "Title" text box.
|
||
- **Page Changes:** The text I type appears in the "Title" field. The exact text typed is not specified, but the field is visibly populated with my input.
|
||
- **Possible Purpose:** The intent is to define the title of the post, which will serve as a headline and attract the attention of other subreddit users. A well-crafted title is crucial for engaging the audience and conveying the post's main topic.
|
||
|
||
#### 6. **Action:** I click inside the "Body" text box.
|
||
- **Page Changes:** The cursor moves to the "Body" text box, making it active for text input.
|
||
- **Possible Purpose:** The purpose is to prepare for writing the main content of the post. The "Body" section allows for detailed explanations, stories, questions, or any other information the user wishes to share with the community.
|
||
|
||
#### 7. **Action:** I begin typing in the "Body" text box.
|
||
- **Page Changes:** As I type, the text appears in the "Body" field. The specific content being typed is not detailed, but the field shows ongoing input.
|
||
- **Possible Purpose:** The intent is to write the substantive part of the post. This section is where the user elaborates on the topic introduced in the title, providing context, details, or engaging content for the subreddit audience.
|
||
|
||
### Summary
|
||
In this video segment, I navigate to the "books" subreddit, explore its content briefly, and then proceed to create a new post. My actions include selecting the subreddit, initiating a new submission, entering a title, and beginning to write the body of the post. Each step is methodical, aimed at contributing meaningful content to the community.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
In this video segment, I perform the following actions:
|
||
|
||
1. **Action**: I click inside the "Title" text box.
|
||
- **Page Changes**: The cursor appears in the "Title" text box, indicating it is active and ready for input.
|
||
- **Possible Purpose**: My intention is to enter a title for the submission.
|
||
|
||
2. **Action**: I type the text "What could machine learning help in the correpong field?" into the "Title" text box.
|
||
- **Page Changes**: The text appears in the "Title" text box as I type.
|
||
- **Possible Purpose**: I am providing a specific title for the submission, likely to describe the topic or question I want to discuss.
|
||
|
||
3. **Action**: I move the cursor to the "Body" text box area but do not click or type anything.
|
||
- **Page Changes**: There are no changes to the webpage; the "Body" text box remains empty.
|
||
- **Possible Purpose**: I might be considering whether to add more detailed content or examples in the "Body" section but have not yet decided to do so.
|
||
|
||
4. **Action**: I click on the "Create submission" button at the bottom of the form.
|
||
- **Page Changes**: The page does not navigate away or show any submission confirmation, suggesting the submission was not completed or there was an issue.
|
||
- **Possible Purpose**: My intention is to submit the post with the current information filled in the form. However, since the submission did not go through, it may indicate missing required fields or an error that needs addressing.
|
||
|
||
These actions are strictly based on the observations from the provided video segment, focusing solely on the interactions and immediate outcomes within this part of the video.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_Forums",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Forums\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_books \u2014 books",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_label(\"books \u2014 books\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/forums"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/forums"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Submit",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_label(\"Submit\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/f/books"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/f/books"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Title *",
|
||
"idx": 26,
|
||
"action_repr": "frame.pressget_by_label(\"Title *\")ArrowLeft",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/submit/books"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/submit/books"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Create submission",
|
||
"idx": 27,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Create submission\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/submit/books"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:9999/submit/books"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |