334 lines
16 KiB
Plaintext
334 lines
16 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Create a new forum named Cyberpunk, with a description of Welcome to the future, and include ['Games', 'Books', 'Movies', 'Future'] in the sidebar?
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action:** I click on the "Create forum" button located on the right side of the page.
|
||
- **Page Changes:** The page transitions to a new form titled "Create new forum," which includes fields for Name, Title, Description, Sidebar, and Tags.
|
||
- **Possible Purpose:** The likely intent is to initiate the process of creating a new forum by accessing the form required for inputting the necessary details.
|
||
|
||
2. **Action:** I click inside the "Name" text box.
|
||
- **Page Changes:** The cursor appears in the "Name" text box, indicating it is active and ready for text input.
|
||
- **Possible Purpose:** The intent is to enter a unique name for the new forum, which is a required step in the forum creation process.
|
||
|
||
3. **Action:** I type "//Cyberpunk" into the "Name" text box.
|
||
- **Page Changes:** The text "//Cyberpunk" is displayed in the "Name" text box.
|
||
- **Possible Purpose:** The purpose is to assign a specific name to the forum, potentially following a naming convention or guideline that includes a prefix like "//".
|
||
|
||
4. **Action:** I click inside the "Title" text box.
|
||
- **Page Changes:** The cursor appears in the "Title" text box, indicating it is active and ready for text input.
|
||
- **Possible Purpose:** The intent is to enter a title for the forum, which will likely be a more descriptive name or heading for users to understand the forum's purpose.
|
||
|
||
5. **Action:** I type "Cyberpunk" into the "Title" text box.
|
||
- **Page Changes:** The text "Cyberpunk" is displayed in the "Title" text box.
|
||
- **Possible Purpose:** The purpose is to provide a clear and concise title for the forum, making it easily identifiable to users.
|
||
|
||
### Summary:
|
||
In this video segment, I initiated the creation of a new forum by clicking the "Create forum" button, then proceeded to fill in the "Name" and "Title" fields with the specified text. Each action was focused on progressing through the forum creation form by providing the necessary initial details.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
1. **Action:** I click inside the "Title" text box.
|
||
- **Page Changes:** The cursor appears in the "Title" text box, indicating it is active and ready for input.
|
||
- **Possible Purpose:** The likely intent is to enter or modify the title of the forum.
|
||
|
||
2. **Action:** I type "Cyberpunk" into the "Title" text box.
|
||
- **Page Changes:** The text "Cyberpunk" appears in the "Title" text box.
|
||
- **Possible Purpose:** The purpose is to set the title of the forum to "Cyberpunk."
|
||
|
||
3. **Action:** I click inside the "Description" text box.
|
||
- **Page Changes:** The cursor appears in the "Description" text box, indicating it is active and ready for input.
|
||
- **Possible Purpose:** The likely intent is to enter or modify the description of the forum.
|
||
|
||
4. **Action:** I type "Welcome to the future" into the "Description" text box.
|
||
- **Page Changes:** The text "Welcome to the future" appears in the "Description" text box.
|
||
- **Possible Purpose:** The purpose is to provide a brief description for the forum, which will be displayed to users and search engines.
|
||
|
||
5. **Action:** I click inside the "Sidebar" text box.
|
||
- **Page Changes:** The cursor appears in the "Sidebar" text box, indicating it is active and ready for input.
|
||
- **Possible Purpose:** The likely intent is to enter or modify the content that will appear in the sidebar of the forum.
|
||
|
||
6. **Action:** I type "Games" into the "Sidebar" text box.
|
||
- **Page Changes:** The text "Games" appears in the "Sidebar" text box.
|
||
- **Possible Purpose:** The purpose is to add "Games" as a piece of content to be displayed in the sidebar.
|
||
|
||
7. **Action:** I press the Enter key while the cursor is still in the "Sidebar" text box.
|
||
- **Page Changes:** A new line is created in the "Sidebar" text box.
|
||
- **Possible Purpose:** The likely intent is to create a new line for additional content in the sidebar.
|
||
|
||
8. **Action:** I type "Book" into the "Sidebar" text box on the new line.
|
||
- **Page Changes:** The text "Book" appears on the new line in the "Sidebar" text box.
|
||
- **Possible Purpose:** The purpose is to add "Book" as another piece of content to be displayed in the sidebar, on a new line below "Games."
|
||
|
||
9. **Action:** I click the "Preview" button.
|
||
- **Page Changes:** The page updates to show a preview of the forum based on the entered information.
|
||
- **Possible Purpose:** The likely intent is to review how the forum will appear with the current settings before finalizing the creation.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
In this video segment, I perform the following actions:
|
||
|
||
1. **Action**: I click inside the "Sidebar" text box.
|
||
- **Page Changes**: The cursor appears inside the "Sidebar" text box, indicating it is active and ready for input.
|
||
- **Possible Purpose**: My likely intent is to add or modify content in the "Sidebar" section of the forum creation form.
|
||
|
||
2. **Action**: I type the word "Movies" into the "Sidebar" text box.
|
||
- **Page Changes**: The word "Movies" appears as the latest entry in the list within the "Sidebar" text box.
|
||
- **Possible Purpose**: I am populating the "Sidebar" with relevant categories or links that will be displayed on the forum's sidebar.
|
||
|
||
3. **Action**: I press the Enter key after typing "Movies."
|
||
- **Page Changes**: The word "Movies" is added as a new line item in the "Sidebar" text box.
|
||
- **Possible Purpose**: By pressing Enter, I am finalizing the entry of "Movies" and preparing to add another item to the list.
|
||
|
||
4. **Action**: I type the word "Future" into the "Sidebar" text box.
|
||
- **Page Changes**: The word "Future" appears as the latest entry in the list within the "Sidebar" text box.
|
||
- **Possible Purpose**: I continue to populate the "Sidebar" with additional categories or links relevant to the forum's theme.
|
||
|
||
5. **Action**: I press the Enter key after typing "Future."
|
||
- **Page Changes**: The word "Future" is added as a new line item in the "Sidebar" text box.
|
||
- **Possible Purpose**: Similar to the previous action, pressing Enter finalizes the entry of "Future" and allows me to add more items if needed.
|
||
|
||
6. **Action**: I scroll down slightly on the page.
|
||
- **Page Changes**: The view shifts downward, revealing more of the form elements below the "Sidebar" section, such as the "Tags" field and the "Create forum" button.
|
||
- **Possible Purpose**: I am likely preparing to interact with elements further down the page, such as adding tags or submitting the form.
|
||
|
||
7. **Action**: I click on the "Create forum" button.
|
||
- **Page Changes**: The button is highlighted, indicating it has been clicked. However, the exact outcome (such as navigation to a new page or display of a confirmation message) is not shown in this segment.
|
||
- **Possible Purpose**: My intention is to submit the form and create the new forum with the details I have provided.
|
||
|
||
These actions are strictly based on the observations from the provided video segment, focusing solely on the interactions and changes visible in this part of the video.
|
||
|
||
---
|
||
|
||
## Part 4
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action:** I click on the "Create forum" button.
|
||
- **Page Changes:** The page transitions from the forum creation form to a new forum page titled "/f/Cyberpunk". The new page displays sections for "Submissions", "Comments", and a sidebar with options like "Unsubscribe", "Hide this forum", "Toolbox", and "Moderators".
|
||
- **Possible Purpose:** The likely intent is to finalize the creation of the forum by submitting the form, resulting in the display of the newly created forum page.
|
||
|
||
2. **Action:** I scroll down slightly on the new forum page.
|
||
- **Page Changes:** The view shifts slightly downward, but no significant changes occur in the content displayed.
|
||
- **Possible Purpose:** The purpose appears to be to get a better view of the lower part of the forum page, possibly to inspect additional elements or options that are initially out of view.
|
||
|
||
3. **Action:** I click on the "Edit forum" link in the "Toolbox" section of the sidebar.
|
||
- **Page Changes:** The page transitions to an edit forum interface where various settings and details of the forum can be modified.
|
||
- **Possible Purpose:** The intent is to access the forum editing options to make adjustments or updates to the forum's configuration or content.
|
||
|
||
### Summary:
|
||
- **Initial State:** On the forum creation form page.
|
||
- **Final State:** On the forum editing interface after creating and then selecting to edit the new forum.
|
||
- **Key Actions:** Clicking "Create forum", slight scrolling, and clicking "Edit forum". Each action logically progresses towards setting up and customizing the newly created forum.
|
||
|
||
---
|
||
|
||
## Part 5
|
||
In this video segment, the following actions are observed:
|
||
|
||
1. **Action**: I hover over the "Unsubscribe" button located under the "Cyberpunk" section on the right sidebar.
|
||
- **Page Changes**: There are no immediate changes to the webpage as a result of this action.
|
||
- **Possible Purpose**: The likely intent is to explore the option to unsubscribe from the "Cyberpunk" forum, possibly to stop receiving notifications or updates related to this forum.
|
||
|
||
2. **Action**: I click on the "Unsubscribe" button.
|
||
- **Page Changes**: After clicking, the "Unsubscribe" button changes to "Subscribe," indicating that the unsubscribe action has been successfully executed.
|
||
- **Possible Purpose**: The purpose of this action is to unsubscribe from the "Cyberpunk" forum, effectively removing myself from its subscriber list and stopping any associated notifications or updates.
|
||
|
||
3. **Action**: I move the cursor away from the "Subscribe" button (formerly "Unsubscribe").
|
||
- **Page Changes**: No further changes occur on the page as a result of moving the cursor.
|
||
- **Possible Purpose**: This action suggests that after unsubscribing, I am either finished with this task or preparing to perform another action on the page.
|
||
|
||
These steps comprehensively describe the actions taken in the provided video segment, focusing solely on the observable interactions and their direct outcomes.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_Forums",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Forums\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Create forum",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Create forum\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/create_forum"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/forums"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_/f/",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_label(\"/f/\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/create_forum"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/create_forum"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Title *",
|
||
"idx": 3,
|
||
"action_repr": "frame.clickget_by_label(\"Title *\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/create_forum"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/create_forum"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Description *",
|
||
"idx": 4,
|
||
"action_repr": "frame.clickget_by_label(\"Description *\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/create_forum"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/create_forum"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Sidebar *",
|
||
"idx": 8,
|
||
"action_repr": "frame.pressget_by_label(\"Sidebar *\")Meta+z",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/create_forum"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/create_forum"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Create forum",
|
||
"idx": 9,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Create forum\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/create_forum"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/create_forum"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "heading_/f/Cyberpunk",
|
||
"idx": 10,
|
||
"action_repr": "frame.clickget_by_role(\"heading\", name=\"/f/Cyberpunk\").get_by_role(\"strong\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/f/Cyberpunk"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/f/Cyberpunk"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "text_Games Books Movies Future",
|
||
"idx": 11,
|
||
"action_repr": "frame.clickget_by_text(\"Games Books Movies Future\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/f/Cyberpunk"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:9999/f/Cyberpunk"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |