387 lines
20 KiB
Plaintext
387 lines
20 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Start a private project web_agent_android_xl with Android template and add primer, convexegg, abishek as members
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions:
|
||
|
||
#### 1. **Action:** I click on the "New project" button.
|
||
- **Page Changes:** The page transitions to a new interface titled "Create blank project," where I can input details for a new project.
|
||
- **Possible Purpose:** The intent is to initiate the creation of a new project within the GitLab environment.
|
||
|
||
#### 2. **Action:** I click inside the "Project name" text box.
|
||
- **Page Changes:** The text box becomes active, allowing me to type in a name for the project.
|
||
- **Possible Purpose:** To enter a specific name for the new project, which is a required step in the project creation process.
|
||
|
||
#### 3. **Action:** I type "web_agent_android" into the "Project name" text box.
|
||
- **Page Changes:** As I type, the text appears in the "Project name" field. Additionally, the "Project slug" field automatically updates to reflect a URL-friendly version of the project name ("web_agent_android").
|
||
- **Possible Purpose:** To assign a meaningful and relevant name to the project, which will also determine part of the project's URL.
|
||
|
||
#### 4. **Action:** I scroll down slightly on the page.
|
||
- **Page Changes:** The visible portion of the page shifts downward, revealing more options such as "Visibility Level" and "Project Configuration."
|
||
- **Possible Purpose:** To review and configure additional settings for the project before finalizing its creation.
|
||
|
||
#### 5. **Action:** I select the "Public" option under "Visibility Level."
|
||
- **Page Changes:** The "Public" radio button is marked as selected, and the description next to it indicates that the project will be accessible without any authentication.
|
||
- **Possible Purpose:** To set the project's visibility to "Public," making it accessible to anyone without requiring login credentials.
|
||
|
||
#### 6. **Action:** I scroll further down to the "Project Configuration" section.
|
||
- **Page Changes:** The view now includes options like "Initialize repository with a README" and "Enable Static Application Security Testing (SAST)."
|
||
- **Possible Purpose:** To review and decide on additional configurations for the project setup.
|
||
|
||
#### 7. **Action:** I check the box for "Initialize repository with a README."
|
||
- **Page Changes:** The checkbox is marked, indicating that a README file will be created when the project is initialized.
|
||
- **Possible Purpose:** To ensure that the project repository starts with a README file, which is often used to provide initial information about the project.
|
||
|
||
#### 8. **Action:** I click the "Create project" button.
|
||
- **Page Changes:** The button press initiates the project creation process. The page likely transitions to a new view showing the newly created project's dashboard or repository.
|
||
- **Possible Purpose:** To finalize and create the project with the specified settings and configurations.
|
||
|
||
### Summary:
|
||
In this video segment, I systematically navigate through the steps to create a new project named "web_agent_android" in GitLab. I input the project name, set its visibility to "Public," initialize it with a README, and finally confirm the creation. Each action is deliberate, focusing on configuring the project according to specific requirements before completing the setup.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the "Project name" text box.
|
||
- **Page Changes:** The text box becomes active, allowing me to input or modify text.
|
||
- **Possible Purpose:** My intention is to either enter a new project name or edit an existing one. This is a preparatory step for creating or configuring a project.
|
||
|
||
#### 2. **Action:** I type "web_agent_android" into the "Project name" text box.
|
||
- **Page Changes:** The text "web_agent_android" appears in the text box, and the "Project slug" field automatically updates to reflect a URL-friendly version of the project name ("web_agent_android").
|
||
- **Possible Purpose:** I am naming the project "web_agent_android," likely to identify it as related to an Android web agent application. The automatic update of the "Project slug" ensures that the URL for this project will be correctly formatted.
|
||
|
||
#### 3. **Action:** I scroll down the page.
|
||
- **Page Changes:** The view shifts to reveal more options and settings below the initial project creation fields.
|
||
- **Possible Purpose:** I am exploring additional settings or options that might be relevant for configuring the project, such as visibility levels, project templates, or other configurations.
|
||
|
||
#### 4. **Action:** I hover over the "Visibility Level" section.
|
||
- **Page Changes:** No immediate changes occur, but the section becomes highlighted, indicating focus.
|
||
- **Possible Purpose:** I am reviewing the visibility options to decide how accessible I want the project to be. This could involve choosing between Private, Internal, or Public settings based on my needs for collaboration and access control.
|
||
|
||
#### 5. **Action:** I select the "Private" radio button under "Visibility Level."
|
||
- **Page Changes:** The "Private" option is marked as selected, and a description explains that project access must be granted explicitly to each user.
|
||
- **Possible Purpose:** I am setting the project's visibility to "Private," which restricts access to only those users I explicitly grant permission to. This ensures controlled access to the project's contents.
|
||
|
||
#### 6. **Action:** I scroll further down the page.
|
||
- **Page Changes:** More options become visible, including sections for "Project Configuration" and "Import project."
|
||
- **Possible Purpose:** I am continuing to review all available settings to ensure the project is configured correctly before finalizing its creation.
|
||
|
||
#### 7. **Action:** I hover over the "Create project" button at the bottom of the form.
|
||
- **Page Changes:** The button is highlighted, indicating it is ready to be clicked.
|
||
- **Possible Purpose:** I am preparing to submit the form and create the project with the specified settings. Hovering over the button is a common action before confirming and executing such an important task.
|
||
|
||
### Summary
|
||
In this video segment, I focused on naming the project "web_agent_android," configuring its visibility to "Private," and reviewing additional settings. My actions suggest a methodical approach to project creation, ensuring that all necessary details are correctly set before finalizing the setup. The sequence of actions reflects a clear intent to establish a new project with specific parameters tailored to my requirements.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action:** I click on the "Members" option in the left sidebar menu.
|
||
- **Page Changes:** The page transitions to the "Project members" section, displaying a list of current members and options to invite new members or groups.
|
||
- **Possible Purpose:** The intent is to manage project members, specifically to add new members to the project.
|
||
|
||
2. **Action:** I click on the "Invite members" button.
|
||
- **Page Changes:** A modal window titled "Invite members" appears, prompting me to enter a username or email address and select a role for the new member.
|
||
- **Possible Purpose:** The goal is to initiate the process of adding a new member to the project by providing the necessary details.
|
||
|
||
3. **Action:** I click inside the "Username or email address" text box and type "abis".
|
||
- **Page Changes:** As I type, a dropdown list appears below the text box, suggesting usernames that match the entered text.
|
||
- **Possible Purpose:** The action aims to find and select a specific user to invite by entering part of their username.
|
||
|
||
4. **Action:** I use the arrow keys or mouse to select a username from the dropdown list.
|
||
- **Page Changes:** The selected username is added to the text box, confirming the choice of the user to be invited.
|
||
- **Possible Purpose:** This ensures the correct user is chosen for invitation based on the provided suggestions.
|
||
|
||
5. **Action:** I click on the "Select a role" dropdown menu and choose "Maintainer".
|
||
- **Page Changes:** The role "Maintainer" is displayed next to the selected username, indicating the access level for the invited user.
|
||
- **Possible Purpose:** The intent is to assign a specific role with defined permissions to the new member, in this case, "Maintainer," which typically grants broader access and responsibilities.
|
||
|
||
6. **Action:** I click the "Invite" button within the modal window.
|
||
- **Page Changes:** The modal window closes, and the main page updates to reflect the new member's addition, showing their username and assigned role under "Project members."
|
||
- **Possible Purpose:** The final step confirms the invitation, officially adding the selected user to the project with the specified role.
|
||
|
||
### Summary:
|
||
The sequence of actions involves navigating to the project members section, initiating an invitation, selecting a user via a search function, assigning them a role, and confirming the invitation. Each step is methodically executed to ensure the correct user is added with the appropriate permissions.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_New project",
|
||
"idx": 4,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"New project\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Create blank project Create a blank project to store your files, plan your work, and collaborate on code, among other things.",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Create blank project Create a blank project to store your files, plan your work, and collaborate on code, among other things.\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Project name",
|
||
"idx": 7,
|
||
"action_repr": "frame.clickget_by_label(\"Project name\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new#blank_project"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new#blank_project"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "lock-icon",
|
||
"idx": 3,
|
||
"action_repr": "frame.clicklocator(\"#blank-project-pane\").get_by_test_id(\"lock-icon\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new#blank_project"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new#blank_project"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Create from template Create a project pre-populated with the necessary files to get you started quickly.",
|
||
"idx": 5,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Create from template Create a project pre-populated with the necessary files to get you started quickly.\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new#create_from_template"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new#create_from_template"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "use_template_android",
|
||
"idx": 6,
|
||
"action_repr": "frame.clickget_by_test_id(\"use_template_android\").get_by_text(\"Use template\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new#create_from_template"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new#create_from_template"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_8",
|
||
"idx": 8,
|
||
"action_repr": "frame.clicklocator(\".gl-form-radio\").first",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new#create_from_template"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new#create_from_template"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Create project",
|
||
"idx": 9,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Create project\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/web_agent_android/-/import"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new#create_from_template"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Project information",
|
||
"idx": 10,
|
||
"action_repr": "frame.clickget_by_label(\"Project information\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/web_agent_android/-/import"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/web_agent_android/-/import"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Members",
|
||
"idx": 11,
|
||
"action_repr": "frame.clickget_by_label(\"Members\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/web_agent_android/-/project_members"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/web_agent_android/activity"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_12",
|
||
"idx": 12,
|
||
"action_repr": "frame.clicklocator(\"[data-test-id=\\\"invite-members-button\\\"]\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/web_agent_android/-/project_members"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/web_agent_android/-/project_members"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "members-token-select-input",
|
||
"idx": 17,
|
||
"action_repr": "frame.clickget_by_test_id(\"members-token-select-input\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/web_agent_android/-/project_members"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/web_agent_android/-/project_members"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "menuitem_Abishek S abisubramanya27",
|
||
"idx": 14,
|
||
"action_repr": "frame.clickget_by_role(\"menuitem\", name=\"Abishek S abisubramanya27\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/web_agent_android/-/project_members"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/web_agent_android/-/project_members"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "menuitem_Primer primer",
|
||
"idx": 16,
|
||
"action_repr": "frame.clickget_by_role(\"menuitem\", name=\"Primer primer\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/web_agent_android/-/project_members"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/web_agent_android/-/project_members"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "menuitem_Convex Eggtart convexegg",
|
||
"idx": 18,
|
||
"action_repr": "frame.clickget_by_role(\"menuitem\", name=\"Convex Eggtart convexegg\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/web_agent_android/-/project_members"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/web_agent_android/-/project_members"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Select a role",
|
||
"idx": 19,
|
||
"action_repr": "frame.selectOptionget_by_label(\"Select a role\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/web_agent_android/-/project_members"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/web_agent_android/-/project_members"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "invite-modal",
|
||
"idx": 20,
|
||
"action_repr": "frame.clickget_by_test_id(\"invite-modal\").get_by_role(\"button\", name=\"Invite\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/web_agent_android/-/project_members"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/web_agent_android/-/project_members"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |