326 lines
16 KiB
Plaintext
326 lines
16 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Create a new public project "awesome-llms" and add primer, convexegg, abishek as members
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the "New project" button.
|
||
- **Page Changes:** The page transitions to a new section titled "Create blank project," where I can input details for a new project.
|
||
- **Possible Purpose:** The intent is to initiate the creation of a new project within the GitLab environment.
|
||
|
||
#### 2. **Action:** I click inside the "Project name" text box and type "awesome-lims."
|
||
- **Page Changes:** As I type, the text "awesome-lims" appears in the "Project name" field. The "Project slug" field automatically updates to "awesome-lims" as well.
|
||
- **Possible Purpose:** The purpose is to name the new project "awesome-lims," which will also set the project's URL slug to the same name for consistency.
|
||
|
||
#### 3. **Action:** I click inside the "Project URL" text box.
|
||
- **Page Changes:** The cursor moves to the "Project URL" field, but no text is entered or modified in this step.
|
||
- **Possible Purpose:** The likely intent is to either verify the auto-generated URL or prepare to edit it if necessary.
|
||
|
||
#### 4. **Action:** I scroll down slightly to view more options on the page.
|
||
- **Page Changes:** Additional options such as "Visibility Level" and "Project Configuration" become more visible.
|
||
- **Possible Purpose:** The intent is to review and possibly configure further settings for the project before finalizing its creation.
|
||
|
||
#### 5. **Action:** I click the "Create project" button.
|
||
- **Page Changes:** The button is highlighted, indicating an interaction. However, the exact outcome (such as project creation confirmation) is not shown in this segment.
|
||
- **Possible Purpose:** The goal is to submit the form and create the project with the specified details.
|
||
|
||
### Summary
|
||
In this video segment, I initiate the creation of a new project named "awesome-lims" in GitLab. I input the project name, review the auto-generated URL, and proceed to create the project by submitting the form. Each step is methodical, focusing on accurately setting up the project with the desired configurations.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action:** I click on the "Members" option in the left sidebar menu.
|
||
- **Page Changes:** The page transitions to the "Project members" section, displaying a list of current members and options to invite new members.
|
||
- **Possible Purpose:** The likely intent is to manage project members, either by viewing existing members or adding new ones.
|
||
|
||
2. **Action:** I click on the "Invite members" button.
|
||
- **Page Changes:** A modal window titled "Invite members" appears, providing fields to enter a username or email address, select a role, and set an access expiration date.
|
||
- **Possible Purpose:** The intention is to invite a new member to the project by providing the necessary details.
|
||
|
||
3. **Action:** I click inside the "Username or email address" text box and type "primer".
|
||
- **Page Changes:** As I type, a dropdown list appears below the text box, suggesting usernames that match the input "primer".
|
||
- **Possible Purpose:** The action aims to find and select a specific user to invite by entering part of their username.
|
||
|
||
4. **Action:** I use the arrow keys or mouse to highlight one of the suggested usernames in the dropdown list.
|
||
- **Page Changes:** The highlighted username becomes selected, ready to be confirmed for invitation.
|
||
- **Possible Purpose:** This step ensures the correct user is chosen from the suggestions before proceeding with the invitation.
|
||
|
||
### Summary:
|
||
In this video segment, I navigate to the "Members" section of a GitLab project, initiate the process to invite a new member, enter a partial username to search for the correct user, and select the desired username from the dropdown list. Each step is methodically performed to accurately invite a specific user to the project.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action**: I click on the "Select a role" dropdown menu.
|
||
- **Page Changes**: The dropdown menu expands, revealing various role options such as "Guest," "Reporter," "Developer," "Maintainer," and "Owner."
|
||
- **Possible Purpose**: The likely intent is to choose a specific role for the members being invited to the project. This step is necessary to define the level of access and permissions these members will have within the project.
|
||
|
||
#### 2. **Action**: I hover over the "Developer" option in the dropdown menu.
|
||
- **Page Changes**: No immediate changes occur on the page; the "Developer" option remains highlighted as I hover over it.
|
||
- **Possible Purpose**: Hovering over the "Developer" option suggests that I am considering or about to select this role. The purpose is to ensure that the correct role is chosen before finalizing the selection.
|
||
|
||
#### 3. **Action**: I click on the "Developer" option in the dropdown menu.
|
||
- **Page Changes**: The "Developer" role is selected, and the dropdown menu collapses. The text "Developer" is now displayed in the "Select a role" field.
|
||
- **Possible Purpose**: By selecting "Developer," I am assigning this specific role to the members being invited. This grants them a defined set of permissions appropriate for developers within the project.
|
||
|
||
#### 4. **Action**: I move my cursor towards the "Invite" button at the bottom of the modal.
|
||
- **Page Changes**: No changes occur on the page as I move the cursor; however, the "Invite" button becomes the focus of attention.
|
||
- **Possible Purpose**: Moving the cursor towards the "Invite" button indicates the intention to finalize the invitation process. This action prepares to submit the form and invite the selected members with the chosen role.
|
||
|
||
#### 5. **Action**: I click on the "Invite" button.
|
||
- **Page Changes**: After clicking the "Invite" button, the modal window closes, and a notification appears stating "Members were successfully added." The list of project members updates to include the newly invited members with their respective roles.
|
||
- **Possible Purpose**: Clicking the "Invite" button submits the form, officially inviting the selected members to the project with the specified "Developer" role. This completes the invitation process, integrating the new members into the project team.
|
||
|
||
### Summary
|
||
In this video segment, I perform a series of actions to invite members to a GitLab project and assign them the "Developer" role. The steps include selecting the role from a dropdown menu, confirming the selection, and finalizing the invitation by clicking the "Invite" button. The successful addition of members is confirmed by a notification and an updated list of project members. Each action is focused on ensuring the correct setup of member roles and permissions within the project.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_New project",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"New project\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Create blank project Create a blank project to store your files, plan your work, and collaborate on code, among other things.",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Create blank project Create a blank project to store your files, plan your work, and collaborate on code, among other things.\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Project name",
|
||
"idx": 4,
|
||
"action_repr": "frame.clickget_by_label(\"Project name\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new#blank_project"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new#blank_project"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Project slug",
|
||
"idx": 3,
|
||
"action_repr": "frame.clickget_by_label(\"Project slug\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new#blank_project"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new#blank_project"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Create project",
|
||
"idx": 5,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Create project\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new#blank_project"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/projects/new#blank_project"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Project information",
|
||
"idx": 6,
|
||
"action_repr": "frame.clickget_by_label(\"Project information\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/awesome-llms"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/awesome-llms"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Members",
|
||
"idx": 7,
|
||
"action_repr": "frame.clickget_by_label(\"Members\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/awesome-llms/activity"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/awesome-llms/activity"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_8",
|
||
"idx": 8,
|
||
"action_repr": "frame.clicklocator(\"[data-test-id=\\\"invite-members-button\\\"]\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/awesome-llms/-/project_members"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/awesome-llms/-/project_members"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "menuitem_Primer primer",
|
||
"idx": 9,
|
||
"action_repr": "frame.clickget_by_role(\"menuitem\", name=\"Primer primer\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/awesome-llms/-/project_members"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/awesome-llms/-/project_members"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "members-token-select-input",
|
||
"idx": 12,
|
||
"action_repr": "frame.clickget_by_test_id(\"members-token-select-input\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/awesome-llms/-/project_members"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/awesome-llms/-/project_members"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "menuitem_Convex Eggtart convexegg",
|
||
"idx": 11,
|
||
"action_repr": "frame.clickget_by_role(\"menuitem\", name=\"Convex Eggtart convexegg\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/awesome-llms/-/project_members"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/awesome-llms/-/project_members"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "menuitem_Abishek S abisubramanya27",
|
||
"idx": 13,
|
||
"action_repr": "frame.clickget_by_role(\"menuitem\", name=\"Abishek S abisubramanya27\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/awesome-llms/-/project_members"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/awesome-llms/-/project_members"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Select a role",
|
||
"idx": 14,
|
||
"action_repr": "frame.selectOptionget_by_label(\"Select a role\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/awesome-llms/-/project_members"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/awesome-llms/-/project_members"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "invite-modal",
|
||
"idx": 15,
|
||
"action_repr": "frame.clickget_by_test_id(\"invite-modal\").get_by_role(\"button\", name=\"Invite\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/awesome-llms/-/project_members"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/awesome-llms/-/project_members"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |