426 lines
19 KiB
Plaintext
426 lines
19 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Create a milestone for the upcoming task of merging all branches to main starting on March 15, 2044 and ending on March 30, 2044
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action:** I hover over the "Repository" option in the left sidebar menu.
|
||
- **Page Changes:** No immediate changes occur on the page as a result of this action.
|
||
- **Possible Purpose:** The likely intent is to either select the "Repository" option or to navigate through the menu options for further actions.
|
||
|
||
2. **Action:** I click on the "Issues" option in the left sidebar menu.
|
||
- **Page Changes:** The page transitions to the "Issues" section of the GitLab project. This section is designed to display and manage issues related to the project.
|
||
- **Possible Purpose:** The purpose of this action is to view or manage issues within the project, which could involve creating, updating, or reviewing existing issues.
|
||
|
||
3. **Action:** I move the cursor over the "Settings" button located near the top of the page.
|
||
- **Page Changes:** A dropdown menu appears under the "Settings" button, revealing additional options such as "General," "Repository," "CI/CD," etc.
|
||
- **Possible Purpose:** The intent is likely to access one of the settings-related options to configure or modify project settings.
|
||
|
||
4. **Action:** I click on the "Settings" button.
|
||
- **Page Changes:** The page navigates to the "Settings" section of the project. This section provides various configuration options for the project.
|
||
- **Possible Purpose:** The purpose is to enter the settings area to make adjustments or review configurations related to the project.
|
||
|
||
### Summary:
|
||
In this video segment, I first hovered over the "Repository" option, then clicked on the "Issues" option, causing the page to transition to the Issues section. Next, I moved the cursor over the "Settings" button, which revealed a dropdown menu, and subsequently clicked on the "Settings" button, navigating to the Settings section of the project. Each action was performed with the apparent intent of navigating through different sections of the GitLab project interface for management or configuration purposes.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action:** I click on the "Milestones" option in the left sidebar menu.
|
||
- **Page Changes:** The page transitions from the "Issues" section to the "Milestones" section, displaying a prompt to use milestones for tracking issues and merge requests over a fixed period of time. A "New milestone" button becomes visible.
|
||
- **Possible Purpose:** The intent is to navigate to the Milestones section to create or manage project milestones.
|
||
|
||
2. **Action:** I click on the "New milestone" button.
|
||
- **Page Changes:** A new form titled "New Milestone" appears, containing fields for "Title," "Start Date," "Due Date," and "Description."
|
||
- **Possible Purpose:** The goal is to initiate the creation of a new milestone by accessing the form required to input milestone details.
|
||
|
||
3. **Action:** I click inside the "Title" text box.
|
||
- **Page Changes:** The cursor focuses on the "Title" field, allowing text input.
|
||
- **Possible Purpose:** The intention is to enter a title for the new milestone.
|
||
|
||
4. **Action:** I type "Merging all branches to main" into the "Title" field.
|
||
- **Page Changes:** The text "Merging all branches to main" appears in the "Title" field.
|
||
- **Possible Purpose:** The purpose is to define the milestone's title, specifying its objective or scope.
|
||
|
||
5. **Action:** I click inside the "Start Date" text box.
|
||
- **Page Changes:** A date picker calendar appears, allowing the selection of a start date.
|
||
- **Possible Purpose:** The aim is to set a specific start date for the milestone.
|
||
|
||
6. **Action:** I select the date "2023-10-10" from the date picker calendar.
|
||
- **Page Changes:** The selected date "2023-10-10" is populated in the "Start Date" field.
|
||
- **Possible Purpose:** The intention is to assign a concrete start date to the milestone, marking when it should begin.
|
||
|
||
### Summary:
|
||
In this video segment, I navigated from the "Issues" section to the "Milestones" section, initiated the creation of a new milestone, entered a title for the milestone, and set a start date. Each action was performed with the clear intent of defining and scheduling a new project milestone within the GitLab interface.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action:** I click on the "Start Date" input field.
|
||
- **Page Changes:** A date picker calendar appears, allowing me to select a specific start date.
|
||
- **Possible Purpose:** The likely intent is to set a start date for the new milestone.
|
||
|
||
2. **Action:** I select the date "2044-03-15" from the date picker.
|
||
- **Page Changes:** The "Start Date" field updates to display "2044-03-15".
|
||
- **Possible Purpose:** This action sets the official start date for the milestone to March 15, 2044.
|
||
|
||
3. **Action:** I click on the "Due Date" input field.
|
||
- **Page Changes:** Another date picker calendar appears, enabling me to choose a due date.
|
||
- **Possible Purpose:** The intention is to set a due date for the milestone.
|
||
|
||
4. **Action:** I select the date "2044-03-30" from the date picker.
|
||
- **Page Changes:** The "Due Date" field updates to display "2044-03-30".
|
||
- **Possible Purpose:** This action sets the official due date for the milestone to March 30, 2044.
|
||
|
||
5. **Action:** I click on the "Description" text box.
|
||
- **Page Changes:** The text box becomes active, allowing me to type or paste text.
|
||
- **Possible Purpose:** The intent is to add a description for the milestone.
|
||
|
||
6. **Action:** I type "Merging all branches to main" into the "Description" text box.
|
||
- **Page Changes:** The text "Merging all branches to main" appears in the description field.
|
||
- **Possible Purpose:** This provides a brief explanation of what the milestone entails.
|
||
|
||
### Summary:
|
||
In this video segment, I interact with the "New Milestone" form in GitLab by setting both the start and due dates for a milestone and adding a description. Each step is methodical, focusing on accurately defining the timeline and purpose of the milestone. The actions are straightforward, aiming to create a structured and clear milestone entry.
|
||
|
||
---
|
||
|
||
## Part 4
|
||
In this video segment, I perform the following actions:
|
||
|
||
1. **Action**: I click on the "Create milestone" button.
|
||
- **Page Changes**: The page does not navigate away, but the action seems to be part of a test or demonstration, as indicated by the highlighted code at the bottom: `get_by_role("button", name="Create milestone")`.
|
||
- **Possible Purpose**: The likely intent is to finalize and create the new milestone with the details provided (Title: "Merging all branches to main", Start Date: "2044-03-15", Due Date: "2044-03-30", and Description: "Merging all branches to main"). The use of the code snippet suggests that this action is being verified programmatically, possibly as part of an automated testing scenario.
|
||
|
||
2. **Action**: I do not interact further with the page after clicking the "Create milestone" button.
|
||
- **Page Changes**: There are no additional changes or interactions observed on the page.
|
||
- **Possible Purpose**: The focus remains on the confirmation of the "Create milestone" button's functionality, ensuring it responds correctly to user input as part of the testing process.
|
||
|
||
The segment strictly shows the interaction with the "Create milestone" button and the immediate aftermath, with no further actions taken. The primary objective appears to be the validation of this specific UI element's behavior.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "action_0",
|
||
"idx": 0,
|
||
"action_repr": "frame.clicklocator(\"a\").filter(has_text=\"Issues 0\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Milestones",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Milestones\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_New milestone",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"New milestone\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Title",
|
||
"idx": 21,
|
||
"action_repr": "frame.clickget_by_label(\"Title\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_4",
|
||
"idx": 4,
|
||
"action_repr": "frame.clicklocator(\"div\").filter(has_text=\"Clear start date\").nth(4)",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_5",
|
||
"idx": 5,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Select start date\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Next Month",
|
||
"idx": 6,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Next Month\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_10",
|
||
"idx": 7,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"10\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_8",
|
||
"idx": 8,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Select start date\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_9",
|
||
"idx": 9,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Select start date\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_10",
|
||
"idx": 10,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Select due date\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_11",
|
||
"idx": 11,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Select start date\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_12",
|
||
"idx": 12,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Select start date\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_13",
|
||
"idx": 13,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Write milestone description...\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_14",
|
||
"idx": 14,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Write milestone description...\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_15",
|
||
"idx": 15,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Select due date\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_16",
|
||
"idx": 16,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Write milestone description...\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_17",
|
||
"idx": 17,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Write milestone description...\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_18",
|
||
"idx": 18,
|
||
"action_repr": "frame.clicklocator(\"div\").filter(has_text=\"Title\").nth(4)",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_22",
|
||
"idx": 22,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Write milestone description...\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Create milestone",
|
||
"idx": 23,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Create milestone\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/byteblaze/dotfiles/-/milestones/new"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |