298 lines
13 KiB
Plaintext
298 lines
13 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Assign the issue regarding 404 in a11yproject to myself.
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Part 1: Initial Page and Navigation
|
||
|
||
#### Action:
|
||
I click on the text "Projects" located in the navigation bar at the top of the page.
|
||
|
||
#### Page Changes:
|
||
The page transitions to display a list of projects under the "Projects" section. The projects are categorized under "Personal," and each project entry includes details such as the project name, description, owner/maintainer, star count, fork count, and last updated date.
|
||
|
||
#### Possible Purpose:
|
||
The likely intent is to navigate to the Projects section to view or manage available projects.
|
||
|
||
---
|
||
|
||
#### Action:
|
||
I click on the project titled "The A11Y Project / a11yproject.com."
|
||
|
||
#### Page Changes:
|
||
The page navigates to the detailed view of the "a11yproject.com" project. This page includes sections like Project information, Repository, Issues, Merge requests, CI/CD, Deployments, Infrastructure, Monitor, Analytics, Wiki, Snippets, and Settings on the left sidebar. The main content area displays project details, including a brief description, statistics (e.g., commits, branches, tags), and options like "Star," "Fork," and "Clone."
|
||
|
||
#### Possible Purpose:
|
||
The purpose is to access the specific project "a11yproject.com" to view its details, files, issues, or other related information.
|
||
|
||
---
|
||
|
||
#### Action:
|
||
I click on the "Issues" link in the left sidebar.
|
||
|
||
#### Page Changes:
|
||
The page updates to show the Issues section of the "a11yproject.com" project. It lists open and closed issues with filters for "Open," "Closed," and "All." The page indicates there are no open issues currently.
|
||
|
||
#### Possible Purpose:
|
||
The intent is to check the Issues section of the project, possibly to review existing issues, track progress, or create a new issue.
|
||
|
||
---
|
||
|
||
#### Action:
|
||
I click on the "New issue" button located at the top right of the Issues page.
|
||
|
||
#### Page Changes:
|
||
A new page opens where I can create a new issue. This page includes fields for the issue title, description, labels, milestone, assignees, and due date.
|
||
|
||
#### Possible Purpose:
|
||
The purpose is to initiate the creation of a new issue within the project, likely to report a problem, suggest an enhancement, or document a task.
|
||
|
||
---
|
||
|
||
#### Action:
|
||
I click on the "Sign in/Register" link at the top right corner of the page.
|
||
|
||
#### Page Changes:
|
||
The page redirects to the GitLab sign-in page. This page features input fields for "Username or email" and "Password," along with a "Sign in" button, a "Remember me" checkbox, and links for "Forgot your password?" and "Register now."
|
||
|
||
#### Possible Purpose:
|
||
The intent is to access the sign-in page, presumably to log into an account for actions that require authentication, such as creating or commenting on issues, managing projects, or accessing private repositories.
|
||
|
||
---
|
||
|
||
### Summary of Actions:
|
||
1. Navigated to the Projects section and selected the "a11yproject.com" project.
|
||
2. Accessed the Issues section of the project.
|
||
3. Initiated the creation of a new issue.
|
||
4. Navigated to the sign-in page by clicking the "Sign in/Register" link.
|
||
|
||
Each step logically follows from the previous one, focusing on accessing and interacting with project-related features in GitLab.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
In this video segment, I perform the following actions:
|
||
|
||
1. **Action**: I click inside the "Password" text box.
|
||
- **Page Changes**: The cursor appears inside the "Password" text box, indicating it is active and ready for input.
|
||
- **Possible Purpose**: The likely intent is to enter a password for logging into the GitLab Community Edition account.
|
||
|
||
2. **Action**: I type a password into the "Password" text box.
|
||
- **Page Changes**: As I type, the characters appear as dots (•) in the text box, masking the actual password for security.
|
||
- **Possible Purpose**: The purpose is to input the necessary password to authenticate the login credentials.
|
||
|
||
3. **Action**: I move the cursor over the "Sign in" button and click it.
|
||
- **Page Changes**: After clicking, there is no immediate visible change on the page; it remains on the login screen.
|
||
- **Possible Purpose**: The intent is to submit the login credentials (username or email and password) to access the GitLab account. The lack of page change could indicate that the credentials are being processed, or there might be an error that hasn't been displayed yet.
|
||
|
||
4. **Action**: I hover over the footer links at the bottom of the page.
|
||
- **Page Changes**: No significant changes occur on the page as a result of this action.
|
||
- **Possible Purpose**: This action might be exploratory, intending to review additional resources or information provided by the links such as "Explore," "Help," "About GitLab," or "Community forum."
|
||
|
||
These actions are strictly based on the observations from the provided video segment, focusing solely on the interactions and their immediate effects without additional context or assumptions.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
In this video segment, I perform the following actions:
|
||
|
||
1. **Action**: I click inside the "Password" text box.
|
||
- **Page Changes**: The cursor appears inside the "Password" text box, indicating it is active and ready for input.
|
||
- **Possible Purpose**: The likely intent is to enter a password for logging into the GitLab Community Edition account.
|
||
|
||
2. **Action**: I type a password into the "Password" text box.
|
||
- **Page Changes**: As I type, the characters appear as dots (•) in the text box, obscuring the actual password for security.
|
||
- **Possible Purpose**: The purpose is to input the necessary password to authenticate the login credentials.
|
||
|
||
3. **Action**: I move the mouse cursor over the "Sign in" button.
|
||
- **Page Changes**: The "Sign in" button may visually change slightly, such as highlighting or changing color, indicating it is hovered over.
|
||
- **Possible Purpose**: This action suggests an intention to proceed with the login process after entering the password.
|
||
|
||
4. **Action**: I click on the "Sign in" button.
|
||
- **Page Changes**: After clicking, the page begins to process the login request. There might be a brief loading period, and if the credentials are correct, the page will navigate to the user's dashboard or main interface. If incorrect, an error message may appear.
|
||
- **Possible Purpose**: The intent is to submit the login credentials and access the GitLab Community Edition platform.
|
||
|
||
5. **Action**: I observe the bottom of the screen where a tooltip appears over the footer links.
|
||
- **Page Changes**: A tooltip displays the text "Explore Help About GitLab Community forum," which is a list of footer navigation links.
|
||
- **Possible Purpose**: This action could be unintentional or part of exploring the page elements, possibly to understand more about the available resources or help options.
|
||
|
||
These steps strictly document the observable actions within the provided video segment, focusing solely on the interactions and their immediate effects on the webpage.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_The A11Y Project\\n/ a11yproject.com",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"The A11Y Project\\n/ a11yproject.com\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_1",
|
||
"idx": 1,
|
||
"action_repr": "frame.clicklocator(\"a\").filter(has_text=\"Issues 40\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "filtered-search-term-input",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_test_id(\"filtered-search-term-input\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/issues"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/issues"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "filtered-search-token-segment",
|
||
"idx": 3,
|
||
"action_repr": "frame.pressget_by_test_id(\"filtered-search-token-segment\").get_by_label(\"Search\")Enter",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/issues"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/issues"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_[Bug] 404s, bad host, timeouts, bad urls for URLs linked from website",
|
||
"idx": 4,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"[Bug] 404s, bad host, timeouts, bad urls for URLs linked from website\")",
|
||
"before": {
|
||
"url": "http://metis.lti.cs.cmu.edu:8023/a11yproject/a11yproject.com/-/issues/1478"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/issues/?search=404&sort=updated_desc&state=opened&first_page_size=20"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Sign in / Register",
|
||
"idx": 5,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Sign in / Register\")",
|
||
"before": {
|
||
"url": "http://metis.lti.cs.cmu.edu:8023/users/sign_in?redirect_to_referer=yes"
|
||
},
|
||
"after": {
|
||
"url": "http://metis.lti.cs.cmu.edu:8023/users/sign_in?redirect_to_referer=yes"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "username-field",
|
||
"idx": 7,
|
||
"action_repr": "frame.pressget_by_test_id(\"username-field\")Tab",
|
||
"before": {
|
||
"url": "http://metis.lti.cs.cmu.edu:8023/users/sign_in?redirect_to_referer=yes"
|
||
},
|
||
"after": {
|
||
"url": "http://metis.lti.cs.cmu.edu:8023/users/sign_in?redirect_to_referer=yes"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "password-field",
|
||
"idx": 8,
|
||
"action_repr": "frame.clickget_by_test_id(\"password-field\")",
|
||
"before": {
|
||
"url": "http://metis.lti.cs.cmu.edu:8023/users/sign_in?redirect_to_referer=yes"
|
||
},
|
||
"after": {
|
||
"url": "http://metis.lti.cs.cmu.edu:8023/users/sign_in?redirect_to_referer=yes"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |