232 lines
12 KiB
Plaintext
232 lines
12 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
How many commits did kilian make on 3/5/2023?
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action:** I hover over the "Security & Compliance" menu item in the sidebar.
|
||
- **Page Changes:** The "Security & Compliance" menu item is highlighted, indicating it is being hovered over.
|
||
- **Possible Purpose:** The likely intent is to either select this menu item or explore its sub-options.
|
||
|
||
2. **Action:** I click on the "Security & Compliance" menu item in the sidebar.
|
||
- **Page Changes:** The page transitions to the "Security & Compliance" section of the project. The content specific to security and compliance features is displayed.
|
||
- **Possible Purpose:** The purpose is to navigate to the "Security & Compliance" section to view or manage security-related features and settings for the project.
|
||
|
||
3. **Action:** I scroll down slightly on the "Security & Compliance" page.
|
||
- **Page Changes:** The visible content on the page shifts downward, revealing more information related to security and compliance features.
|
||
- **Possible Purpose:** The intent is to review additional details or options available under the "Security & Compliance" section that were not initially visible.
|
||
|
||
4. **Action:** I hover over the "CI/CD" menu item in the sidebar.
|
||
- **Page Changes:** The "CI/CD" menu item is highlighted, indicating it is being hovered over.
|
||
- **Possible Purpose:** The likely intent is to either select this menu item or explore its sub-options.
|
||
|
||
5. **Action:** I click on the "CI/CD" menu item in the sidebar.
|
||
- **Page Changes:** The page transitions to the "CI/CD" section of the project. The content specific to CI/CD pipelines and configurations is displayed.
|
||
- **Possible Purpose:** The purpose is to navigate to the "CI/CD" section to view or manage continuous integration and continuous deployment settings for the project.
|
||
|
||
6. **Action:** I scroll down slightly on the "CI/CD" page.
|
||
- **Page Changes:** The visible content on the page shifts downward, revealing more information related to CI/CD pipelines and configurations.
|
||
- **Possible Purpose:** The intent is to review additional details or options available under the "CI/CD" section that were not initially visible.
|
||
|
||
### Summary:
|
||
In this video segment, I navigated from the main project page to the "Security & Compliance" section and then to the "CI/CD" section. Each navigation involved hovering over and clicking the respective menu items in the sidebar, followed by slight scrolling to view more content within each section. The actions suggest a focus on reviewing and potentially managing security, compliance, and CI/CD features for the project.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action:** I click on the "Search by author" text box.
|
||
- **Page Changes:** The text box becomes active, and a dropdown menu appears below it, showing suggestions or previous entries.
|
||
- **Possible Purpose:** My intention is to search for commits by a specific author, which requires me to input the author's name into the search field.
|
||
|
||
2. **Action:** I type "killian" into the "Search by author" text box.
|
||
- **Page Changes:** As I type, the dropdown menu updates to show relevant suggestions based on the entered text. The suggestions include options like "Byte Blaze," "Roshan Jossey," "The A11Y Project," and "Any Author."
|
||
- **Possible Purpose:** By typing "killian," I am narrowing down the search to find commits authored by someone with that name. This helps in filtering the commit list to only those relevant to "killian."
|
||
|
||
3. **Action:** I pause typing and observe the dropdown menu.
|
||
- **Page Changes:** The dropdown remains open, displaying the filtered suggestions based on the typed text "killian."
|
||
- **Possible Purpose:** I am reviewing the suggestions to determine if "killian" appears as an exact match or if I need to refine my search further.
|
||
|
||
### Summary of Observations:
|
||
- **Primary Focus:** The primary focus of these actions is to utilize the "Search by author" feature to locate commits associated with a specific author named "killian."
|
||
- **User Interaction:** The interaction involves clicking into a text box, typing a name, and observing the resulting suggestions.
|
||
- **Outcome:** The outcome is a filtered list of author suggestions, though "killian" does not appear as an exact match in the provided suggestions. This suggests either a need for further refinement of the search term or acknowledgment that "killian" may not be among the listed authors.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action:** I click on the "Author" filter text box.
|
||
- **Page Changes:** The text box becomes active, and a placeholder text inside it reads "Search by message".
|
||
- **Possible Purpose:** The likely intent is to filter the commit list by specifying an author's name or part of a name to narrow down the displayed commits.
|
||
|
||
2. **Action:** I type "kilian" into the active "Author" filter text box.
|
||
- **Page Changes:** As I type, the commit list updates in real-time to show only the commits authored by someone with "kilian" in their name. One commit appears in the filtered list, authored by "Kilian Valkhof".
|
||
- **Possible Purpose:** The purpose is to locate specific commits made by an author whose name includes "kilian". This helps in reviewing or managing contributions from that particular author.
|
||
|
||
3. **Action:** I hover over the highlighted commit in the filtered list.
|
||
- **Page Changes:** A tooltip appears, showing the location identifier `location="#commit-21356486"`.
|
||
- **Possible Purpose:** Hovering over the commit likely aims to gather more information about it, such as its unique identifier, which can be useful for referencing or linking to the specific commit.
|
||
|
||
### Summary:
|
||
In this video segment, I interact with the "Author" filter to find commits by a specific author. I activate the filter, type "kilian", and observe the filtered results. I then hover over the resulting commit to view additional details, likely for further inspection or reference. Each action is performed to efficiently locate and examine relevant commits within the repository.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_2,320 Commits",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"2,320 Commits\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Author",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Author\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_2",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Search\", exact=True)",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "text_14 Mar, 2023 2 commits",
|
||
"idx": 3,
|
||
"action_repr": "frame.clickget_by_text(\"14 Mar, 2023 2 commits\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/commits/main"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_4",
|
||
"idx": 4,
|
||
"action_repr": "frame.clickget_by_placeholder(\"Search by message\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/commits/main"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/commits/main"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_5",
|
||
"idx": 5,
|
||
"action_repr": "frame.pressget_by_placeholder(\"Search by message\")Enter",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/commits/main"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:8023/a11yproject/a11yproject.com/-/commits/main"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |