219 lines
11 KiB
Plaintext
219 lines
11 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Find the customer name and email with phone number 8015551212
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Step-by-Step Actions:
|
||
|
||
1. **Action:** I click on the "Customers" menu item in the left sidebar.
|
||
- **Page Changes:** The page transitions from the "Dashboard" view to the "Customers" section. A modal window titled "Customers" appears, displaying options such as "All Customers," "New Online," "Login as Customer Log," and "Customer Groups."
|
||
- **Possible Purpose:** The likely intent is to navigate to the customer management section to view or manage customer-related data.
|
||
|
||
2. **Action:** I hover over the "Customers" modal window.
|
||
- **Page Changes:** No significant changes occur on the page; the modal remains open with the same options visible.
|
||
- **Possible Purpose:** The purpose of this action could be to review the available options under the "Customers" section before making a selection.
|
||
|
||
3. **Action:** I click on the "All Customers" option within the "Customers" modal window.
|
||
- **Page Changes:** The modal window closes, and the main content area updates to display the "All Customers" page. This page lists all customers with details such as their names, items purchased, and total spending.
|
||
- **Possible Purpose:** The intent is to access a comprehensive list of all customers to view detailed information about each customer's transactions and activity.
|
||
|
||
4. **Action:** I scroll down slightly on the "All Customers" page.
|
||
- **Page Changes:** The visible portion of the page shifts downward, revealing more customer entries in the list.
|
||
- **Possible Purpose:** The purpose of scrolling is to view additional customer information that was initially out of sight.
|
||
|
||
### Summary:
|
||
- **Initial State:** The video starts with the "Dashboard" page open.
|
||
- **Sequence of Actions:**
|
||
1. Navigate to the "Customers" section via the sidebar.
|
||
2. Review the options in the "Customers" modal.
|
||
3. Select "All Customers" to view the full list of customers.
|
||
4. Scroll through the list of customers to see more entries.
|
||
- **Final State:** The "All Customers" page is displayed, showing a detailed list of customers with their respective transaction data.
|
||
|
||
This sequence of actions suggests a focus on accessing and reviewing customer data for management or analysis purposes.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the search bar located at the top of the customer list.
|
||
- **Page Changes:** The search bar becomes active, and a blinking cursor appears inside it, indicating that it is ready for input.
|
||
- **Possible Purpose:** The likely intent is to enter a specific keyword or identifier to filter the customer list.
|
||
|
||
#### 2. **Action:** I type "8015551212" into the search bar.
|
||
- **Page Changes:** As I type, the text "8015551212" appears in the search bar. After completing the entry, the page updates to show a filtered list of customers matching the search term. A message indicates "70 records found."
|
||
- **Possible Purpose:** The purpose is to locate customers associated with the phone number "8015551212" by filtering the customer list based on this specific keyword.
|
||
|
||
#### 3. **Action:** I hover over the search bar after typing the keyword.
|
||
- **Page Changes:** No immediate changes occur on the page as a result of hovering; however, a tooltip or suggestion box appears, displaying a script-like text: `get_by_role("textbox", name="Search by keyword").get_by_role("button", name="Search")`.
|
||
- **Possible Purpose:** The hovering action might be intended to inspect or verify the search functionality, possibly for debugging or instructional purposes.
|
||
|
||
#### 4. **Action:** I click the magnifying glass icon (search button) next to the search bar.
|
||
- **Page Changes:** The page confirms the search query, and the results are displayed with the same "70 records found" message. The search term "8015551212" remains visible in the search bar.
|
||
- **Possible Purpose:** The action reinforces the search query to ensure the results are updated or to trigger the search explicitly if needed.
|
||
|
||
#### 5. **Action:** I scroll down slightly on the page.
|
||
- **Page Changes:** The customer list scrolls down, revealing more entries that match the search criteria.
|
||
- **Possible Purpose:** The scrolling is likely to review additional customer records that were not initially visible on the screen.
|
||
|
||
#### 6. **Action:** I hover over a specific customer entry in the list.
|
||
- **Page Changes:** Hovering over the entry highlights it, making it visually distinct from other entries. A tooltip or additional information may appear, though it is not explicitly shown in the video.
|
||
- **Possible Purpose:** The purpose could be to select or inspect details of a particular customer record for further actions or verification.
|
||
|
||
#### 7. **Action:** I click on the "Actions" dropdown menu in the customer list header.
|
||
- **Page Changes:** The "Actions" dropdown menu expands, showing various options such as "Mass Action" choices for the selected customers.
|
||
- **Possible Purpose:** The intent is likely to perform a bulk action on the selected customer records, such as exporting, deleting, or updating them.
|
||
|
||
#### 8. **Action:** I move the cursor over different elements on the page, including the "Filters," "Default View," "Columns," and "Export" buttons.
|
||
- **Page Changes:** Hovering over these elements may highlight them or display tooltips, but no specific actions are triggered.
|
||
- **Possible Purpose:** The cursor movement suggests an inspection or evaluation of available options for managing or customizing the customer list view.
|
||
|
||
### Summary
|
||
The observed actions focus on searching for specific customers using a phone number, reviewing the search results, and exploring options for managing the customer list. The steps are methodical, aimed at locating and potentially performing actions on relevant customer records.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_\ue603 Customers",
|
||
"idx": 5,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"\ue603 Customers\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/admin/dashboard/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/admin/dashboard/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_The information in this tab has been changed. This tab contains invalid data. Please resolve this before saving. Loading... Customers",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"The information in this tab has been changed. This tab contains invalid data. Please resolve this before saving. Loading... Customers\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/admin/dashboard/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/admin/dashboard/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_All Customers",
|
||
"idx": 6,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"All Customers\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/customer/index/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/customer/index/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "textbox_Search by keyword",
|
||
"idx": 7,
|
||
"action_repr": "frame.clickget_by_role(\"textbox\", name=\"Search by keyword\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/customer/index/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/customer/index/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Search",
|
||
"idx": 8,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Search\")",
|
||
"before": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/customer/index/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-133-227-75.us-east-2.compute.amazonaws.com:7780/admin/customer/index/"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |