440 lines
18 KiB
Plaintext
440 lines
18 KiB
Plaintext
# Instruction
|
||
- You are an expert in cleaning process data descriptions. Given a task, you are provided with a set of annotation description
|
||
data for a certain visual LLM related to human user operation videos. Plus, You are provided with full trace of playwright action,
|
||
whic includes action and url before and after the action.
|
||
- You need to analyze all the descriptive data and ultimately summarize a complete and reasonable user operation description that can accomplish the given task.
|
||
- For each strategy, give a clear list of the low level action sequence.
|
||
|
||
# Task
|
||
Get the date of the most recent canlled order
|
||
|
||
# Annotation description
|
||
## Part 1
|
||
### Part 1: Initial State - Dashboard Page
|
||
|
||
**Action:**
|
||
I hover over the "Reports" menu item in the left sidebar.
|
||
|
||
**Page Changes:**
|
||
No immediate changes occur on the page; the dashboard remains visible with its current content.
|
||
|
||
**Possible Purpose:**
|
||
The likely intent is to access the reports section to view or generate specific reports related to sales, orders, or other business metrics.
|
||
|
||
---
|
||
|
||
### Part 2: Accessing the Reports Menu
|
||
|
||
**Action:**
|
||
I click on the "Reports" menu item in the left sidebar.
|
||
|
||
**Page Changes:**
|
||
The page transitions from the dashboard to the "Reports" overview page. The new page displays various report categories such as "Marketing," "Sales," "Customers," and "Business Intelligence."
|
||
|
||
**Possible Purpose:**
|
||
The purpose is to navigate to the reports section to select a specific type of report for detailed analysis.
|
||
|
||
---
|
||
|
||
### Part 3: Selecting the Orders Report
|
||
|
||
**Action:**
|
||
I click on the "Orders" link under the "Sales" category in the Reports menu.
|
||
|
||
**Page Changes:**
|
||
The page updates to display the "Orders Report" interface. This new page includes filter options such as "Date Used," "Period," "Order Status," and buttons like "Show Report."
|
||
|
||
**Possible Purpose:**
|
||
The intent is to access the Orders Report to analyze order data based on specific criteria.
|
||
|
||
---
|
||
|
||
### Part 4: Setting the Date Range
|
||
|
||
**Action:**
|
||
I click on the "From" date field under the "Filter" section.
|
||
|
||
**Page Changes:**
|
||
A date picker calendar appears, allowing me to select a start date for the report.
|
||
|
||
**Possible Purpose:**
|
||
The purpose is to specify the beginning date of the desired date range for the orders report.
|
||
|
||
---
|
||
|
||
### Part 5: Choosing the Start Date
|
||
|
||
**Action:**
|
||
I click on a specific date (e.g., September 19, 2023) in the date picker calendar.
|
||
|
||
**Page Changes:**
|
||
The selected date is populated in the "From" date field.
|
||
|
||
**Possible Purpose:**
|
||
The intent is to set the start date for the report to filter orders created from this specific date onward.
|
||
|
||
---
|
||
|
||
### Part 6: Setting the End Date
|
||
|
||
**Action:**
|
||
I click on the "To" date field under the "Filter" section.
|
||
|
||
**Page Changes:**
|
||
Another date picker calendar appears, enabling me to select an end date for the report.
|
||
|
||
**Possible Purpose:**
|
||
The purpose is to specify the ending date of the desired date range for the orders report.
|
||
|
||
---
|
||
|
||
### Part 7: Choosing the End Date
|
||
|
||
**Action:**
|
||
I click on a specific date (e.g., September 23, 2023) in the date picker calendar.
|
||
|
||
**Page Changes:**
|
||
The selected date is populated in the "To" date field.
|
||
|
||
**Possible Purpose:**
|
||
The intent is to set the end date for the report to filter orders created up to this specific date.
|
||
|
||
---
|
||
|
||
### Part 8: Showing the Report
|
||
|
||
**Action:**
|
||
I click the "Show Report" button.
|
||
|
||
**Page Changes:**
|
||
The page processes the filter criteria and displays the orders report based on the selected date range. The report includes details such as order numbers, dates, statuses, and totals.
|
||
|
||
**Possible Purpose:**
|
||
The purpose is to generate and view the orders report for the specified date range to analyze order data within that period.
|
||
|
||
---
|
||
|
||
### Summary of Actions:
|
||
1. Hover over "Reports" in the sidebar.
|
||
2. Click "Reports" to navigate to the Reports page.
|
||
3. Click "Orders" under the "Sales" category to access the Orders Report.
|
||
4. Click the "From" date field to open the date picker.
|
||
5. Select a start date from the date picker.
|
||
6. Click the "To" date field to open the date picker.
|
||
7. Select an end date from the date picker.
|
||
8. Click "Show Report" to generate and display the orders report for the specified date range.
|
||
|
||
---
|
||
|
||
## Part 2
|
||
### Part 1: Initial State and First Actions
|
||
|
||
#### Step 1: Opening the Date Picker
|
||
- **Action**: I click on the calendar icon next to the "From" date field under the "Filter" section.
|
||
- **Page Changes**: A date picker modal appears, displaying a calendar for September 2023.
|
||
- **Possible Purpose**: The likely intent is to select a specific start date for the report filter.
|
||
|
||
#### Step 2: Navigating the Date Picker
|
||
- **Action**: I hover over different dates in the calendar.
|
||
- **Page Changes**: The hovered dates are highlighted, but no selection is made yet.
|
||
- **Possible Purpose**: I am reviewing available dates to choose an appropriate start date for the report.
|
||
|
||
#### Step 3: Selecting a Date
|
||
- **Action**: I click on September 23, 2023, in the date picker.
|
||
- **Page Changes**: The selected date, "09/23/2023," is populated in the "From" date field.
|
||
- **Possible Purpose**: I have chosen September 23, 2023, as the starting point for the report's date range.
|
||
|
||
#### Step 4: Closing the Date Picker
|
||
- **Action**: I click the "Close" button on the date picker modal.
|
||
- **Page Changes**: The date picker modal closes, and the focus returns to the main filter section.
|
||
- **Possible Purpose**: To finalize the date selection and proceed with setting other filter parameters.
|
||
|
||
#### Step 5: Selecting the "Order Status"
|
||
- **Action**: I click on the "Order Status" dropdown menu.
|
||
- **Page Changes**: A list of order statuses appears, including options like "Processing," "Complete," "Closed," etc.
|
||
- **Possible Purpose**: I intend to specify the order status to refine the report's criteria.
|
||
|
||
#### Step 6: Choosing an Order Status
|
||
- **Action**: I select "Canceled" from the order status dropdown.
|
||
- **Page Changes**: The "Canceled" status is now displayed in the "Order Status" field.
|
||
- **Possible Purpose**: I want to generate a report specifically for canceled orders.
|
||
|
||
### Summary of Part 1:
|
||
In this segment, I focused on setting up the filter criteria for an orders report. Specifically, I selected a start date of September 23, 2023, and chose "Canceled" as the order status. These actions suggest that my goal is to create a report that analyzes canceled orders from the specified date onward. No other parts of the video are considered in this description.
|
||
|
||
---
|
||
|
||
## Part 3
|
||
### Step-by-Step Actions in the Video Segment
|
||
|
||
#### 1. **Action:** I click on the "Show Report" button.
|
||
- **Page Changes:** The page transitions to display a detailed orders report table. This table includes columns for dates, quantities, and various monetary values related to orders.
|
||
- **Possible Purpose:** The likely intent is to generate and view a comprehensive report of order data for analysis or record-keeping.
|
||
|
||
#### 2. **Action:** I hover over the "Show Report" button again.
|
||
- **Page Changes:** There are no changes to the page content; it remains displaying the orders report.
|
||
- **Possible Purpose:** This action might be to ensure the button's functionality or to reaffirm the action just taken.
|
||
|
||
#### 3. **Action:** I use the browser's developer tools to select an element within the orders report table.
|
||
- **Page Changes:** A small blue box highlights a specific cell in the table, indicating the selected element. No other changes occur on the page.
|
||
- **Possible Purpose:** The purpose is likely to inspect the properties or structure of the selected element, possibly for debugging, customization, or understanding the layout of the report.
|
||
|
||
#### 4. **Action:** I scroll down slightly within the orders report table.
|
||
- **Page Changes:** The visible portion of the table updates to show later entries in the report, with the header remaining fixed at the top.
|
||
- **Possible Purpose:** The intent is to review additional data in the report that was not initially visible.
|
||
|
||
#### 5. **Action:** I use the developer tools to select another element, this time targeting a cell containing a monetary value ("$0.00").
|
||
- **Page Changes:** The developer tools highlight the targeted cell, showing its specific details such as class names and styling.
|
||
- **Possible Purpose:** This action aims to examine the formatting or properties of the monetary value cells, perhaps to address styling, alignment, or data display issues.
|
||
|
||
### Summary
|
||
In this video segment, my actions focus on generating and reviewing an orders report, ensuring its visibility and functionality, and using developer tools to inspect specific elements within the report table. Each step appears methodical, aimed at verifying the report's content and presentation.
|
||
|
||
# Playwright action
|
||
[
|
||
{
|
||
"action_uid": "link_\ue602 Content",
|
||
"idx": 0,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"\ue602 Content\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/admin/dashboard/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/admin/dashboard/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_\ue609 Marketing",
|
||
"idx": 1,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"\ue609 Marketing\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/admin/dashboard/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/admin/dashboard/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_\ue60a Reports",
|
||
"idx": 2,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"\ue60a Reports\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/admin/dashboard/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/admin/dashboard/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_Orders",
|
||
"idx": 3,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"Orders\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "text_Order Created Order Updated The Order Updated report is created in real time and",
|
||
"idx": 4,
|
||
"action_repr": "frame.clickget_by_text(\"Order Created Order Updated The Order Updated report is created in real time and\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_5",
|
||
"idx": 5,
|
||
"action_repr": "frame.clicklocator(\"#sales_report_base_fieldset div\").filter(has_text=\"From undefined\").get_by_role(\"button\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "text_PreviousNextJanFebMarAprMayJunJulAugSepOctNovDec19231924192519261927192819291930",
|
||
"idx": 6,
|
||
"action_repr": "frame.clickget_by_text(\"PreviousNextJanFebMarAprMayJunJulAugSepOctNovDec19231924192519261927192819291930\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "label_Select year",
|
||
"idx": 7,
|
||
"action_repr": "frame.selectOptionget_by_label(\"Select year\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "link_6",
|
||
"idx": 8,
|
||
"action_repr": "frame.clickget_by_role(\"link\", name=\"6\", exact=True)",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_9",
|
||
"idx": 9,
|
||
"action_repr": "frame.clicklocator(\"#sales_report_base_fieldset div\").filter(has_text=\"To undefined\").get_by_role(\"button\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Go Today",
|
||
"idx": 11,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Go Today\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "text_To undefined",
|
||
"idx": 12,
|
||
"action_repr": "frame.clickget_by_text(\"To undefined\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "action_13",
|
||
"idx": 13,
|
||
"action_repr": "frame.selectOptionget_by_label(\"Order Status\", exact=True)",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "group_Filter",
|
||
"idx": 14,
|
||
"action_repr": "frame.selectOptionget_by_role(\"group\", name=\"Filter\").get_by_label(\"\", exact=True)",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
}
|
||
},
|
||
{
|
||
"action_uid": "button_Show Report",
|
||
"idx": 15,
|
||
"action_repr": "frame.clickget_by_role(\"button\", name=\"Show Report\")",
|
||
"before": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
},
|
||
"after": {
|
||
"url": "http://ec2-3-135-39-80.us-east-2.compute.amazonaws.com:7780/admin/reports/report_sales/sales/"
|
||
}
|
||
}
|
||
]
|
||
|
||
# Output format
|
||
- 先总结整个任务的Objective，然后按照Strategy-SubStrategy-action三级层次来给出整个过程，
|
||
- 接着给出整个操作流程后的观察和有趣的发现，最后严格按照json格式输出三级层次的过程描述。
|
||
- 最后的输出json应该是包在```{json}```之间，最底层动作需要包含描述、对应的playwright动作指令顺序编号，以及具体指令内容。
|
||
|
||
# Example
|
||
### Complete User Operation Description to Display Labeled Issues in kkroening/ffmpeg-python
|
||
|
||
**Objective:** Filter and display all issues labeled as "question" in the kkroening/ffmpeg-python repository.
|
||
|
||
---
|
||
|
||
#### **Strategy 1: Navigate to the Repository**
|
||
**Low-Level Action Sequence:**
|
||
1. **Search for the user "kkroening"**
|
||
- Click the global search bar (placeholder: "Search GitLab").
|
||
- Type "kkroening" and press `Enter`.
|
||
2. **Select the user from results**
|
||
- Click the "Users" tab in search results.
|
||
- Click on "Karl Kroening @kkroening" in the user list.
|
||
3. **Access the repository**
|
||
- Navigate to the "Personal projects" section.
|
||
- Click on the "ffmpeg-python" project.
|
||
|
||
---
|
||
|
||
#### **Strategy 2: Filter Issues by Label**
|
||
**Low-Level Action Sequence:**
|
||
1. **Open the Issues tab**
|
||
- Scroll to the left sidebar menu.
|
||
- Click the "Issues" tab (displaying the count, e.g., "Issues 402").
|
||
2. **Apply label filtering**
|
||
- Click the search/filter bar in the issues list.
|
||
- Select the "Label" dropdown from the filter options.
|
||
- Type or select "question" from the label dropdown.
|
||
- Click the search/apply button to confirm the filter.
|
||
|
||
---
|
||
|
||
#### **Final Oberservation**
|
||
The issues list will refresh to show only issues with the "question" label. The URL will reflect the filter:
|
||
`.../ffmpeg-python/-/issues/?label_name[]=question`.
|
||
|
||
---
|
||
|
||
### Key Observations from Playwright Trace
|
||
- The final URL after filtering:
|
||
`http://ec2-3-135-39-80.../ffmpeg-python/-/issues/?label_name%5B%5D=question`
|
||
confirms the "question" label filter is applied.
|
||
- Critical interactions include selecting the "Label" dropdown and explicitly choosing "question" to refine results.
|
||
|
||
### Final output
|
||
```json
|
||
[{
|
||
"strategy" : "Navigate to the Repository",
|
||
"substrategies": [
|
||
{
|
||
"substrategy": "Search for the user \"kkroening\"",
|
||
"actions" : [
|
||
{
|
||
"description": "Click the global search bar (placeholder: \"Search GitLab\"). ",
|
||
"playwright_idx" : 18,
|
||
"playwright_instruction" : "frame.pressget_by_placeholder(\"Search GitLab\")Enter"
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"substrategy": "Select the user from results",
|
||
"actions" : [
|
||
]
|
||
}
|
||
]
|
||
},
|
||
{
|
||
"strategy" : "Filter Issues by Label",
|
||
"substrategies" : [
|
||
]
|
||
}]
|
||
``` |