refactor: 重构项目结构并优化类型定义
- 移除extension模块，将功能迁移至node-agent - 修复类型导出问题，使用export type明确类型导出 - 统一数据库连接方式，从直接导入改为使用config/database - 更新文档中的项目结构描述 - 添加多个服务的实用方法，如getForecast、getBalances等 - 修复类型错误和TS1205警告 - 优化RedisService调用方式 - 添加新的实体类型定义 - 更新审计日志格式，统一字段命名
This commit is contained in:
@@ -1,7 +1,21 @@
|
||||
# 插件文档索引
|
||||
# Node Agent 文档索引
|
||||
|
||||
> **模块**: 04_Plugin - 浏览器插件架构
|
||||
> **更新日期**: 2026-03-19
|
||||
> **模块**: 04_Plugin → Node Agent (Playwright 自动化)
|
||||
> **更新日期**: 2026-03-20
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 架构变更说明
|
||||
|
||||
**Extension (浏览器插件) 已废弃**，由 **Node Agent (Playwright 自动化)** 替代。
|
||||
|
||||
| 对比项 | Extension (旧) | Node Agent (新) |
|
||||
|--------|---------------|-----------------|
|
||||
| 运行环境 | 浏览器内 | 独立进程 |
|
||||
| 自动化引擎 | Chrome Extension API | Playwright |
|
||||
| 反检测能力 | 受限 | 完整指纹控制 |
|
||||
| 并发能力 | 单标签 | 多浏览器实例 |
|
||||
| 任务调度 | 简单消息 | Hub 拉取模式 |
|
||||
|
||||
---
|
||||
|
||||
@@ -9,18 +23,35 @@
|
||||
|
||||
| 文档 | 描述 | 状态 |
|
||||
|------|------|------|
|
||||
| [01_Plugin_Design](01_Plugin_Design.md) | 插件架构设计 | ✅ |
|
||||
| [01_NodeAgent_Design](01_NodeAgent_Design.md) | Node Agent 架构设计 | ✅ |
|
||||
| [02_DOM_Interaction](02_DOM_Interaction.md) | DOM交互规范 | ✅ |
|
||||
|
||||
---
|
||||
|
||||
## 架构图
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
||||
│ Server │◄────►│ Hub │◄────►│ Node-Agent │
|
||||
│ (主控端) │ │ (任务调度) │ │ (Playwright)│
|
||||
└─────────────┘ └─────────────┘ └─────────────┘
|
||||
│
|
||||
┌──────▼──────┐
|
||||
│ Chromium │
|
||||
│ (无API平台) │
|
||||
└─────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 关联模块
|
||||
|
||||
- [架构模块](../01_Architecture/00_Architecture_Index.md)
|
||||
- [前端模块](../03_Frontend/00_Frontend_Index.md)
|
||||
- [后端模块](../02_Backend/00_Backend_Index.md)
|
||||
|
||||
---
|
||||
|
||||
## 最近更新
|
||||
|
||||
- 2026-03-20: Extension 废弃，迁移至 Node Agent
|
||||
- 2026-03-19: 重构插件文档结构，统一命名规范
|
||||
|
||||
263
docs/04_Plugin/01_NodeAgent_Design.md
Normal file
263
docs/04_Plugin/01_NodeAgent_Design.md
Normal file
@@ -0,0 +1,263 @@
|
||||
# Node Agent 设计文档
|
||||
|
||||
> **定位**: Win Node Agent - 无 API 平台执行代理
|
||||
> **更新日期**: 2026-03-20
|
||||
> **最高优先级参考**: [Business_ClosedLoops.md](../00_Business/Business_ClosedLoops.md)
|
||||
|
||||
---
|
||||
|
||||
## 1. 概述
|
||||
|
||||
### 1.1 背景
|
||||
|
||||
Extension (浏览器插件) 方案存在以下局限性：
|
||||
- Manifest V3 权限限制
|
||||
- 反检测能力不足
|
||||
- 无法多实例并发
|
||||
- 任务调度不可靠
|
||||
|
||||
**Node Agent** 基于 Playwright 构建，提供更强大的自动化能力。
|
||||
|
||||
### 1.2 核心能力
|
||||
|
||||
| 能力 | 说明 |
|
||||
|------|------|
|
||||
| **无 API 平台采集** | TikTok Shop, Temu, 1688 等 |
|
||||
| **自动化操作** | 刊登、订单处理、广告投放 |
|
||||
| **反检测** | 指纹隔离、代理 IP、浏览器上下文隔离 |
|
||||
| **多实例并发** | 支持多店铺同时运行 |
|
||||
|
||||
---
|
||||
|
||||
## 2. 技术栈
|
||||
|
||||
| 层级 | 技术 | 版本 | 用途 |
|
||||
|------|------|------|------|
|
||||
| **Runtime** | Node.js | 18+ | 运行环境 |
|
||||
| **Automation** | Playwright | 1.58+ | 浏览器自动化 |
|
||||
| **Language** | TypeScript | 5.x | 开发语言 |
|
||||
| **Communication** | Axios | - | HTTP 通信 |
|
||||
|
||||
---
|
||||
|
||||
## 3. 架构设计
|
||||
|
||||
### 3.1 系统架构
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Crawlful Hub │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ Server │ │ Hub │ │ Dashboard │ │
|
||||
│ │ (业务逻辑) │◄───►│ (任务调度) │◄───►│ (控制台) │ │
|
||||
│ └──────────────┘ └──────┬───────┘ └──────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────┐ │
|
||||
│ │ Node Agent │ │
|
||||
│ │ (Playwright) │ │
|
||||
│ └────────┬─────────┘ │
|
||||
│ │ │
|
||||
│ ┌───────────────────┼───────────────────┐ │
|
||||
│ ▼ ▼ ▼ │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ Browser #1 │ │ Browser #2 │ │ Browser #N │ │
|
||||
│ │ (店铺A) │ │ (店铺B) │ │ (店铺N) │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 3.2 核心类设计
|
||||
|
||||
```typescript
|
||||
export class NodeAgent {
|
||||
private hubUrl: string;
|
||||
private nodeId: string;
|
||||
private heartbeatInterval: number = 30000;
|
||||
private pollInterval: number = 10000;
|
||||
|
||||
async start(): Promise<void>;
|
||||
async register(): Promise<void>;
|
||||
async heartbeat(): Promise<void>;
|
||||
async pollTasks(): Promise<void>;
|
||||
async executeTask(task: NodeTask): Promise<void>;
|
||||
async reportReceipt(receipt: any): Promise<void>;
|
||||
}
|
||||
```
|
||||
|
||||
### 3.3 任务类型
|
||||
|
||||
```typescript
|
||||
export enum TaskType {
|
||||
COLLECT_PRODUCT = 'COLLECT_PRODUCT',
|
||||
COLLECT_ORDER = 'COLLECT_ORDER',
|
||||
PUBLISH_PRODUCT = 'PUBLISH_PRODUCT',
|
||||
PROCESS_ORDER = 'PROCESS_ORDER',
|
||||
SYNC_INVENTORY = 'SYNC_INVENTORY',
|
||||
MANAGE_AD = 'MANAGE_AD',
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 核心功能
|
||||
|
||||
### 4.1 节点注册
|
||||
|
||||
```typescript
|
||||
private async register() {
|
||||
await axios.post(`${this.hubUrl}/api/v1/nodes/register`, {
|
||||
nodeId: this.nodeId,
|
||||
version: '1.0.0',
|
||||
os: os.type(),
|
||||
hostname: os.hostname(),
|
||||
shops: []
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
### 4.2 任务轮询
|
||||
|
||||
```typescript
|
||||
private async pollTasks() {
|
||||
const response = await axios.get(
|
||||
`${this.hubUrl}/api/v1/nodes/pull-task?nodeId=${this.nodeId}`
|
||||
);
|
||||
|
||||
if (response.data?.task) {
|
||||
await this.executeTask(response.data.task);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 4.3 任务执行
|
||||
|
||||
```typescript
|
||||
private async executeTask(task: NodeTask) {
|
||||
try {
|
||||
const browser = await chromium.launch({
|
||||
headless: false,
|
||||
proxy: { server: task.proxyUrl }
|
||||
});
|
||||
|
||||
const context = await browser.newContext({
|
||||
viewport: { width: 1920, height: 1080 }
|
||||
});
|
||||
|
||||
const page = await context.newPage();
|
||||
|
||||
// 执行具体任务...
|
||||
|
||||
await browser.close();
|
||||
|
||||
await this.reportReceipt({
|
||||
taskId: task.id,
|
||||
traceId: task.traceId,
|
||||
status: 'success',
|
||||
resultData: {}
|
||||
});
|
||||
} catch (err) {
|
||||
await this.reportReceipt({
|
||||
taskId: task.id,
|
||||
traceId: task.traceId,
|
||||
status: 'failed',
|
||||
errorMessage: err.message
|
||||
});
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 店铺隔离策略
|
||||
|
||||
### 5.1 一店一上下文
|
||||
|
||||
```typescript
|
||||
interface ShopContext {
|
||||
shopId: string;
|
||||
profileDir: string;
|
||||
proxy: ProxyConfig;
|
||||
fingerprintPolicy: FingerprintConfig;
|
||||
}
|
||||
```
|
||||
|
||||
### 5.2 同店任务串行
|
||||
|
||||
```typescript
|
||||
class TaskQueue {
|
||||
private queues: Map<string, TaskQueue> = new Map();
|
||||
|
||||
async addTask(shopId: string, task: Task) {
|
||||
if (!this.queues.has(shopId)) {
|
||||
this.queues.set(shopId, new TaskQueue());
|
||||
}
|
||||
await this.queues.get(shopId).add(task);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. 反检测策略
|
||||
|
||||
| 策略 | 说明 |
|
||||
|------|------|
|
||||
| **指纹隔离** | 每个店铺独立浏览器指纹 |
|
||||
| **代理 IP** | 每个店铺独立代理 |
|
||||
| **行为模拟** | 随机延迟、鼠标轨迹 |
|
||||
| **User-Agent** | 随机 UA 轮换 |
|
||||
|
||||
---
|
||||
|
||||
## 7. 目录结构
|
||||
|
||||
```
|
||||
node-agent/
|
||||
├── src/
|
||||
│ ├── main.ts # 入口
|
||||
│ ├── index.ts # NodeAgent 类
|
||||
│ ├── tasks/ # 任务处理器
|
||||
│ │ ├── collectProduct.ts
|
||||
│ │ ├── publishProduct.ts
|
||||
│ │ └── processOrder.ts
|
||||
│ ├── platforms/ # 平台适配器
|
||||
│ │ ├── tiktok.ts
|
||||
│ │ ├── temu.ts
|
||||
│ │ └── ali1688.ts
|
||||
│ └── utils/ # 工具函数
|
||||
│ ├── logger.ts
|
||||
│ └── fingerprint.ts
|
||||
├── package.json
|
||||
└── tsconfig.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. 配置
|
||||
|
||||
```typescript
|
||||
interface AgentConfig {
|
||||
hubUrl: string;
|
||||
nodeId: string;
|
||||
heartbeatInterval: number;
|
||||
pollInterval: number;
|
||||
maxConcurrentBrowsers: number;
|
||||
proxyPool: ProxyConfig[];
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. 相关文档
|
||||
|
||||
- [DOM Interaction](02_DOM_Interaction.md)
|
||||
- [Backend Design](../02_Backend/Backend_Design.md)
|
||||
- [Business ClosedLoops](../00_Business/Business_ClosedLoops.md)
|
||||
|
||||
---
|
||||
|
||||
*本文档基于业务闭环设计，最后更新: 2026-03-20*
|
||||
@@ -1,573 +0,0 @@
|
||||
# Plugin Design (Crawlful Hub)
|
||||
|
||||
> **定位**：Crawlful Hub 浏览器插件架构设计文档 - 包含技术栈、目录结构、核心功能及开发规范。
|
||||
> **更新日期**: 2026-03-18
|
||||
> **最高优先级参考**: [Business_ClosedLoops.md](../00_Business/Business_ClosedLoops.md)
|
||||
|
||||
---
|
||||
|
||||
## 1. 技术栈 (Tech Stack)
|
||||
|
||||
| 层级 | 技术 | 版本 | 用途 |
|
||||
|------|------|------|------|
|
||||
| **Framework** | WebExtensions API | MV3 | 浏览器扩展标准 |
|
||||
| **Language** | TypeScript | 5.x | 开发语言 |
|
||||
| **Build Tool** | Vite | 5.x | 构建工具 |
|
||||
| **Bundler** | Rollup | 4.x | 代码打包 |
|
||||
| **UI** | React + Tailwind | 18.x + 3.x | 弹窗/选项页 UI |
|
||||
| **Testing** | Vitest | 1.x | 单元测试 |
|
||||
|
||||
---
|
||||
|
||||
## 2. 目录结构 (Directory Structure)
|
||||
|
||||
```
|
||||
extension/
|
||||
│
|
||||
├─ manifest.json # 扩展清单 (MV3)
|
||||
│
|
||||
├─ src/
|
||||
│ │
|
||||
│ ├─ background/ # Service Worker (后台脚本)
|
||||
│ │ ├─ index.ts # 入口
|
||||
│ │ ├─ handlers/
|
||||
│ │ │ ├─ messageHandler.ts # 消息处理
|
||||
│ │ │ ├─ alarmHandler.ts # 定时任务
|
||||
│ │ │ └─ commandHandler.ts # 快捷键处理
|
||||
│ │ └─ services/
|
||||
│ │ ├─ crawlerService.ts # 网页采集服务（无API平台）
|
||||
│ │ ├─ syncService.ts # 同步服务
|
||||
│ │ └─ authService.ts # 认证服务
|
||||
│ │
|
||||
│ ├─ content/ # 内容脚本 (注入页面)
|
||||
│ │ ├─ index.ts # 入口
|
||||
│ │ ├─ crawlers/ # 采集器
|
||||
│ │ │ ├─ amazonCrawler.ts
|
||||
│ │ │ ├─ ebayCrawler.ts
|
||||
│ │ │ ├─ shopifyCrawler.ts
|
||||
│ │ │ └─ aliexpressCrawler.ts
|
||||
│ │ ├─ automation/ # 自动化操作
|
||||
│ │ │ ├─ listingAutomation.ts
|
||||
│ │ │ ├─ orderAutomation.ts
|
||||
│ │ │ └─ adAutomation.ts
|
||||
│ │ └─ utils/
|
||||
│ │ ├─ domUtils.ts
|
||||
│ │ ├─ selectorUtils.ts
|
||||
│ │ └─ eventUtils.ts
|
||||
│ │
|
||||
│ ├─ popup/ # 弹窗页面
|
||||
│ │ ├─ index.tsx
|
||||
│ │ ├─ components/
|
||||
│ │ │ ├─ QuickActions.tsx
|
||||
│ │ │ ├─ StatusPanel.tsx
|
||||
│ │ │ └─ RecentTasks.tsx
|
||||
│ │ └─ hooks/
|
||||
│ │ └─ useBackground.ts
|
||||
│ │
|
||||
│ ├─ options/ # 选项页面
|
||||
│ │ ├─ index.tsx
|
||||
│ │ ├─ components/
|
||||
│ │ │ ├─ GeneralSettings.tsx
|
||||
│ │ │ ├─ PlatformSettings.tsx
|
||||
│ │ │ ├─ AccountSettings.tsx
|
||||
│ │ │ └─ AdvancedSettings.tsx
|
||||
│ │ └─ stores/
|
||||
│ │ └─ settingsStore.ts
|
||||
│ │
|
||||
│ ├─ shared/ # 共享资源
|
||||
│ │ ├─ types/
|
||||
│ │ │ ├─ messaging.ts # 消息类型定义
|
||||
│ │ │ ├─ crawler.ts # 采集类型
|
||||
│ │ │ └─ platform.ts # 平台类型
|
||||
│ │ ├─ constants/
|
||||
│ │ │ ├─ platforms.ts # 平台常量
|
||||
│ │ │ └─ selectors.ts # 选择器常量
|
||||
│ │ └─ utils/
|
||||
│ │ ├─ logger.ts
|
||||
│ │ ├─ storage.ts
|
||||
│ │ └─ crypto.ts
|
||||
│ │
|
||||
│ └─ injected/ # 注入脚本 (隔离环境)
|
||||
│ ├─ index.ts
|
||||
│ └─ services/
|
||||
│ └─ bridgeService.ts
|
||||
│
|
||||
├─ assets/ # 静态资源
|
||||
│ ├─ icons/
|
||||
│ │ ├─ icon-16.png
|
||||
│ │ ├─ icon-32.png
|
||||
│ │ ├─ icon-48.png
|
||||
│ │ └─ icon-128.png
|
||||
│ └─ styles/
|
||||
│ └─ global.css
|
||||
│
|
||||
├─ _locales/ # 国际化
|
||||
│ ├─ en/
|
||||
│ │ └─ messages.json
|
||||
│ └─ zh_CN/
|
||||
│ └─ messages.json
|
||||
│
|
||||
└─ dist/ # 构建输出
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 架构设计 (Architecture)
|
||||
|
||||
### 3.1 核心组件关系
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Browser Extension │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ Popup │ │ Options │ │ Content │ │
|
||||
│ │ (UI) │◄───►│ (Settings) │◄───►│ (Page) │ │
|
||||
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
|
||||
│ │ │ │ │
|
||||
│ └────────────────────┼────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────┐ │
|
||||
│ │ Service Worker │ │
|
||||
│ │ (Background) │ │
|
||||
│ └────────┬─────────┘ │
|
||||
│ │ │
|
||||
│ ┌───────────────────┼───────────────────┐ │
|
||||
│ ▼ ▼ ▼ │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ Crawler │ │ Sync │ │ Auth │ │
|
||||
│ │ Engine │ │ Engine │ │ Engine │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 3.2 通信机制
|
||||
|
||||
#### 消息类型 (Messaging Types)
|
||||
|
||||
```typescript
|
||||
// src/shared/types/messaging.ts
|
||||
|
||||
export enum MessageType {
|
||||
// 采集相关
|
||||
CRAWL_PRODUCT = 'CRAWL_PRODUCT',
|
||||
CRAWL_ORDER = 'CRAWL_ORDER',
|
||||
CRAWL_COMPLETE = 'CRAWL_COMPLETE',
|
||||
|
||||
// 自动化相关
|
||||
AUTO_LISTING = 'AUTO_LISTING',
|
||||
AUTO_ORDER = 'AUTO_ORDER',
|
||||
AUTO_AD = 'AUTO_AD',
|
||||
|
||||
// 同步相关
|
||||
SYNC_DATA = 'SYNC_DATA',
|
||||
SYNC_STATUS = 'SYNC_STATUS',
|
||||
|
||||
// 认证相关
|
||||
AUTH_LOGIN = 'AUTH_LOGIN',
|
||||
AUTH_LOGOUT = 'AUTH_LOGOUT',
|
||||
AUTH_REFRESH = 'AUTH_REFRESH',
|
||||
|
||||
// 设置相关
|
||||
GET_SETTINGS = 'GET_SETTINGS',
|
||||
SET_SETTINGS = 'SET_SETTINGS',
|
||||
|
||||
// 任务相关
|
||||
TASK_STATUS = 'TASK_STATUS',
|
||||
TASK_RESULT = 'TASK_RESULT',
|
||||
}
|
||||
|
||||
export interface MessagePayload {
|
||||
type: MessageType;
|
||||
data?: any;
|
||||
error?: string;
|
||||
traceId?: string;
|
||||
}
|
||||
```
|
||||
|
||||
#### 通信流程
|
||||
|
||||
```
|
||||
Content Script ──► Background Service Worker ──► Backend API
|
||||
│ │
|
||||
│◄─────────────────────┘
|
||||
│
|
||||
Popup/Options ◄─── Chrome Storage
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 核心功能模块
|
||||
|
||||
### 4.1 数据采集模块 (Crawler)
|
||||
|
||||
**功能定位**
|
||||
- 从各电商平台采集商品、订单数据
|
||||
- **仅处理无API平台**：TikTok Shop, Temu, 部分1688页面等
|
||||
- **有API平台由后端处理**：Amazon MWS, eBay API, Shopee Open API
|
||||
- 支持沙箱模式（测试环境）
|
||||
|
||||
**采集策略矩阵**
|
||||
|
||||
| 平台 | 类型 | 采集方式 | 登录要求 | 反爬策略 |
|
||||
|------|------|----------|----------|----------|
|
||||
| Amazon | 有API | ❌ 后端处理 | OAuth | API限流 |
|
||||
| eBay | 有API | ❌ 后端处理 | OAuth | API限流 |
|
||||
| Shopee | 有API | ❌ 后端处理 | OAuth | API限流 |
|
||||
| TikTok Shop | 无API | ✅ 插件采集 | 需登录 | 指纹隔离+代理IP |
|
||||
| Temu | 无API | ✅ 插件采集 | 需登录 | 指纹隔离+代理IP |
|
||||
| 1688(部分) | 无API | ✅ 插件采集 | 可选 | 频率控制 |
|
||||
|
||||
**⚠️ 重要约束**
|
||||
- **后端严禁直接爬取**（IP封禁风险、法律合规问题）
|
||||
- 所有网页级采集必须通过插件在用户浏览器端执行
|
||||
- 插件必须实现店铺隔离（一店一IP一指纹）
|
||||
|
||||
**采集器实现**
|
||||
|
||||
```typescript
|
||||
// src/content/crawlers/amazonCrawler.ts
|
||||
|
||||
export class AmazonCrawler {
|
||||
private selectors = {
|
||||
title: '#productTitle',
|
||||
price: '.a-price-whole, .a-offscreen',
|
||||
images: '#landingImage, .a-dynamic-image',
|
||||
description: '#feature-bullets, #productDescription',
|
||||
reviews: '#acrCustomerReviewText',
|
||||
};
|
||||
|
||||
async crawlProduct(): Promise<ProductData> {
|
||||
const title = this.extractText(this.selectors.title);
|
||||
const price = this.extractPrice(this.selectors.price);
|
||||
const images = this.extractImages(this.selectors.images);
|
||||
const description = this.extractText(this.selectors.description);
|
||||
|
||||
return {
|
||||
platform: 'AMAZON',
|
||||
title,
|
||||
price,
|
||||
images,
|
||||
description,
|
||||
url: window.location.href,
|
||||
crawledAt: new Date().toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
private extractText(selector: string): string {
|
||||
const element = document.querySelector(selector);
|
||||
return element?.textContent?.trim() || '';
|
||||
}
|
||||
|
||||
private extractPrice(selector: string): number {
|
||||
const element = document.querySelector(selector);
|
||||
const text = element?.textContent?.replace(/[^\d.]/g, '') || '0';
|
||||
return parseFloat(text);
|
||||
}
|
||||
|
||||
private extractImages(selector: string): string[] {
|
||||
const elements = document.querySelectorAll(selector);
|
||||
return Array.from(elements)
|
||||
.map(el => el.getAttribute('src') || el.getAttribute('data-src'))
|
||||
.filter(Boolean) as string[];
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**采集流程**
|
||||
|
||||
```
|
||||
1. 用户点击采集按钮 / 定时任务触发
|
||||
2. Content Script 注入采集器
|
||||
3. 采集器解析 DOM 提取数据
|
||||
4. 数据发送至 Background
|
||||
5. Background 发送至后端 API
|
||||
6. 返回采集结果
|
||||
```
|
||||
|
||||
### 4.2 自动化操作模块 (Automation)
|
||||
|
||||
**功能定位**
|
||||
- 自动刊登商品
|
||||
- 自动处理订单
|
||||
- 自动投放广告
|
||||
|
||||
**自动化实现**
|
||||
|
||||
```typescript
|
||||
// src/content/automation/listingAutomation.ts
|
||||
|
||||
export class ListingAutomation {
|
||||
async autoListing(productData: ProductData, platform: string): Promise<boolean> {
|
||||
switch (platform) {
|
||||
case 'AMAZON':
|
||||
return this.listOnAmazon(productData);
|
||||
case 'EBAY':
|
||||
return this.listOnEbay(productData);
|
||||
case 'SHOPIFY':
|
||||
return this.listOnShopify(productData);
|
||||
default:
|
||||
throw new Error(`Unsupported platform: ${platform}`);
|
||||
}
|
||||
}
|
||||
|
||||
private async listOnAmazon(product: ProductData): Promise<boolean> {
|
||||
// 1. 导航到刊登页面
|
||||
await this.navigateTo('/inventory/add');
|
||||
|
||||
// 2. 填写商品信息
|
||||
await this.fillInput('#title', product.title);
|
||||
await this.fillInput('#price', product.price.toString());
|
||||
await this.fillTextarea('#description', product.description);
|
||||
|
||||
// 3. 上传图片
|
||||
for (const imageUrl of product.images) {
|
||||
await this.uploadImage(imageUrl);
|
||||
}
|
||||
|
||||
// 4. 提交刊登
|
||||
await this.click('#submit-button');
|
||||
|
||||
// 5. 等待结果
|
||||
return this.waitForSuccess();
|
||||
}
|
||||
|
||||
private async navigateTo(path: string): Promise<void> {
|
||||
window.location.href = `https://sellercentral.amazon.com${path}`;
|
||||
await this.waitForElement('#title', 10000);
|
||||
}
|
||||
|
||||
private async fillInput(selector: string, value: string): Promise<void> {
|
||||
const input = document.querySelector(selector) as HTMLInputElement;
|
||||
if (input) {
|
||||
input.value = value;
|
||||
input.dispatchEvent(new Event('input', { bubbles: true }));
|
||||
}
|
||||
}
|
||||
|
||||
private async waitForElement(selector: string, timeout: number): Promise<void> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const startTime = Date.now();
|
||||
const check = () => {
|
||||
if (document.querySelector(selector)) {
|
||||
resolve();
|
||||
} else if (Date.now() - startTime > timeout) {
|
||||
reject(new Error('Timeout waiting for element'));
|
||||
} else {
|
||||
setTimeout(check, 100);
|
||||
}
|
||||
};
|
||||
check();
|
||||
});
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 4.3 数据同步模块 (Sync)
|
||||
|
||||
**功能定位**
|
||||
- 定时同步订单数据
|
||||
- 同步库存状态
|
||||
- 同步广告数据
|
||||
|
||||
**同步配置**
|
||||
|
||||
```typescript
|
||||
// src/background/services/syncService.ts
|
||||
|
||||
export class SyncService {
|
||||
private syncIntervals: Record<string, number> = {
|
||||
orders: 5 * 60 * 1000, // 5分钟
|
||||
inventory: 10 * 60 * 1000, // 10分钟
|
||||
ads: 30 * 60 * 1000, // 30分钟
|
||||
};
|
||||
|
||||
async startSync(): Promise<void> {
|
||||
// 创建定时任务
|
||||
chrome.alarms.create('syncOrders', { periodInMinutes: 5 });
|
||||
chrome.alarms.create('syncInventory', { periodInMinutes: 10 });
|
||||
chrome.alarms.create('syncAds', { periodInMinutes: 30 });
|
||||
}
|
||||
|
||||
async handleAlarm(alarmName: string): Promise<void> {
|
||||
switch (alarmName) {
|
||||
case 'syncOrders':
|
||||
await this.syncOrders();
|
||||
break;
|
||||
case 'syncInventory':
|
||||
await this.syncInventory();
|
||||
break;
|
||||
case 'syncAds':
|
||||
await this.syncAds();
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
private async syncOrders(): Promise<void> {
|
||||
const platforms = await this.getEnabledPlatforms();
|
||||
for (const platform of platforms) {
|
||||
try {
|
||||
const orders = await this.crawlOrders(platform);
|
||||
await this.sendToBackend('/api/v1/orders/sync', orders);
|
||||
} catch (error) {
|
||||
console.error(`Failed to sync orders for ${platform}:`, error);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 安全与隐私
|
||||
|
||||
### 5.1 数据存储
|
||||
|
||||
```typescript
|
||||
// src/shared/utils/storage.ts
|
||||
|
||||
export class SecureStorage {
|
||||
// 存储敏感数据（加密）
|
||||
static async setSecure(key: string, value: string): Promise<void> {
|
||||
const encrypted = await this.encrypt(value);
|
||||
await chrome.storage.local.set({ [key]: encrypted });
|
||||
}
|
||||
|
||||
// 读取敏感数据（解密）
|
||||
static async getSecure(key: string): Promise<string | null> {
|
||||
const result = await chrome.storage.local.get(key);
|
||||
if (result[key]) {
|
||||
return this.decrypt(result[key]);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// 存储普通数据
|
||||
static async set(key: string, value: any): Promise<void> {
|
||||
await chrome.storage.local.set({ [key]: value });
|
||||
}
|
||||
|
||||
// 读取普通数据
|
||||
static async get(key: string): Promise<any> {
|
||||
const result = await chrome.storage.local.get(key);
|
||||
return result[key];
|
||||
}
|
||||
|
||||
private static async encrypt(text: string): Promise<string> {
|
||||
// 使用 Chrome 的加密 API
|
||||
// 实际实现需要更复杂的加密逻辑
|
||||
return btoa(text);
|
||||
}
|
||||
|
||||
private static async decrypt(encrypted: string): Promise<string> {
|
||||
return atob(encrypted);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5.2 权限控制
|
||||
|
||||
```json
|
||||
// manifest.json
|
||||
{
|
||||
"manifest_version": 3,
|
||||
"name": "Crawlful Hub",
|
||||
"version": "1.0.0",
|
||||
"permissions": [
|
||||
"storage",
|
||||
"alarms",
|
||||
"activeTab",
|
||||
"scripting"
|
||||
],
|
||||
"host_permissions": [
|
||||
"https://sellercentral.amazon.com/*",
|
||||
"https://www.ebay.com/*",
|
||||
"https://*.myshopify.com/*",
|
||||
"https://*.tiktok.com/*"
|
||||
],
|
||||
"background": {
|
||||
"service_worker": "src/background/index.ts"
|
||||
},
|
||||
"content_scripts": [
|
||||
{
|
||||
"matches": [
|
||||
"https://sellercentral.amazon.com/*",
|
||||
"https://www.ebay.com/*"
|
||||
],
|
||||
"js": ["src/content/index.ts"]
|
||||
}
|
||||
],
|
||||
"action": {
|
||||
"default_popup": "src/popup/index.html"
|
||||
},
|
||||
"options_page": "src/options/index.html"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. 开发规范
|
||||
|
||||
### 6.1 代码规范
|
||||
|
||||
- 使用 TypeScript 严格模式
|
||||
- 使用函数式组件 + Hooks
|
||||
- 避免使用 `any` 类型
|
||||
- 所有消息类型必须在 `messaging.ts` 中定义
|
||||
|
||||
### 6.2 测试规范
|
||||
|
||||
```typescript
|
||||
// __tests__/crawler.test.ts
|
||||
import { AmazonCrawler } from '../src/content/crawlers/amazonCrawler';
|
||||
|
||||
describe('AmazonCrawler', () => {
|
||||
let crawler: AmazonCrawler;
|
||||
|
||||
beforeEach(() => {
|
||||
crawler = new AmazonCrawler();
|
||||
});
|
||||
|
||||
test('should extract product title', async () => {
|
||||
// Mock DOM
|
||||
document.body.innerHTML = `
|
||||
<span id="productTitle">Test Product</span>
|
||||
`;
|
||||
|
||||
const product = await crawler.crawlProduct();
|
||||
expect(product.title).toBe('Test Product');
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### 6.3 构建与发布
|
||||
|
||||
```bash
|
||||
# 开发模式
|
||||
npm run dev
|
||||
|
||||
# 生产构建
|
||||
npm run build
|
||||
|
||||
# 打包扩展
|
||||
npm run package
|
||||
|
||||
# 运行测试
|
||||
npm run test
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. 相关文档
|
||||
|
||||
- [DOM Interaction](./DOM_Interaction.md)
|
||||
- [Automation Scripts](./Automation_Scripts.md)
|
||||
- [Backend Design](../02_Backend/Backend_Design.md)
|
||||
- [Business ClosedLoops](../00_Business/Business_ClosedLoops.md)
|
||||
|
||||
---
|
||||
|
||||
*本文档基于业务闭环设计，最后更新: 2026-03-18*
|
||||
Reference in New Issue
Block a user