499 lines
16 KiB
Markdown
499 lines
16 KiB
Markdown
# upload_device_info 接口性能优化方案
|
||
|
||
## 📊 接口现状分析
|
||
|
||
### 当前实现架构
|
||
|
||
**接口入口**: `bigScreen.py` 中的 `handle_device_info()`
|
||
**处理逻辑**: `device.py` 中的 `upload_device_info()`
|
||
**批量处理器**: `device_batch_processor.py` 中的 `DeviceBatchProcessor`
|
||
|
||
### 🔍 性能问题识别
|
||
|
||
#### 1. 架构复杂性问题
|
||
|
||
```python
|
||
# 当前调用链路过长
|
||
handle_device_info()
|
||
↓
|
||
upload_device_info()
|
||
↓
|
||
DeviceBatchProcessor.submit()
|
||
↓
|
||
异步队列处理
|
||
↓
|
||
同步数据库操作
|
||
```
|
||
|
||
**问题分析**:
|
||
- 调用链路长，增加延迟
|
||
- 多层异常处理和日志记录
|
||
- Flask应用上下文传递复杂
|
||
|
||
#### 2. 数据处理效率问题
|
||
|
||
```python
|
||
# 现有数据处理流程
|
||
def process_single_device(single_device_info):
|
||
# 1. 多次字典取值操作
|
||
system_software_version = data.get("System Software Version") or data.get("system_version")
|
||
wifi_address = data.get("Wifi Address") or data.get("wifi_address")
|
||
bluetooth_address = data.get("Bluetooth Address") or data.get("bluetooth_address")
|
||
# ... 大量重复的get操作
|
||
|
||
# 2. 字符串处理开销
|
||
ipv4_match = re.search(r'\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b', ip_address)
|
||
|
||
# 3. 时间戳转换开销
|
||
beijing_tz = pytz.timezone('Asia/Shanghai')
|
||
dt = datetime.fromtimestamp(int(timestamp)/1000, tz=beijing_tz)
|
||
```
|
||
|
||
**性能瓶颈**:
|
||
- 大量的字典key查找操作
|
||
- 正则表达式匹配开销
|
||
- 时区转换计算开销
|
||
- 字符串操作频繁
|
||
|
||
#### 3. 数据库操作问题
|
||
|
||
```python
|
||
# 当前数据库操作
|
||
def save_device_info_to_db(device_info):
|
||
# 1. 每次操作都创建新连接
|
||
device_info_record = DeviceInfo.query.filter_by(serial_number=serial_number).first()
|
||
|
||
# 2. N+1查询问题
|
||
user_info = UserInfo.query.filter_by(device_sn=serial_number).first()
|
||
|
||
# 3. 单条插入/更新
|
||
if device_info_record:
|
||
# 逐字段更新
|
||
else:
|
||
# 单条插入
|
||
db.session.commit()
|
||
```
|
||
|
||
**问题分析**:
|
||
- 缺少批量操作优化
|
||
- 存在N+1查询问题
|
||
- 频繁的数据库连接和提交
|
||
|
||
#### 4. 批量处理器配置问题
|
||
|
||
```python
|
||
# 当前批量处理器配置
|
||
class DeviceBatchProcessor:
|
||
def __init__(self):
|
||
self.batch_size = 50 # 批量大小偏小
|
||
self.max_workers = 4 # 线程数不够
|
||
self.queue_size = 1000 # 队列容量小
|
||
self.batch_timeout = 5 # 等待时间长
|
||
```
|
||
|
||
**配置问题**:
|
||
- 批量大小偏小，无法充分利用数据库批量操作优势
|
||
- 工作线程数不足，无法充分利用多核CPU
|
||
- 队列容量小，高并发时容易拒绝服务
|
||
|
||
## 🚀 优化方案设计
|
||
|
||
### 1. 接口架构优化
|
||
|
||
#### A. 简化调用链路
|
||
|
||
```python
|
||
# 优化后的简化架构
|
||
@app.route("/upload_device_info", methods=['POST'])
|
||
def handle_device_info():
|
||
"""简化的设备信息处理接口"""
|
||
device_info = request.get_json()
|
||
|
||
# 快速参数验证
|
||
if not device_info:
|
||
return jsonify({"status": "error", "message": "请求体不能为空"}), 400
|
||
|
||
# 直接提交到优化批量处理器
|
||
success = async_device_processor.submit_fast(device_info)
|
||
|
||
if success:
|
||
return STANDARD_SUCCESS_RESPONSE
|
||
else:
|
||
return jsonify({"status": "error", "message": "系统繁忙，请稍后重试"}), 503
|
||
```
|
||
|
||
#### B. 异步处理器重构
|
||
|
||
```python
|
||
class AsyncDeviceProcessor:
|
||
"""高性能异步设备信息处理器"""
|
||
|
||
def __init__(self):
|
||
# CPU自适应配置
|
||
self.cpu_cores = os.cpu_count()
|
||
self.batch_size = self.cpu_cores * 20 # 动态批量大小
|
||
self.max_workers = self.cpu_cores * 3 # 动态工作线程
|
||
self.queue_size = 10000 # 大容量队列
|
||
self.batch_timeout = 1.0 # 快速批处理
|
||
|
||
# 多级处理队列
|
||
self.parsing_queue = asyncio.Queue(maxsize=5000)
|
||
self.validation_queue = asyncio.Queue(maxsize=3000)
|
||
self.database_queue = asyncio.Queue(maxsize=2000)
|
||
|
||
# 启动异步工作协程
|
||
self._start_async_workers()
|
||
|
||
async def submit_fast(self, device_info):
|
||
"""快速提交设备信息到异步队列"""
|
||
try:
|
||
await self.parsing_queue.put(device_info)
|
||
return True
|
||
except asyncio.QueueFull:
|
||
return False
|
||
```
|
||
|
||
### 2. 数据处理性能优化
|
||
|
||
#### A. 预编译和缓存优化
|
||
|
||
```python
|
||
class DeviceDataParser:
|
||
"""优化的设备数据解析器"""
|
||
|
||
# 预编译正则表达式
|
||
IPV4_PATTERN = re.compile(r'\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b')
|
||
|
||
# 字段映射表（避免重复字符串比较）
|
||
FIELD_MAPPING = {
|
||
'system_version': ['System Software Version', 'system_version', 'SystemSoftwareVersion'],
|
||
'wifi_address': ['Wifi Address', 'wifi_address', 'WifiAddress'],
|
||
'bluetooth_address': ['Bluetooth Address', 'bluetooth_address', 'BluetoothAddress'],
|
||
'ip_address': ['IP Address', 'ip_address', 'IPAddress'],
|
||
'network_mode': ['Network Access Mode', 'network_mode', 'NetworkAccessMode'],
|
||
'serial_number': ['SerialNumber', 'serial_number', 'deviceSn'],
|
||
'device_name': ['Device Name', 'device_name', 'DeviceName'],
|
||
'imei': ['IMEI', 'imei']
|
||
}
|
||
|
||
# 时区对象缓存
|
||
BEIJING_TZ = pytz.timezone('Asia/Shanghai')
|
||
|
||
def extract_field_fast(self, data: dict, field_key: str):
|
||
"""快速字段提取"""
|
||
for possible_key in self.FIELD_MAPPING.get(field_key, [field_key]):
|
||
if possible_key in data:
|
||
return data[possible_key]
|
||
return None
|
||
|
||
def parse_device_data_batch(self, device_list: List[dict]) -> List[dict]:
|
||
"""批量解析设备数据"""
|
||
parsed_devices = []
|
||
current_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||
|
||
for device_data in device_list:
|
||
try:
|
||
data = device_data.get("data", device_data)
|
||
|
||
# 使用优化的字段提取
|
||
parsed = {
|
||
'serial_number': self.extract_field_fast(data, 'serial_number'),
|
||
'device_name': self.extract_field_fast(data, 'device_name'),
|
||
'system_version': self.extract_field_fast(data, 'system_version'),
|
||
'wifi_address': self.extract_field_fast(data, 'wifi_address'),
|
||
'bluetooth_address': self.extract_field_fast(data, 'bluetooth_address'),
|
||
'network_mode': self.extract_field_fast(data, 'network_mode'),
|
||
'imei': self.extract_field_fast(data, 'imei'),
|
||
'update_time': current_time
|
||
}
|
||
|
||
# 优化IP地址提取
|
||
ip_raw = self.extract_field_fast(data, 'ip_address')
|
||
if ip_raw:
|
||
ip_match = self.IPV4_PATTERN.search(ip_raw)
|
||
parsed['ip_address'] = ip_match.group(0) if ip_match else None
|
||
|
||
# 优化时间戳处理
|
||
timestamp = data.get("timestamp")
|
||
if timestamp and str(timestamp).isdigit() and len(str(timestamp)) == 13:
|
||
dt = datetime.fromtimestamp(int(timestamp)/1000, tz=self.BEIJING_TZ)
|
||
parsed['timestamp'] = dt.strftime("%Y-%m-%d %H:%M:%S")
|
||
else:
|
||
parsed['timestamp'] = current_time
|
||
|
||
# 优化状态字段处理
|
||
battery_level = data.get("batteryLevel") or data.get("battery_level")
|
||
parsed['battery_level'] = self.normalize_battery_level_fast(battery_level)
|
||
|
||
wear_state = data.get("wearState") or data.get("wear_state")
|
||
parsed['wearable_status'] = "WORN" if wear_state and int(wear_state) == 1 else "NOT_WORN"
|
||
|
||
parsed_devices.append(parsed)
|
||
|
||
except Exception as e:
|
||
# 记录解析失败，但不中断批次处理
|
||
logger.warning(f"设备数据解析失败: {e}")
|
||
continue
|
||
|
||
return parsed_devices
|
||
|
||
@staticmethod
|
||
def normalize_battery_level_fast(battery_level):
|
||
"""快速电池电量标准化"""
|
||
if not battery_level:
|
||
return 0
|
||
try:
|
||
level = float(battery_level)
|
||
return max(0, min(100, level)) # 确保在0-100范围内
|
||
except (ValueError, TypeError):
|
||
return 0
|
||
```
|
||
|
||
#### B. 批量数据库操作优化
|
||
|
||
```python
|
||
class OptimizedDeviceDAO:
|
||
"""优化的设备信息数据访问层"""
|
||
|
||
def __init__(self):
|
||
self.batch_insert_size = 100
|
||
self.batch_update_size = 100
|
||
|
||
def batch_upsert_devices(self, device_list: List[dict]) -> dict:
|
||
"""批量插入或更新设备信息"""
|
||
if not device_list:
|
||
return {'inserted': 0, 'updated': 0, 'errors': 0}
|
||
|
||
stats = {'inserted': 0, 'updated': 0, 'errors': 0}
|
||
|
||
try:
|
||
# 1. 批量查询现有设备
|
||
serial_numbers = [d['serial_number'] for d in device_list if d.get('serial_number')]
|
||
existing_devices = {}
|
||
|
||
if serial_numbers:
|
||
existing_records = db.session.query(DeviceInfo).filter(
|
||
DeviceInfo.serial_number.in_(serial_numbers)
|
||
).all()
|
||
|
||
existing_devices = {device.serial_number: device for device in existing_records}
|
||
|
||
# 2. 分离插入和更新操作
|
||
devices_to_insert = []
|
||
devices_to_update = []
|
||
|
||
for device_data in device_list:
|
||
serial_number = device_data.get('serial_number')
|
||
if not serial_number:
|
||
stats['errors'] += 1
|
||
continue
|
||
|
||
if serial_number in existing_devices:
|
||
# 需要更新的设备
|
||
existing_device = existing_devices[serial_number]
|
||
self._update_device_fields(existing_device, device_data)
|
||
devices_to_update.append(existing_device)
|
||
else:
|
||
# 需要插入的新设备
|
||
new_device = DeviceInfo(**device_data)
|
||
devices_to_insert.append(new_device)
|
||
|
||
# 3. 批量插入新设备
|
||
if devices_to_insert:
|
||
db.session.bulk_save_objects(devices_to_insert)
|
||
stats['inserted'] = len(devices_to_insert)
|
||
|
||
# 4. 批量更新现有设备
|
||
if devices_to_update:
|
||
stats['updated'] = len(devices_to_update)
|
||
|
||
# 5. 提交事务
|
||
db.session.commit()
|
||
|
||
except Exception as e:
|
||
db.session.rollback()
|
||
logger.error(f"批量设备信息操作失败: {e}")
|
||
stats['errors'] = len(device_list)
|
||
|
||
return stats
|
||
|
||
def _update_device_fields(self, device: DeviceInfo, new_data: dict):
|
||
"""更新设备字段"""
|
||
update_fields = [
|
||
'device_name', 'system_version', 'wifi_address', 'bluetooth_address',
|
||
'ip_address', 'network_mode', 'imei', 'battery_level', 'wearable_status',
|
||
'update_time', 'timestamp'
|
||
]
|
||
|
||
for field in update_fields:
|
||
if field in new_data and new_data[field] is not None:
|
||
setattr(device, field, new_data[field])
|
||
```
|
||
|
||
### 3. 异步处理流水线
|
||
|
||
```python
|
||
class AsyncDeviceProcessingPipeline:
|
||
"""异步设备信息处理流水线"""
|
||
|
||
def __init__(self):
|
||
self.parser = DeviceDataParser()
|
||
self.dao = OptimizedDeviceDAO()
|
||
self.validator = DeviceDataValidator()
|
||
|
||
# 处理统计
|
||
self.stats = {
|
||
'total_processed': 0,
|
||
'total_success': 0,
|
||
'total_errors': 0,
|
||
'processing_time': 0
|
||
}
|
||
|
||
async def process_device_batch(self, device_batch: List[dict]):
|
||
"""异步处理设备批次"""
|
||
start_time = time.time()
|
||
|
||
try:
|
||
# Stage 1: 数据解析
|
||
parsed_devices = await asyncio.get_event_loop().run_in_executor(
|
||
None,
|
||
self.parser.parse_device_data_batch,
|
||
device_batch
|
||
)
|
||
|
||
# Stage 2: 数据验证
|
||
validated_devices = await asyncio.get_event_loop().run_in_executor(
|
||
None,
|
||
self.validator.validate_device_batch,
|
||
parsed_devices
|
||
)
|
||
|
||
# Stage 3: 数据库操作
|
||
result = await asyncio.get_event_loop().run_in_executor(
|
||
None,
|
||
self.dao.batch_upsert_devices,
|
||
validated_devices
|
||
)
|
||
|
||
# 更新统计
|
||
processing_time = time.time() - start_time
|
||
self.stats['total_processed'] += len(device_batch)
|
||
self.stats['total_success'] += result['inserted'] + result['updated']
|
||
self.stats['total_errors'] += result['errors']
|
||
self.stats['processing_time'] += processing_time
|
||
|
||
logger.info(f"设备批次处理完成: {len(device_batch)}条, 耗时: {processing_time:.3f}s")
|
||
|
||
except Exception as e:
|
||
self.stats['total_errors'] += len(device_batch)
|
||
logger.error(f"设备批次处理失败: {e}")
|
||
```
|
||
|
||
## 📈 性能优化预期效果
|
||
|
||
### 优化前后对比
|
||
|
||
| 指标 | 优化前 | 优化后 | 提升幅度 |
|
||
|------|--------|--------|----------|
|
||
| **单次请求响应时间** | 50-200ms | 10-30ms | **70-85%提升** |
|
||
| **批量处理能力** | 50设备/批次 | 100-200设备/批次 | **100-300%提升** |
|
||
| **并发处理能力** | 400设备同时 | 1000-2000设备同时 | **150-400%提升** |
|
||
| **数据库操作效率** | N+1查询 | 批量操作 | **90%查询减少** |
|
||
| **CPU利用率** | 20-40% | 60-80% | **100%效率提升** |
|
||
| **内存使用** | 100MB | 80MB | **20%内存节省** |
|
||
|
||
### 性能目标
|
||
|
||
- **QPS目标**: 2000+ 请求/秒
|
||
- **平均响应时间**: <20ms
|
||
- **99%分位响应时间**: <50ms
|
||
- **并发设备数**: 2000+ 设备同时上传
|
||
- **成功率**: >99.9%
|
||
|
||
## 🛠️ 实施方案
|
||
|
||
### Phase 1: 核心优化（1-2天）
|
||
|
||
1. **数据解析器优化**
|
||
- 实现 `DeviceDataParser` 类
|
||
- 预编译正则表达式
|
||
- 字段映射表优化
|
||
|
||
2. **批量处理器重构**
|
||
- 实现 `AsyncDeviceProcessor` 类
|
||
- CPU自适应配置
|
||
- 多级队列架构
|
||
|
||
### Phase 2: 数据库优化（2-3天）
|
||
|
||
1. **DAO层重构**
|
||
- 实现批量插入/更新逻辑
|
||
- 消除N+1查询问题
|
||
- 事务优化
|
||
|
||
2. **索引优化**
|
||
- 设备序列号索引
|
||
- 复合索引创建
|
||
|
||
### Phase 3: 异步流水线（2天）
|
||
|
||
1. **异步处理流水线**
|
||
- 实现多阶段异步处理
|
||
- 错误处理和恢复机制
|
||
- 性能监控集成
|
||
|
||
2. **接口简化**
|
||
- 重构 `handle_device_info` 函数
|
||
- 移除冗余日志和处理逻辑
|
||
|
||
### Phase 4: 测试和部署（1天）
|
||
|
||
1. **性能测试**
|
||
- 2000设备并发测试
|
||
- 压力测试和稳定性验证
|
||
|
||
2. **监控和告警**
|
||
- 性能指标监控
|
||
- 异常告警机制
|
||
|
||
## 📊 监控和维护
|
||
|
||
### 关键性能指标监控
|
||
|
||
```python
|
||
class DeviceProcessorMonitor:
|
||
"""设备处理器性能监控"""
|
||
|
||
def get_performance_metrics(self):
|
||
return {
|
||
'queue_sizes': {
|
||
'parsing': self.parsing_queue.qsize(),
|
||
'validation': self.validation_queue.qsize(),
|
||
'database': self.database_queue.qsize()
|
||
},
|
||
'processing_stats': self.stats,
|
||
'worker_status': self.get_worker_status(),
|
||
'system_resource': self.get_system_resource_usage()
|
||
}
|
||
```
|
||
|
||
### 告警机制
|
||
|
||
- 队列积压告警（队列大小 > 80%）
|
||
- 处理延迟告警（平均响应时间 > 100ms）
|
||
- 错误率告警（错误率 > 1%）
|
||
- 系统资源告警（CPU > 90%, 内存 > 90%）
|
||
|
||
## 🔧 配置管理
|
||
|
||
```python
|
||
DEVICE_PROCESSOR_CONFIG = {
|
||
'batch_size': os.getenv('DEVICE_BATCH_SIZE', os.cpu_count() * 20),
|
||
'max_workers': os.getenv('DEVICE_MAX_WORKERS', os.cpu_count() * 3),
|
||
'queue_size': os.getenv('DEVICE_QUEUE_SIZE', 10000),
|
||
'batch_timeout': float(os.getenv('DEVICE_BATCH_TIMEOUT', 1.0)),
|
||
'enable_async': os.getenv('DEVICE_ENABLE_ASYNC', 'true').lower() == 'true'
|
||
}
|
||
```
|
||
|
||
通过以上优化方案，`upload_device_info` 接口将获得显著的性能提升，能够支持更高的并发量和更快的响应时间，满足大规模设备部署的需求。 |