业务日志入库实战指南

业务日志入库实战指南一、概述在微服务架构中将关键业务操作的日志记录到数据库表中是常见需求。与文件日志不同数据库日志支持结构化查询、状态追踪和重试机制。但日志入库如果设计不当可能影响主流程性能甚至导致业务失败。本文围绕如何安全、高效地将业务日志写入数据库展开涵盖同步/异步方案、异常隔离、事务边界等核心问题。注博客https://blog.csdn.net/badao_liumang_qizhi二、示例场景假设有一个支付回调处理服务需要记录每次回调的处理日志ServicepublicclassPaymentCallbackService{ResourceprivatePaymentLogRepositorypaymentLogRepository;ResourceprivateOrderRepositoryorderRepository;publicBooleanhandleCallback(PaymentCallbackDtocallbackDto){// 1. 校验签名// 2. 更新订单状态// 3. 记录处理日志returntrue;}}日志表结构CREATETABLEpayment_callback_log(idBIGINTUNSIGNEDAUTO_INCREMENTPRIMARYKEY,order_noVARCHAR(50)NOTNULLCOMMENT订单号,callback_dataTEXTNOTNULLCOMMENT回调原始数据,process_flagVARCHAR(1)NOTNULLCOMMENT处理结果Y-成功P-失败,error_msgVARCHAR(500)NULLCOMMENT错误信息,create_timeDATETIMENOTNULLCOMMENT创建时间);三、方案一同步写入 try-catch 隔离3.1 实现方式ServicepublicclassPaymentCallbackService{ResourceprivatePaymentLogRepositorypaymentLogRepository;ResourceprivateOrderRepositoryorderRepository;Transactional(rollbackForException.class)publicBooleanhandleCallback(PaymentCallbackDtocallbackDto){StringrequestDataJSON.toJSONString(callbackDto);// 1. 业务处理OrderorderorderRepository.findByOrderNo(callbackDto.getOrderNo());if(ordernull){saveLog(callbackDto.getOrderNo(),requestData,P,订单不存在);returnfalse;}order.setStatus(OrderStatus.PAID);orderRepository.save(order);// 2. 记录成功日志saveLog(callbackDto.getOrderNo(),requestData,Y,null);returntrue;}privatevoidsaveLog(StringorderNo,Stringdata,Stringflag,StringerrorMsg){try{PaymentCallbackLoglognewPaymentCallbackLog();log.setOrderNo(orderNo);log.setCallbackData(data);log.setProcessFlag(flag);log.setErrorMsg(errorMsg);log.setCreateTime(newDate());paymentLogRepository.save(log);}catch(Exceptione){log.warn(日志记录失败, orderNo{}, error{},orderNo,e.getMessage());}}}3.2 优缺点优点缺点实现简单代码直观日志写入耗时会增加接口响应时间try-catch 确保日志失败不影响主流程日志与业务在同一事务中日志表锁可能影响业务数据一致性好同事务提交高并发下日志写入可能成为瓶颈3.3 适用场景日志写入频率不高非高并发接口对日志完整性要求高必须与业务同时成功或失败项目初期快速实现四、方案二事务外异步写入Async4.1 实现方式ServicepublicclassPaymentCallbackService{ResourceprivateAsyncLogServiceasyncLogService;ResourceprivateOrderRepositoryorderRepository;Transactional(rollbackForException.class)publicBooleanhandleCallback(PaymentCallbackDtocallbackDto){StringrequestDataJSON.toJSONString(callbackDto);OrderorderorderRepository.findByOrderNo(callbackDto.getOrderNo());if(ordernull){asyncLogService.saveLog(callbackDto.getOrderNo(),requestData,P,订单不存在);returnfalse;}order.setStatus(OrderStatus.PAID);orderRepository.save(order);asyncLogService.saveLog(callbackDto.getOrderNo(),requestData,Y,null);returntrue;}}ServicepublicclassAsyncLogService{ResourceprivatePaymentLogRepositorypaymentLogRepository;Async(logExecutor)publicvoidsaveLog(StringorderNo,Stringdata,Stringflag,StringerrorMsg){try{PaymentCallbackLoglognewPaymentCallbackLog();log.setOrderNo(orderNo);log.setCallbackData(data);log.setProcessFlag(flag);log.setErrorMsg(errorMsg);log.setCreateTime(newDate());paymentLogRepository.save(log);}catch(Exceptione){log.warn(异步日志记录失败, orderNo{}, error{},orderNo,e.getMessage());}}}4.2 线程池配置ConfigurationEnableAsyncpublicclassAsyncConfig{Bean(logExecutor)publicExecutorlogExecutor(){ThreadPoolTaskExecutorexecutornewThreadPoolTaskExecutor();executor.setCorePoolSize(2);executor.setMaxPoolSize(5);executor.setQueueCapacity(1000);executor.setThreadNamePrefix(log-async-);executor.setRejectedExecutionHandler(newThreadPoolExecutor.CallerRunsPolicy());executor.initialize();returnexecutor;}}4.3 优缺点优点缺点不阻塞主流程接口响应更快日志可能丢失应用崩溃时队列中的日志丢失日志写入与业务事务解耦日志记录时间可能晚于业务完成时间线程池可控制并发度需要额外配置线程池日志表锁不影响业务事务事务回滚时日志仍会写入不一致4.4 适用场景高并发接口日志写入不能拖慢响应日志丢失可接受非关键审计日志日志与业务不需要强一致五、方案三事务提交后写入TransactionSynchronization5.1 实现方式确保日志只在业务事务成功提交后才写入避免事务回滚但日志已写入的不一致问题。ServicepublicclassPaymentCallbackService{ResourceprivatePaymentLogRepositorypaymentLogRepository;ResourceprivateOrderRepositoryorderRepository;Transactional(rollbackForException.class)publicBooleanhandleCallback(PaymentCallbackDtocallbackDto){StringrequestDataJSON.toJSONString(callbackDto);OrderorderorderRepository.findByOrderNo(callbackDto.getOrderNo());if(ordernull){saveLogAfterCommit(callbackDto.getOrderNo(),requestData,P,订单不存在);returnfalse;}order.setStatus(OrderStatus.PAID);orderRepository.save(order);saveLogAfterCommit(callbackDto.getOrderNo(),requestData,Y,null);returntrue;}privatevoidsaveLogAfterCommit(StringorderNo,Stringdata,Stringflag,StringerrorMsg){TransactionSynchronizationManager.registerSynchronization(newTransactionSynchronization(){OverridepublicvoidafterCommit(){try{PaymentCallbackLoglognewPaymentCallbackLog();log.setOrderNo(orderNo);log.setCallbackData(data);log.setProcessFlag(flag);log.setErrorMsg(errorMsg);log.setCreateTime(newDate());paymentLogRepository.save(log);}catch(Exceptione){log.warn(事务后日志记录失败, orderNo{},orderNo,e);}}});}}5.2 优缺点优点缺点保证日志只在业务成功后写入实现稍复杂避免事务回滚但日志已存在的问题日志写入仍是同步的在事务提交后日志不参与业务事务不会导致回滚应用崩溃在 commit 后、日志写入前会丢失5.3 适用场景日志必须与业务结果一致成功才记录不希望日志写入失败导致业务回滚对日志写入时机有严格要求六、方案四MQ 异步写入6.1 实现方式ServicepublicclassPaymentCallbackService{ResourceprivateKafkaTemplateString,StringkafkaTemplate;Transactional(rollbackForException.class)publicBooleanhandleCallback(PaymentCallbackDtocallbackDto){// 业务处理...// 发送日志消息到 MQLogMessagemsgnewLogMessage();msg.setOrderNo(callbackDto.getOrderNo());msg.setData(JSON.toJSONString(callbackDto));msg.setFlag(Y);msg.setTimestamp(newDate());kafkaTemplate.send(payment-log-topic,JSON.toJSONString(msg));returntrue;}}// 消费者异步写入数据库ComponentpublicclassPaymentLogConsumer{ResourceprivatePaymentLogRepositorypaymentLogRepository;KafkaListener(topicspayment-log-topic)publicvoidconsume(Stringmessage){try{LogMessagemsgJSON.parseObject(message,LogMessage.class);PaymentCallbackLoglognewPaymentCallbackLog();log.setOrderNo(msg.getOrderNo());log.setCallbackData(msg.getData());log.setProcessFlag(msg.getFlag());log.setCreateTime(msg.getTimestamp());paymentLogRepository.save(log);}catch(Exceptione){log.error(MQ日志消费失败: {},message,e);}}}6.2 优缺点优点缺点完全异步对主流程零影响架构复杂度高依赖 MQ 中间件MQ 有持久化机制不易丢失日志写入有延迟可批量消费写入效率高需要处理消费失败和重试天然削峰填谷运维成本增加6.3 适用场景超高并发场景万级 QPS项目已有 MQ 基础设施日志写入延迟可接受需要削峰填谷能力七、异常捕获策略7.1 核心原则日志记录永远不能影响主业务流程。无论采用哪种方案都必须确保日志写入失败时主流程正常返回。7.2 三层防护privatevoidsaveLog(StringorderNo,Stringdata,Stringflag,StringerrorMsg){try{// 第一层参数安全处理if(data!nulldata.length()65535){datadata.substring(0,65535);// 防止超出 TEXT 字段长度}if(errorMsg!nullerrorMsg.length()500){errorMsgerrorMsg.substring(0,500);// 防止超出 VARCHAR 长度}// 第二层数据库写入PaymentCallbackLoglognewPaymentCallbackLog();log.setOrderNo(orderNo);log.setCallbackData(data);log.setProcessFlag(flag);log.setErrorMsg(errorMsg);log.setCreateTime(newDate());paymentLogRepository.save(log);}catch(DataAccessExceptione){// 第三层数据库异常连接超时、死锁等log.warn(日志入库失败(数据库异常), orderNo{}, error{},orderNo,e.getMessage());}catch(Exceptione){// 兜底任何未预期的异常log.warn(日志入库失败(未知异常), orderNo{}, error{},orderNo,e.getMessage());}}7.3 常见异常场景异常类型原因处理方式DataIntegrityViolationException字段超长、非空约束截断数据后重试或直接跳过QueryTimeoutException数据库响应慢记录文件日志跳过DeadlockLoserDataAccessException死锁可重试一次或跳过CannotAcquireLockException锁等待超时跳过记录文件日志TransientDataAccessException临时性故障可重试八、事务边界考量8.1 日志与业务同事务Transactional(rollbackForException.class)publicvoidprocess(){// 业务操作orderRepository.save(order);// 日志操作同事务logRepository.save(log);}风险日志写入失败会导致整个事务回滚业务也失败。解决用 try-catch 包裹日志写入或将日志放在事务外。8.2 日志独立事务REQUIRES_NEWServicepublicclassLogService{Transactional(propagationPropagation.REQUIRES_NEW)publicvoidsaveLog(PaymentCallbackLoglog){paymentLogRepository.save(log);}}效果日志在独立事务中失败不影响外层业务事务。注意REQUIRES_NEW 会挂起外层事务占用额外数据库连接。高并发下可能耗尽连接池。8.3 无事务写入// 日志方法不加 Transactional使用自动提交privatevoidsaveLog(...){try{paymentLogRepository.save(log);// 自动提交}catch(Exceptione){// 忽略}}效果最简单的隔离方式日志写入不参与任何事务。注意如果外层有事务且 Repository 被事务管理器管理实际上仍会加入外层事务。需要确认 Spring 事务传播行为。九、方案对比总结维度同步 try-catchAsync事务后写入MQ 异步实现复杂度⭐⭐⭐⭐⭐⭐⭐⭐⭐主流程影响有写入耗时无有commit 后同步无数据一致性强弱中弱日志丢失风险低中应用崩溃中低MQ 持久化高并发适应性差中中优事务隔离需手动隔离天然隔离天然隔离天然隔离适用 QPS 500500 ~ 5000500 ~ 5000 5000十、最佳实践清单永远用 try-catch 包裹日志写入无论同步还是异步字段长度防护写入前截断超长字段避免 DataIntegrityViolation日志方法独立抽取为 private/独立 Service 方法职责单一选择合适方案低并发用同步高并发用异步或 MQ监控日志写入失败率通过文件日志 warn 级别记录失败便于排查避免日志表成为热点考虑分表、定期归档异步方案设置合理的线程池/队列大小防止 OOMMQ 方案处理消费失败设置死信队列或重试机制