Jonas Notes

2025-10-26

4 min read

Weekly Engineering Report — 20 – 26 Oct 2025

Snapshot

  • Dataservice Portal dominated the week: heavy observability fixes plus AV management and mock API overhauls.
  • No material commits landed in dataservice-query, bigdataplatform-gateway, or other dataservice repos during this window.
  • Core themes: trace viewer stability, performance dashboards, mock preview robustness, and AV auto-refresh parity.
graph TD
    Jonas[Jonas Goh] -->|TraceView refactor| Monitoring
    Jonas -->|API performance metrics| Monitoring
    Jonas -->|Tracing log parity| Platform
    Jingying[Wang Jingying] -->|Mock preview refactor| Mocking
    Jingying -->|Validation hardening| Mocking
    Yue[Yue Wang] -->|AV insights & refresh| AV
    Julie[Julie Zhu] -->|App catalog endpoints| AppMgmt
    Julie -->|StarRocks sync| Infra
    Mengyang[Mengyang Chen] -->|Random rule validation| Mocking

Contribution Metrics

ContributorCommits+ Lines- Lines
Jonas Goh401,087733
Wang Jingying13439343
Yue Wang23910
Julie Zhu21816
Mengyang Chen1115

Contributor Highlights

Jonas Goh

  • Reworked TraceViewService to unify data-api vs service-api lookups behind a single getTraceList path, layering in request-type helpers (TraceViewRequestDTO#isDataApiTraceQuery) and BFS span exclusion to suppress gateway duplication (data-service-portal/api/src/main/java/com/shopee/data/dataservice/monitor/service/TraceViewServiceImpl.java).
  • Expanded QueryStatusEnum with HTTP status prefix helpers and mapped Bithon spans into richer DTOs, exposing IP resolution and tag guards so UI charts gain accurate status bins (data-service-portal/api/src/main/java/com/shopee/data/dataservice/common/constant/QueryStatusEnum.java, .../dependencies/bithon/entity/BithonTraceByIdResponseDto.java).
  • Hardened trace metrics: consistent null handling, corrected duration math, and centralized the TraceViewController entry to always resolve trace lists before pagination (.../monitor/controller/TraceViewController.java, .../monitor/dto/TraceDTO.java).
  • Overhauled API performance dashboards by shifting to Prometheus-backed filters, adding human-readable units/formatting, and parallelizing metric fetches with CompletableFuture for latency/QPS payload sections (.../monitor/service/ApiPerformanceServiceImpl.java, .../monitor/dto/MonitorMetricDTO.java).
  • Enabled enhanced Feign logging across all envs and tightened mapper constants/formatting to remove double-counting in monitoring totals (.../resources/application-*.yml, .../monitor/service/RequestMonitorServiceImpl.java).
  • Review: High-impact refactors with solid guard-rails; biggest risk is the wide surface touched in TraceViewServiceImpl, so regression coverage on span filtering and PromQL calls is essential.

Wang Jingying

  • Carved mock preview models (MockCol, MockRule, IMockCol) into the shared dependencies module so portal and query service consume one validation contract, and rewired DTOs to use the new package (data-service-portal/dependencies/src/main/java/com/shopee/data/dataservice/dependencies/query/entity/mock/*.java).
  • Refined validation semantics: stricter random number ranges (string-based min/max with numeric verification), static value length/type guards, array length enforcement, and comprehensive ENUM compatibility checks (MockCol.validate* helpers).
  • Iterated on error ergonomics with clearer MockCol.check messages, normalized mock data type names, and ensured marker offsets stay string-safe for downstream consumers.
  • Review: Strong focus on input hygiene; ensure query-service consumers are bumped in lockstep or feature flags added, otherwise incompatible DTO expectations could surface at runtime.

Yue Wang

  • Added the AV name into validation responses so editors see which acceleration view gates edits (.../api/src/main/java/com/shopee/data/dataservice/av/dto/AvValidateResult.java, AvManagementServiceImpl).
  • Let API lifecycle hooks toggle StarRocks materialized view auto-refresh by inferring MV names via AccelerationViewApiDomain, falling back to legacy namespace lookups for older data (.../speedup/OlapSpeedupSourcesManager.java, SpeedupManager).
  • Review: Sensible enhancements; new AV auto-refresh path depends on domain data completeness—monitor for missing AccelerationViewApiModel links in older namespaces.

Julie Zhu

  • Introduced /all/project/apps to surface the full project/app catalog, refactoring service/domain layers to fetch and group every app instead of user-limited subsets (.../app/controller/AppManagementController.java, AppServiceImpl).
  • Relaxed StarRocks cluster syncs to tolerate missing credentials, defaulting null username/password to blanks when building updater payloads (.../tasking/task/StarrocksClusterFromRamUpdater.java).
  • Review: Changes reduce friction but broaden exposure; audit auth checks around the new “all apps” API and verify downstream clients can tolerate blank StarRocks credentials.

Mengyang Chen

  • Shifted mock random-number bounds to strings and reworked validation to reject empty inputs while comparing parsed doubles, preventing silent 0-defaults that were slipping through (.../api/dto/mock/MockPreviewRequest.java, MockRule).
  • Review: Targeted fix with low blast radius; keep an eye on serialization since min/max are now strings—downstream services must accept the new types.

Repository Coverage

  • data-service-portal: active across monitoring, mock, AV, app-management, and infra modules.
  • dataservice-query, bigdataplatform-gateway, dataservice-scheduled-tasks, SDK repos: no commits detected between 20 – 26 Oct 2025.

Follow-ups & Risks

  • Trace viewer refactor introduces broader span filtering; recommend targeted regression tests on cache-hit traces and mixed gateway/query spans.
  • Mock validation moved to shared models—ensure downstream query service syncs to the same commit before tagging a release.
  • AV auto-refresh now depends on acceleration view linkages; confirm production has backfilled AccelerationViewApiModel records to avoid skipping refresh toggles.