Zum Hauptinhalt springen

Data Miner operation and interpretation

Mit FlammiCraft ausgelieferte Mod-Version: 1.13.0

Diese Detailseite ist derzeit auf Englisch verfügbar.

Handbook overview

Requirements

  • Minecraft 1.21.1
  • NeoForge 21.1.256+
  • Java 21
  • No dependencies

Quick Start

  1. Drop data_miner-*.jar in mods/ (client and/or server).
  2. For a client, add -Ddata_miner.enabled=true to the launcher's Java/JVM arguments. Dedicated servers enable Data Miner automatically.
  3. Start the process. Every enabled launch automatically dumps all nine supported registries and writes startup reports into a new session directory.
  4. Basic tick monitoring and latency tracing start automatically when a server starts. Use the commands below for additional recordings or another dump.

JVM parameters

Parameter Effect Default without the parameter
-Ddata_miner.enabled=true Enable Data Miner, including all automatic startup dumps. Set false to disable diagnostics for this process. Disabled on physical clients, including singleplayer/integrated servers; enabled on dedicated servers.
-Ddata_miner.detailedProfiling=true Add method timings and mod counters through optional profiling hooks. Set false to leave these hooks disabled. Requires Data Miner to be enabled. Disabled on both clients and dedicated servers.

For client registry dumps and basic diagnostics, use:

-Ddata_miner.enabled=true

For a client with additional method profiling, use both parameters:

-Ddata_miner.enabled=true -Ddata_miner.detailedProfiling=true

For a dedicated server with additional method profiling, add:

-Ddata_miner.detailedProfiling=true

Put these in the launcher's JVM arguments, or before -jar in a Java command. Changes require a full process restart. Values must be true or false; invalid activation values disable diagnostics and are logged. Detailed profiling adds measurement work and should be enabled when those timings are needed.

Data Miner no longer registers a mod config or reads startup settings from files. Existing config/data_miner-common.toml and config/data_miner-startup.properties files are ignored and left unchanged. There are no individual dump switches.

Dumps and session output

Each enabled process writes into a separate session directory:

data_miner/info_client/<UTC-start-timestamp>_<UUID>/
data_miner/info_server/<UTC-start-timestamp>_<UUID>/

All nine registry dumps are written under startup/registries/: blocks, items, entity types, status effects, sound events, creative tabs, potions, villager professions and attributes. Dynamic biome, enchantment and dimension registries are outside the supported dumps.

Output in the session directory Contents
startup/ Registry dumps, loaded mods, environment information, mod analyses and artifact/configuration fingerprints.
performance/ Tick reports, continuous windows.jsonl, minute checkpoints, resource samples, stall observations and slow-tick incidents.
performance/latency/ Latency observations and checkpoints.
events/ Explicitly started event recordings and their checkpoints.
logs/ Filtered logs, bootstrap log snapshot, capture status and aggregated content issues.
session_state.json Export progress and observed completion state.

Historical sessions are retained. Repeating a registry dump updates that session's registry files. JSONL timelines grow on disk; memory buffers and worker queues are bounded. Checkpoints are saved every minute for active recordings. A forced kill or failed export can leave gaps or lose observations since the last checkpoint. Log snapshots retain source text; review them before sharing.

Commands

Commands require operator permission level 2. Work that writes reports is queued to background workers.

Command Action
/data_miner dump Repeat all supported registry dumps.
/data_miner mods analyze Regenerate the mod analysis.
/data_miner perf start / /data_miner perf stop Start a performance recording, or stop and export it.
/data_miner events start / /data_miner events stop Start a detailed event recording, or stop and export it.
/data_miner latency start / /data_miner latency stop Start latency tracing, or stop and export it. A manual stop lasts until the next explicit start or server start.
/data_miner latency stats Display statistics while latency tracing is running.

Compare performance reports using quoted paths relative to data_miner/info_server/ or data_miner/info_client/:

/data_miner compare "<session>/performance/<before>.json" "<session>/performance/<after>.json"

For explicit external log correlation, the read-only CLI writes JSON to stdout:

rtk proxy python -X utf8 tools/correlate_diagnostics.py "SESSION_DIRECTORY" --log-offset +02:00 --source "SERVER_LOG"

Use the actual fixed UTC offset of timezone-less log lines for the selected run. Additional --source arguments can include console logs, GC logs and restart metrics. The tool reports coverage and parsing gaps.

The same command also reads retained travel_chunk_requests snapshots from up to 32 recent candidate performance JSON files, each limited to 8 MB. It groups request, phase and admission-gate IDs only within the same snapshot and matching recording. Checkpoints remain separate; their repeated observations are never summed. Shared holder work, overlapping envelopes, open gates and missing clocks do not identify exclusive IO, queue, generation or cancellation causes. Missing, unsupported and disabled data remain distinct. Relative offsets have a calendar anchor only when the report declares its monotonic recording mapping. Files are selected by modification time for bounded reading, not to prove recording order.

Interpreting diagnostics

Process-resource observations

performance/resources.jsonl describes the current JVM process, including JVM internal threads. process_cpu_load is a 0–1 fraction across all CPUs in the JMX operating environment, not the utilization of the server thread. The JVM-visible processor count is recorded as context; it does not establish the historic host CPU count or available server-thread headroom. CPU time is a cumulative process counter, not a tick or interval duration.

JMX defines its load window only as recent: process_cpu_load_interval_ms is null. The observer's nominal minimum 10-second poll spacing and observed wall-clock spacing do not define that CPU window. First-poll or regressing-clock spacing remains null. Read timestamps and monotonic read duration bound this observation, but cannot establish the underlying JMX refresh age. Numeric values with a status use null for unavailable, failed, invalid or unsupported readings; an available zero remains zero. A failed process counter does not hide the other process counters. These observations do not identify another process or exclusive disk/queue work.

Server tick boundary and CPU

Schema 5 measures the virtual tickServer(BooleanSupplier) invocation made by MinecraftServer.runServer. It includes work before and after a base-method call in a server override, late Post listeners, synchronous waits and observer work inside that invocation. Runloop tasks and waiting outside the invocation, and diagnostic finalization after the end marker, remain outside the measurement. The hook is part of basic diagnostics; optional method profiling is independent.

Only a normally returned invocation with exactly one valid server Pre/Post event pair in the same recording contributes to mspt, the tick distribution and the dimension denominator. outer_calls separately counts completed ticks, partial calls, invalid clocks, aborted calls, calls without events and invalid event pairs. These six classes sum to observed_calls. Calls without events can include integrated-server pause work; their presence alone does not establish a pause. Their bounded duration remains visible without inventing a completed tick.

legacy_event retains the NORMAL Pre-to-NORMAL Post-entry wall span for the same completed ticks. server_thread_cpu measures the owning thread's CPU delta across the outer invocation. Missing, disabled, failed or regressing counters are explicitly unavailable; valid zero remains zero. CPU means use valid CPU samples, and are neither process CPU percentages nor a measure of pure waiting time.

Recording and window durations use monotonic elapsed time with a fixed calendar anchor. This avoids negative intervals after a clock change, but does not repair correlation with external logs after such a change. A window with no completed ticks has null duration statistics and explicit coverage, rather than 0-ms ticks. Starting or stopping within a call records a partial call; stopped reports freeze immediately and a late return cannot alter them or a subsequent recording.

Slow-call incidents and in-progress stalls share the invocation ID and explicitly state whether they count as completed ticks. Only completed tick incidents carry an mspt sample. Observer timings identify their measured code sections; they are not subtracted from wall time and do not claim to capture all diagnostic cost.

Item-transfer target samples

With detailed profiling enabled at startup, performance reports and checkpoints include d1_transfer_samples (its own schema 1). The sampler selects at entry to TDEnderPipes' target insert attempt, before its result is known, using a private 1/64 random decision. It admits at most eight attempts per network invocation and 64 per observed server call, retains 64 classification buckets and the last 128 detail rows, and exports all budget omissions and overflows. Reaching a budget censors the sample; it does not remain an unrestricted random sample.

The observations distinguish simulation, extraction, execution and restoration, native index/bulk/fallback calls, actual wrapper/child receiver roles and native guard allow/deny decisions. They add no capability, filter, permission or slot queries. A raw index result of -2 has an unknown unsupported reason unless the executed branch is separately observed; a positive candidate/start is not proof of an indexed leaf hit. A guard denial can also mean missing runtime context.

Fine coverage begins at the target attempt. Earlier source-slot search, simulation, filters, route selection and attempts that never reach insert remain outside it. Times are nested inclusive observations of selected attempts, not exclusive claim or leaf costs. Do not extrapolate sample times into complete MSPT or subtract them from the network timer. Existing detailed timers retain their own overhead. Reports without this block did not measure these samples; missing hooks and missing phases must not be read as zero cost.

An independent, bounded network-call sample adds preflight to the same block. It observes native source opening, draining and routing even when no target attempt is reached. Its phases are inclusive and routing still contains target work; counters describe executed calls rather than inferred rejection reasons. See the handbook for sampling and interpretation limits.

Sophisticated save phases

When detailed profiling was enabled at process startup, performance reports and checkpoints include saved_data_phases (its own schema 1). It selects the exact Sophisticated Backpacks storage File-save path and distinguishes the caller's save duration, virtual serialization, contents/access-log assembly, detached snapshot capture, native submission, native-worker execution and compressed write. Helper durations are included within serialization; worker work can overlap the caller. Adding these durations does not yield a tick duration or a useful total.

Capture records the observed copy/capture call span. The status observed_order_unverified keeps that raw span separate from a claim that it covers an entire snapshot provider; wrapper order needs evidence for the actual mod combination. Missing, unapplied or unobserved hooks remain explicit; their durations are unknown rather than zero. Serialized counts refer to entries in the actual returned standard NBT lists, without scanning the live storage maps. These phase observations retain no backpack contents, player names, file paths or backpack UUIDs.

The compressed-write duration includes encoding, compression, temporary-file handling, forcing the file and replacement. It does not isolate hardware I/O or prove durability. A native worker can return normally after its own IOException handler ran; write and worker outcomes are therefore separate. Submission-to-start begins before submission, includes scheduling/submission work, and is not a pure queue-wait measurement. A submitted task whose predecessor failed may never start.

The collector retains at most 256 open attempts and the latest 256 completed detail rows, with separate capacity and eviction counts. Recording stop freezes a snapshot immediately and leaves unfinished or uncertain handoffs explicit. Native tasks continue; late completions cannot change that snapshot or a new recording. recording_complete=true therefore does not imply all save I/O completed. No new executor, save, flush, queue reset or export file is introduced by tracing. Reports without this block did not collect these phases.

Travel and chunk requests

With detailed profiling enabled at startup, existing performance reports and checkpoints include travel_chunk_requests (its own schema 1). Native player portal destination preparation, player dimension commits and actual player movement supply caller scopes. Optional TDUtilities hooks separate its inclusive portal search, POI loading/validation and creation calls. Search includes the POI phase; these durations cannot be added as independent costs. A mining dimension alone does not identify the caller.

The recorder observes actual ServerChunkCache.getChunkFutureMainThread calls, requested status/create flag, the original returned future, and the native getChunk managed wait and owner-thread join result. request_nanos ends at future handoff; method_nanos runs from the cold request entry to getChunk return, excluding earlier cache checks. Request start offsets share the existing recording's monotonic anchor; the outer invocation ID is included when observed. Warm returns create no request row and read no tracer clock. Direct getChunkFuture callers have request/future data, but their separate managed wait and join are not measured by this hook. Off-thread entries are not attributed to an owner-thread travel. A terminal future with no exception does not prove a successful ChunkResult; the original join result is reported separately. Multiple requests may share a future and overlap.

Managed waiting can run unrelated server tasks. Caller and portal-phase context is suspended for those tasks and restored after waiting. Preparation and commit are linked only through the exact player and transition identities. Following movement can carry that travel ID for at most five seconds after a successful commit, in the target dimension. This is a correlation, not proof that travel caused the request. Only bounded weak identity bindings are retained; reports contain neither player identifiers nor chunk coordinates. Recording-local target IDs mean the same requested dimension/coordinates, not a unique generation job.

At most 512 request rows, 128 travel rows, 64 weak bindings and 32 spans per travel are retained. Unknown/unscoped requests select every 64th entry, starting with the first, before observing its result. Known caller scopes are recorded without that sampling. A process-wide cap of 128 outstanding future callbacks survives recording restarts, so never-completing futures cannot accumulate unbounded observers. Evictions, sampling omissions, callback limits and clock failures have explicit coverage counters. Stopped reports and captured checkpoints are detached from later completions. No chunk lookup, ticket, future replacement, new executor or world write is introduced. Initial chunk status and exclusive IO, queue and generation time remain unobserved. This instrumentation does not itself remove chunk waits or establish an MSPT improvement.

Optional C2ME pipeline hooks add pipeline_id/binding offsets to requests and pipeline_phases to the same report block. The binding is taken from the actual C2ME getFutureForStatus0 call, restricted to the exact native ChunkHolder receiver at the server chunk request's original scheduling operation. Incidental scheduling during ticket/distance-manager updates cannot bind another holder to that request. Holder identities are weak, recording-local, and never matched by coordinates or time alone. Up to 128 holders and the last 256 phase operations are retained. Request and phase observers share the existing global limit of 128 callbacks.

chunk_read_upgrade observes the original asynchronous read/upgrade helper; load_step and generation_step observe the actual original ChunkStep.apply operations inside C2ME's subscribed execution path. Their invocation_nanos ends when the original future is returned; completion_nanos ends when its terminal callback is observed. No Rx subscription, worker, future replacement or extra world query is added. A read's normal Optional presence/absence is recorded without inspecting NBT; absence does not prove that a new chunk was generated.

The relation means the same observed backing holder. Work may be shared, already running, unrelated to the sampled requested status, or observed after its request completed. Phases that began before binding remain unobserved. These are inclusive asynchronous envelopes, not exclusive disk, queue, CPU or generation time. The generation-step hook starts after C2ME's spatial lock admission. The separate admission observations below can cover part of the preceding interval; individual neighbor/dependency jobs are not reconstructed. The synchronous read fallback is outside this first pipeline hook. Observer overhead remains in the durations. Recordings made before these hooks have no pipeline rows; empty rows do not prove the corresponding work was absent.

Optional C2ME admission hooks add pipeline_gates to the same report, with up to 128 weak action bindings and the last 256 rows. status_dependency_admission starts at the original dependency-future factory and ends before the deferred context creation. It follows the same freshly created native context-reference object, including graph assembly, subscription, dependencies and critical-executor handoff. generation_task_admission starts before the original lock-radius task helper and ends before that task calls its original supplier. It follows the unchanged supplier and includes helper preparation, lock admission and scheduling. These inclusive, possibly overlapping intervals are not exclusive dependency or lock times and cannot be added to the future envelopes to obtain a total.

Each row reports gate_id, pipeline_id, kind, start_offset_nanos, admission_nanos and outcome. A reached gate does not prove successful work after that gate. scheduling_threw identifies an observed synchronous factory or helper exception. An open row can mean pending, failed, cancelled, collected or cut by recording stop; these causes are not guessed. Reusing one key while its previous admission is still open makes that association ambiguous_key_reuse instead of silently attributing the next callback. Missing time remains null.

The hooks preserve the original Rx chain, subscription count, future, supplier, enqueue and callback operations. They add no future observers, world queries, executor or cancellation polling. The existing startup detail switch is required; no new configuration is introduced. Before-binding work, initial chunk status, exclusive IO/CPU/queue time and dependency edges remain unknown. The new hooks improve observation coverage, not chunk availability or server MSPT by themselves.

Dimension costs

Basic performance recordings automatically measure every observed logical-server dimension tick, without -Ddata_miner.detailedProfiling=true. Full reports and minute checkpoints contain dimension_metrics; completed ten-second windows in performance/windows.jsonl and retained report windows contain the same breakdown. Windows can be shorter at server-phase transitions. Each exported performance report also has a matching *_dimensions.csv with a cost ranking for spreadsheets.

For each dimension, the report includes:

  • Average, minimum, median, P95, P99 and maximum milliseconds per observed server tick of that dimension, plus counts above 50, 100, 250 and 1,000 ms. Percentiles use the existing one-millisecond histogram resolution; overflow above 1,000 ms uses the observed maximum.
  • Total measured milliseconds, milliseconds per all server ticks in the interval, share of measured server time and share of measured dimension time. Rows are ranked by total measured cost. Multiple tick calls in one server tick are combined for the MSPT distribution and counted separately as calls.
  • Achieved TPS: completed dimension tick calls divided by the full recording or window wall-clock duration. Dimensions normally share the server's cadence; 1000 / dimension MSPT is not an independent dimension TPS measurement. Late loads, unloads, pauses and missing events lower this full-interval rate.
  • Observed server-tick fraction and last sampled player/loaded-chunk counts with their timestamp. Counts are sampled around window boundaries, not averaged or scanned every tick; -1 means unavailable. A dimension without a completed observation is explicitly labelled, rather than established as zero cost.

The measurement spans Data Miner's LevelTickEvent.Pre listener at HIGHEST priority through its Post listener at LOWEST priority, on the server thread. It includes synchronous nested work and waits within that boundary, including most level-event listeners. Ordering among listeners with the same priority is not guaranteed. This is elapsed wall time, not exclusive CPU time. Detached workers and work moved outside these events by a scheduler cannot be fully charged to a dimension. Shared server work such as connections and global callbacks stays outside the measured dimension scopes. The server-minus-dimensions remainder is reported separately; it is not proof that this time is unrelated to dimensions. The server denominator uses only the same completed outer ticks; aborted and partial-call dimension observations are discarded. The CSV appends the schema, server boundary, dimension boundary and denominator to preserve this meaning.

Slow-tick incidents above 100 ms include dimension_costs_at_tick for that exact tick. Overlapping scopes or dimension sums exceeding the measured server tick are flagged as non-additive and excluded from the remainder distribution. Incomplete Pre/Post pairs, rejected off-thread events and dimension-capacity rejections are counted. Up to 128 distinct dimension IDs are retained per recording, and report window retention remains bounded at 720; the JSONL timeline keeps older windows. Use comparable normal-operation windows and their context when deciding where to investigate. A dimension's measured cost is not a guaranteed saving from removing it: cross-dimension systems and shared work can change their behavior.

Current reports use schema 5. Comparison can read older reports, but numerical tick comparisons require explicitly compatible measurement boundaries and methods in both reports and their windows. Schema 4 event timings cannot be compared numerically with schema 5 outer timings. Unknown or conflicting window metadata blocks a numerical result instead of silently claiming an improvement. The correlation CLI preserves measurement metadata and labels unknown boundaries; its coverage concerns windows and reports, not every individual incident file. Basic tick measurement begins at ServerStartedEvent. The first 120 seconds are labelled warmup; that policy does not prove initialization is complete. Physical-client FPS is separate from logical-server tick measurements.

Slow ticks above 100 ms retain bounded before/after context. Missing samples, buffer evictions, queue rejections and write failures are reported. Coinciding events and sampled stacks are observations, not proof of a spike's cause. Method timings include both inclusive and self time; adding parent and child inclusive totals double-counts work. Missing or unused optional hooks do not establish zero cost.

Food-use durations and observed block disappearance are not network latency. Block disappearance does not identify which player or world event removed the block. Suggested replacements for unknown content IDs require manual validation; Data Miner does not migrate worlds or inventories.

Builds and automated tests verify source behavior and packaging. Gameplay, compatibility with the full modpack and runtime overhead require a manual run.

For a fresh performance section, stop and verify the automatically started recording before using perf start. Starting an active recording preserves it and reports that it is already running. Caught write failures are retained in export_worker.export_failures and prevent a completed process session.

The file comparison streams past unused dimension histories and tracer details. Its limits are 256 MB per input report, 720 retained windows and a read budget of eight million characters for selected comparison data; the timeline has a separate 128 MB limit. Original reports retain all their detail sections.

For time-only external log lines, a filename date or earlier dated line anchors the day. Backward jumps of at least twelve hours infer the next day and increment inferred_midnight_rollovers, even when the line itself is not a selected event. Smaller backward jumps retain the day and are flagged as ambiguous clock or log-order regressions. A large clock change cannot be reliably distinguished from midnight by this rule. Invalid GC durations are counted and later valid lines are still processed.