Data Miner operation and interpretation
Mit FlammiCraft ausgelieferte Mod-Version: 1.13.0
Diese Detailseite ist derzeit auf Englisch verfügbar.
Requirements
- Minecraft 1.21.1
- NeoForge 21.1.256+
- Java 21
- No dependencies
Quick Start
- Drop
data_miner-*.jarinmods/(client and/or server). - For a client, add
-Ddata_miner.enabled=trueto the launcher's Java/JVM arguments. Dedicated servers enable Data Miner automatically. - Start the process. Every enabled launch automatically dumps all nine supported registries and writes startup reports into a new session directory.
- Basic tick monitoring and latency tracing start automatically when a server starts. Use the commands below for additional recordings or another dump.
JVM parameters
| Parameter | Effect | Default without the parameter |
|---|---|---|
-Ddata_miner.enabled=true |
Enable Data Miner, including all automatic startup dumps. Set false to disable diagnostics for this process. |
Disabled on physical clients, including singleplayer/integrated servers; enabled on dedicated servers. |
-Ddata_miner.detailedProfiling=true |
Add method timings and mod counters through optional profiling hooks. Set false to leave these hooks disabled. Requires Data Miner to be enabled. |
Disabled on both clients and dedicated servers. |
For client registry dumps and basic diagnostics, use:
-Ddata_miner.enabled=true
For a client with additional method profiling, use both parameters:
-Ddata_miner.enabled=true -Ddata_miner.detailedProfiling=true
For a dedicated server with additional method profiling, add:
-Ddata_miner.detailedProfiling=true
Put these in the launcher's JVM arguments, or before -jar in a Java command.
Changes require a full process restart. Values must be true or false;
invalid activation values disable diagnostics and are logged. Detailed profiling
adds measurement work and should be enabled when those timings are needed.
Data Miner no longer registers a mod config or reads startup settings from files.
Existing config/data_miner-common.toml and config/data_miner-startup.properties
files are ignored and left unchanged. There are no individual dump switches.
Dumps and session output
Each enabled process writes into a separate session directory:
data_miner/info_client/<UTC-start-timestamp>_<UUID>/
data_miner/info_server/<UTC-start-timestamp>_<UUID>/
All nine registry dumps are written under startup/registries/: blocks, items,
entity types, status effects, sound events, creative tabs, potions, villager
professions and attributes. Dynamic biome, enchantment and dimension registries
are outside the supported dumps.
| Output in the session directory | Contents |
|---|---|
startup/ |
Registry dumps, loaded mods, environment information, mod analyses and artifact/configuration fingerprints. |
performance/ |
Tick reports, continuous windows.jsonl, minute checkpoints, resource samples, stall observations and slow-tick incidents. |
performance/latency/ |
Latency observations and checkpoints. |
events/ |
Explicitly started event recordings and their checkpoints. |
logs/ |
Filtered logs, bootstrap log snapshot, capture status and aggregated content issues. |
session_state.json |
Export progress and observed completion state. |
Historical sessions are retained. Repeating a registry dump updates that session's registry files. JSONL timelines grow on disk; memory buffers and worker queues are bounded. Checkpoints are saved every minute for active recordings. A forced kill or failed export can leave gaps or lose observations since the last checkpoint. Log snapshots retain source text; review them before sharing.
Commands
Commands require operator permission level 2. Work that writes reports is queued to background workers.
| Command | Action |
|---|---|
/data_miner dump |
Repeat all supported registry dumps. |
/data_miner mods analyze |
Regenerate the mod analysis. |
/data_miner perf start / /data_miner perf stop |
Start a performance recording, or stop and export it. |
/data_miner events start / /data_miner events stop |
Start a detailed event recording, or stop and export it. |
/data_miner latency start / /data_miner latency stop |
Start latency tracing, or stop and export it. A manual stop lasts until the next explicit start or server start. |
/data_miner latency stats |
Display statistics while latency tracing is running. |
Compare performance reports using quoted paths relative to data_miner/info_server/
or data_miner/info_client/:
/data_miner compare "<session>/performance/<before>.json" "<session>/performance/<after>.json"
For explicit external log correlation, the read-only CLI writes JSON to stdout:
rtk proxy python -X utf8 tools/correlate_diagnostics.py "SESSION_DIRECTORY" --log-offset +02:00 --source "SERVER_LOG"
Use the actual fixed UTC offset of timezone-less log lines for the selected run.
Additional --source arguments can include console logs, GC logs and restart
metrics. The tool reports coverage and parsing gaps.
The same command also reads retained travel_chunk_requests snapshots from up to
32 recent candidate performance JSON files, each limited to 8 MB. It groups
request, phase and admission-gate IDs only within the same snapshot and matching
recording. Checkpoints remain separate; their repeated observations are never
summed. Shared holder work, overlapping envelopes, open gates and missing clocks
do not identify exclusive IO, queue, generation or cancellation causes. Missing,
unsupported and disabled data remain distinct. Relative offsets have a calendar
anchor only when the report declares its monotonic recording mapping. Files are
selected by modification time for bounded reading, not to prove recording order.
Interpreting diagnostics
Process-resource observations
performance/resources.jsonl describes the current JVM process, including JVM
internal threads. process_cpu_load is a 0–1 fraction across all CPUs in the JMX
operating environment, not the utilization of the server thread. The JVM-visible
processor count is recorded as context; it does not establish the historic host
CPU count or available server-thread headroom. CPU time is a cumulative process
counter, not a tick or interval duration.
JMX defines its load window only as recent: process_cpu_load_interval_ms is
null. The observer's nominal minimum 10-second poll spacing and observed
wall-clock spacing do not define that CPU window. First-poll or regressing-clock
spacing remains null. Read timestamps and monotonic read duration bound this
observation, but cannot establish the underlying JMX refresh age. Numeric values
with a status use null for unavailable, failed, invalid or unsupported readings;
an available zero remains zero. A failed process counter does not hide the other
process counters. These observations do not identify another process or exclusive
disk/queue work.
Server tick boundary and CPU
Schema 5 measures the virtual tickServer(BooleanSupplier) invocation made by
MinecraftServer.runServer. It includes work before and after a base-method call
in a server override, late Post listeners, synchronous waits and observer work
inside that invocation. Runloop tasks and waiting outside the invocation, and
diagnostic finalization after the end marker, remain outside the measurement.
The hook is part of basic diagnostics; optional method profiling is independent.
Only a normally returned invocation with exactly one valid server Pre/Post event
pair in the same recording contributes to mspt, the tick distribution and the
dimension denominator. outer_calls separately counts completed ticks, partial
calls, invalid clocks, aborted calls, calls without events and invalid event
pairs. These six classes sum to observed_calls. Calls without events can include
integrated-server pause work; their presence alone does not establish a pause.
Their bounded duration remains visible without inventing a completed tick.
legacy_event retains the NORMAL Pre-to-NORMAL Post-entry wall span for the same
completed ticks. server_thread_cpu measures the owning thread's CPU delta across
the outer invocation. Missing, disabled, failed or regressing counters are
explicitly unavailable; valid zero remains zero. CPU means use valid CPU samples,
and are neither process CPU percentages nor a measure of pure waiting time.
Recording and window durations use monotonic elapsed time with a fixed calendar anchor. This avoids negative intervals after a clock change, but does not repair correlation with external logs after such a change. A window with no completed ticks has null duration statistics and explicit coverage, rather than 0-ms ticks. Starting or stopping within a call records a partial call; stopped reports freeze immediately and a late return cannot alter them or a subsequent recording.
Slow-call incidents and in-progress stalls share the invocation ID and explicitly
state whether they count as completed ticks. Only completed tick incidents carry
an mspt sample. Observer timings identify their measured code sections; they
are not subtracted from wall time and do not claim to capture all diagnostic cost.
Item-transfer target samples
With detailed profiling enabled at startup, performance reports and checkpoints
include d1_transfer_samples (its own schema 1). The sampler selects at entry to
TDEnderPipes' target insert attempt, before its result is known, using a private
1/64 random decision. It admits at most eight attempts per network invocation
and 64 per observed server call, retains 64 classification buckets and the last
128 detail rows, and exports all budget omissions and overflows. Reaching a
budget censors the sample; it does not remain an unrestricted random sample.
The observations distinguish simulation, extraction, execution and restoration,
native index/bulk/fallback calls, actual wrapper/child receiver roles and native
guard allow/deny decisions. They add no capability, filter, permission or slot
queries. A raw index result of -2 has an unknown unsupported reason unless the
executed branch is separately observed; a positive candidate/start is not proof
of an indexed leaf hit. A guard denial can also mean missing runtime context.
Fine coverage begins at the target attempt. Earlier source-slot search,
simulation, filters, route selection and attempts that never reach insert
remain outside it. Times are nested inclusive observations of selected attempts,
not exclusive claim or leaf costs. Do not extrapolate sample times into complete
MSPT or subtract them from the network timer. Existing detailed timers retain
their own overhead. Reports without this block did not measure these samples;
missing hooks and missing phases must not be read as zero cost.
An independent, bounded network-call sample adds preflight to the same block.
It observes native source opening, draining and routing even when no target
attempt is reached. Its phases are inclusive and routing still contains target
work; counters describe executed calls rather than inferred rejection reasons.
See the handbook for sampling and interpretation limits.
Sophisticated save phases
When detailed profiling was enabled at process startup, performance reports and
checkpoints include saved_data_phases (its own schema 1). It selects the exact
Sophisticated Backpacks storage File-save path and distinguishes the caller's
save duration, virtual serialization, contents/access-log assembly, detached
snapshot capture, native submission, native-worker execution and compressed write.
Helper durations are included within serialization; worker work can overlap the
caller. Adding these durations does not yield a tick duration or a useful total.
Capture records the observed copy/capture call span. The status
observed_order_unverified keeps that raw span separate from a claim that it
covers an entire snapshot provider; wrapper order needs evidence for the actual
mod combination. Missing, unapplied or unobserved hooks remain explicit; their
durations are unknown rather than zero. Serialized counts refer to entries in the actual
returned standard NBT lists, without scanning the live storage maps. These phase
observations retain no backpack contents, player names, file paths or backpack UUIDs.
The compressed-write duration includes encoding, compression, temporary-file handling, forcing the file and replacement. It does not isolate hardware I/O or prove durability. A native worker can return normally after its own IOException handler ran; write and worker outcomes are therefore separate. Submission-to-start begins before submission, includes scheduling/submission work, and is not a pure queue-wait measurement. A submitted task whose predecessor failed may never start.
The collector retains at most 256 open attempts and the latest 256 completed detail
rows, with separate capacity and eviction counts. Recording stop freezes a snapshot
immediately and leaves unfinished or uncertain handoffs explicit. Native tasks
continue; late completions cannot change that snapshot or a new recording.
recording_complete=true therefore does not imply all save I/O completed.
No new executor, save, flush, queue reset or export file is introduced by tracing.
Reports without this block did not collect these phases.
Travel and chunk requests
With detailed profiling enabled at startup, existing performance reports and
checkpoints include travel_chunk_requests (its own schema 1). Native player
portal destination preparation, player dimension commits and actual player
movement supply caller scopes. Optional TDUtilities hooks separate its inclusive
portal search, POI loading/validation and creation calls. Search includes the POI
phase; these durations cannot be added as independent costs. A mining dimension
alone does not identify the caller.
The recorder observes actual ServerChunkCache.getChunkFutureMainThread calls,
requested status/create flag, the original returned future, and the native
getChunk managed wait and owner-thread join result. request_nanos ends at
future handoff; method_nanos runs from the cold request entry to getChunk
return, excluding earlier cache checks. Request start offsets share the existing
recording's monotonic anchor; the outer invocation ID is included when observed.
Warm returns create no request row and
read no tracer clock. Direct getChunkFuture callers have request/future data,
but their separate managed wait and join are not measured by this hook. Off-thread
entries are not attributed to an owner-thread travel. A terminal future with no
exception does not prove a successful ChunkResult; the original join result is
reported separately. Multiple requests may share a future and overlap.
Managed waiting can run unrelated server tasks. Caller and portal-phase context is suspended for those tasks and restored after waiting. Preparation and commit are linked only through the exact player and transition identities. Following movement can carry that travel ID for at most five seconds after a successful commit, in the target dimension. This is a correlation, not proof that travel caused the request. Only bounded weak identity bindings are retained; reports contain neither player identifiers nor chunk coordinates. Recording-local target IDs mean the same requested dimension/coordinates, not a unique generation job.
At most 512 request rows, 128 travel rows, 64 weak bindings and 32 spans per travel are retained. Unknown/unscoped requests select every 64th entry, starting with the first, before observing its result. Known caller scopes are recorded without that sampling. A process-wide cap of 128 outstanding future callbacks survives recording restarts, so never-completing futures cannot accumulate unbounded observers. Evictions, sampling omissions, callback limits and clock failures have explicit coverage counters. Stopped reports and captured checkpoints are detached from later completions. No chunk lookup, ticket, future replacement, new executor or world write is introduced. Initial chunk status and exclusive IO, queue and generation time remain unobserved. This instrumentation does not itself remove chunk waits or establish an MSPT improvement.
Optional C2ME pipeline hooks add pipeline_id/binding offsets to requests and
pipeline_phases to the same report block. The binding is taken from the actual
C2ME getFutureForStatus0 call, restricted to the exact native ChunkHolder receiver
at the server chunk request's original scheduling operation. Incidental scheduling
during ticket/distance-manager updates cannot bind another holder to that request.
Holder identities are weak, recording-local, and never matched by coordinates or
time alone. Up to 128 holders and the last 256 phase operations are retained.
Request and phase observers share the existing global limit of 128 callbacks.
chunk_read_upgrade observes the original asynchronous read/upgrade helper;
load_step and generation_step observe the actual original ChunkStep.apply
operations inside C2ME's subscribed execution path. Their invocation_nanos ends
when the original future is returned; completion_nanos ends when its terminal
callback is observed. No Rx subscription, worker, future replacement or extra
world query is added. A read's normal Optional presence/absence is recorded without
inspecting NBT; absence does not prove that a new chunk was generated.
The relation means the same observed backing holder. Work may be shared, already running, unrelated to the sampled requested status, or observed after its request completed. Phases that began before binding remain unobserved. These are inclusive asynchronous envelopes, not exclusive disk, queue, CPU or generation time. The generation-step hook starts after C2ME's spatial lock admission. The separate admission observations below can cover part of the preceding interval; individual neighbor/dependency jobs are not reconstructed. The synchronous read fallback is outside this first pipeline hook. Observer overhead remains in the durations. Recordings made before these hooks have no pipeline rows; empty rows do not prove the corresponding work was absent.
Optional C2ME admission hooks add pipeline_gates to the same report, with up to
128 weak action bindings and the last 256 rows. status_dependency_admission
starts at the original dependency-future factory and ends before the deferred
context creation. It follows the same freshly created native context-reference
object, including graph assembly, subscription, dependencies and critical-executor
handoff. generation_task_admission starts before the original lock-radius task
helper and ends before that task calls its original supplier. It follows the
unchanged supplier and includes helper preparation, lock admission and scheduling.
These inclusive, possibly overlapping intervals are not exclusive dependency or
lock times and cannot be added to the future envelopes to obtain a total.
Each row reports gate_id, pipeline_id, kind, start_offset_nanos,
admission_nanos and outcome. A reached gate does not prove successful work
after that gate. scheduling_threw identifies an observed synchronous factory or
helper exception. An open row can mean pending, failed, cancelled, collected or
cut by recording stop; these causes are not guessed. Reusing one key while its
previous admission is still open makes that association ambiguous_key_reuse
instead of silently attributing the next callback. Missing time remains null.
The hooks preserve the original Rx chain, subscription count, future, supplier, enqueue and callback operations. They add no future observers, world queries, executor or cancellation polling. The existing startup detail switch is required; no new configuration is introduced. Before-binding work, initial chunk status, exclusive IO/CPU/queue time and dependency edges remain unknown. The new hooks improve observation coverage, not chunk availability or server MSPT by themselves.
Dimension costs
Basic performance recordings automatically measure every observed logical-server
dimension tick, without -Ddata_miner.detailedProfiling=true. Full reports and
minute checkpoints contain dimension_metrics; completed ten-second windows in
performance/windows.jsonl and retained report windows contain the same breakdown.
Windows can be shorter at server-phase transitions. Each exported performance
report also has a matching *_dimensions.csv with a cost ranking for spreadsheets.
For each dimension, the report includes:
- Average, minimum, median, P95, P99 and maximum milliseconds per observed server tick of that dimension, plus counts above 50, 100, 250 and 1,000 ms. Percentiles use the existing one-millisecond histogram resolution; overflow above 1,000 ms uses the observed maximum.
- Total measured milliseconds, milliseconds per all server ticks in the interval, share of measured server time and share of measured dimension time. Rows are ranked by total measured cost. Multiple tick calls in one server tick are combined for the MSPT distribution and counted separately as calls.
- Achieved TPS: completed dimension tick calls divided by the full recording or
window wall-clock duration. Dimensions normally share the server's cadence;
1000 / dimension MSPTis not an independent dimension TPS measurement. Late loads, unloads, pauses and missing events lower this full-interval rate. - Observed server-tick fraction and last sampled player/loaded-chunk counts with
their timestamp. Counts are sampled around window boundaries, not averaged or
scanned every tick;
-1means unavailable. A dimension without a completed observation is explicitly labelled, rather than established as zero cost.
The measurement spans Data Miner's LevelTickEvent.Pre listener at HIGHEST
priority through its Post listener at LOWEST priority, on the server thread.
It includes synchronous nested work and waits within that boundary, including
most level-event listeners. Ordering among listeners with the same priority is
not guaranteed. This is elapsed wall time, not exclusive CPU time. Detached
workers and work moved outside these events by a scheduler cannot be fully charged
to a dimension. Shared server work such as connections and global callbacks stays
outside the measured dimension scopes. The server-minus-dimensions remainder is
reported separately; it is not proof that this time is unrelated to dimensions.
The server denominator uses only the same completed outer ticks; aborted and
partial-call dimension observations are discarded. The CSV appends the schema,
server boundary, dimension boundary and denominator to preserve this meaning.
Slow-tick incidents above 100 ms include dimension_costs_at_tick for that exact
tick. Overlapping scopes or dimension sums exceeding the measured server tick are
flagged as non-additive and excluded from the remainder distribution. Incomplete
Pre/Post pairs, rejected off-thread events and dimension-capacity rejections are
counted. Up to 128 distinct dimension IDs are retained per recording, and report
window retention remains bounded at 720; the JSONL timeline keeps older windows.
Use comparable normal-operation windows and their context when deciding where to
investigate. A dimension's measured cost is not a guaranteed saving from removing
it: cross-dimension systems and shared work can change their behavior.
Current reports use schema 5. Comparison can read older reports, but numerical
tick comparisons require explicitly compatible measurement boundaries and
methods in both reports and their windows. Schema 4 event timings cannot be
compared numerically with schema 5 outer timings. Unknown or conflicting window
metadata blocks a numerical result instead of silently claiming an improvement.
The correlation CLI preserves measurement metadata and labels unknown boundaries;
its coverage concerns windows and reports, not every individual incident file.
Basic tick
measurement begins at ServerStartedEvent. The first 120 seconds are labelled
warmup; that policy does not prove initialization is complete. Physical-client
FPS is separate from logical-server tick measurements.
Slow ticks above 100 ms retain bounded before/after context. Missing samples, buffer evictions, queue rejections and write failures are reported. Coinciding events and sampled stacks are observations, not proof of a spike's cause. Method timings include both inclusive and self time; adding parent and child inclusive totals double-counts work. Missing or unused optional hooks do not establish zero cost.
Food-use durations and observed block disappearance are not network latency. Block disappearance does not identify which player or world event removed the block. Suggested replacements for unknown content IDs require manual validation; Data Miner does not migrate worlds or inventories.
Builds and automated tests verify source behavior and packaging. Gameplay, compatibility with the full modpack and runtime overhead require a manual run.
For a fresh performance section, stop and verify the automatically started
recording before using perf start. Starting an active recording preserves it
and reports that it is already running. Caught write failures are retained in
export_worker.export_failures and prevent a completed process session.
The file comparison streams past unused dimension histories and tracer details. Its limits are 256 MB per input report, 720 retained windows and a read budget of eight million characters for selected comparison data; the timeline has a separate 128 MB limit. Original reports retain all their detail sections.
For time-only external log lines, a filename date or earlier dated line anchors
the day. Backward jumps of at least twelve hours infer the next day and increment
inferred_midnight_rollovers, even when the line itself is not a selected event.
Smaller backward jumps retain the day and are flagged as ambiguous clock or
log-order regressions. A large clock change cannot be reliably distinguished
from midnight by this rule. Invalid GC durations are counted and later valid
lines are still processed.