Examining Checkpoint Records with the crcat Tool

Examining Checkpoint Records with the `crcat` Tool

The crcat command-line tool allows users and administrators to view the checkpoint record for any Deephaven table location directory. Checkpoint records are consistent snapshots of all the data necessary to continue appending to a given location, including the location size, last modification time, size for each constituent file, source file state, and import state when appropriate, and are usually stored in a file named table.size.

The crcat tool can be useful for verifying that a table location directory has a consistent set of files, seeing how far a near real-time import has progressed, or verifying that data has been committed to persistent storage. The tool is typically installed in the "bin" directory of an Deephaven installation (e.g. /usr/illumon/lastest/bin on Linux), and requires a valid JRE installation identified by the JAVA_HOME environment variable. It depends on certain JAR files installed with Deephaven, which it finds relative to its own path.

The following example command asks for the checkpoint record for Intraday location vm-1/2018-06-07 of the table ProcessEventLog in the DbInternal namespace:

/usr/illumon/latest/bin/crcat /db/Intraday/DbInternal/ProcessEventLog/vm-1/2018-06-07/ProcessEventLog

Sample output:

Reading checkpoint record from /db/Intraday/DbInternal/ProcessEventLog/vm-1/2018-06-07/ProcessEventLog:
CheckpointRecord[
		version=1, //the version of the checkpoint record format
		TableLocationState[size=11132, lastModificationTime=2018-06-07T11:36:09.565-0400],
		DataFileSizeRecords[
			[name=EffectiveUser.dat, size=44528],
			[name=Process.dat, size=44528],
			[name=Host.sym, size=8],
			[name=Level.sym, size=48],
			[name=LogEntry.dat, size=89056],
			[name=Level.sym.bytes, size=30],
			[name=EffectiveUser.sym, size=8],
			[name=LogEntry.bytes, size=1716844],
			[name=Process.sym, size=96],
			[name=AuthenticatedUser.sym, size=8],
			[name=Process.sym.bytes, size=120],
			[name=Host.dat, size=44528],
			[name=EffectiveUser.sym.bytes, size=4],
			[name=AuthenticatedUser.sym.bytes, size=4],
			[name=Timestamp.dat, size=89056],
			[name=Host.sym.bytes, size=58],
			[name=Level.dat, size=44528],
			[name=AuthenticatedUser.dat, size=44528]
		],
SourceFileSizeRecord[name=DbInternal.ProcessEventLog.localhost.bin.2018-06-07.110000.000-0400, size=642414], //latest file ingested and size of data ingested so far
		ImportState[
			class=class com.illumon.iris.db.tables.dataimport.logtailer.ImportStateRowCounter,
			details=[
			 [nRows, 11132]
			]
		]
	]

Last Updated: 16 February 2021 18:06 -04:00 UTC Deephaven v.1.20200928 (See other versions)