Package com.illumon.iris.binarystore

Class BinaryStoreWriterV2

java.lang.Object

com.illumon.iris.binarystore.AbstractBinaryStoreWriter<BinaryRowV2>

com.illumon.iris.binarystore.BinaryStoreWriterV2

All Implemented Interfaces:

Entry, Record, Row, TableWriter<BinaryRowV2>

Direct Known Subclasses:

BinaryStoreAggregatorWriterMultiPartition, BinaryStoreAggregatorWriterStandard

public class BinaryStoreWriterV2
extends AbstractBinaryStoreWriter<BinaryRowV2>

This class implements the writer for a self-descriptive binary table in the V2 binary log format.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static class	BinaryStoreWriterV2.DeferrableRowPartition	A container for binning WritableRowContainers by column partition in order to enable write batching.

Nested classes/interfaces inherited from interface com.illumon.iris.binarystore.Record

Record.RecordSetter<T extends RecordData>

Nested classes/interfaces inherited from interface com.illumon.iris.binarystore.Row

Row.Flags

Field Summary

Fields

Modifier and Type	Field	Description
protected com.fishlib.io.logger.Logger	log
protected BinaryRecordV2	primaryRecordWriter
protected Row	primaryRowWriter

Fields inherited from interface com.illumon.iris.binarystore.TableWriter

NULL_SETTER

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	BinaryStoreWriterV2(String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, Map<String,Object> constantColumnValues, int applicationVersion, int targetWriteSize, com.fishlib.io.logger.Logger log)
protected	BinaryStoreWriterV2(String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, Map<String,Object> constantColumnValues, int applicationVersion, com.fishlib.io.logger.Logger log)	Note: this is for internal use, and does not call setChannelManager or init().
	BinaryStoreWriterV2(String path, String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, com.fishlib.io.logger.Logger log)
	BinaryStoreWriterV2(String path, String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, DateFormat rollFormat, com.fishlib.io.logger.Logger log)
	BinaryStoreWriterV2(String path, String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, DateFormat rollFormat, Map<String,Object> constantColumnValues, int applicationVersion, int targetWriteSize, com.fishlib.io.logger.Logger log)
	BinaryStoreWriterV2(String path, String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, DateFormat rollFormat, Map<String,Object> constantColumnValues, int applicationVersion, com.fishlib.io.logger.Logger log)
	BinaryStoreWriterV2(String path, String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, DateFormat rollFormat, Map<String,Object> constantColumnValues, com.fishlib.io.logger.Logger log)
	BinaryStoreWriterV2(String path, String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, Map<String,Object> constantColumnValues, int applicationVersion, com.fishlib.io.logger.Logger log)
	BinaryStoreWriterV2(String path, String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, Map<String,Object> constantColumnValues, com.fishlib.io.logger.Logger log)

Method Summary

Modifier and Type	Method	Description
ByteBuffer	computeHeaderBuffer()	Compute the header buffer for this writer and return it in a new buffer (direct).
protected BinaryRecordV2	createBinaryRecord()	Implementation point for creating a binary record writer (BinaryRecordV2).
protected BinaryRowV2	createBinaryRow(String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata)
void	flushDeferredRows()	Write any rows buffered by TableWriter.writeDeferredRow(WritableRowContainer) to storage.
protected void	flushDeferredRows(BinaryStoreWriterV2.DeferrableRowPartition part)	Flush all of the deferred rows in the input collection to the underlying stream.
String[]	getColumnNames()	Gets the column names for the table.
String	getColumnPartitionValue()	For rows that are to be used with file managers that allow dynamic column partition selection, retrieve the column partition value.
Class[]	getColumnTypes()	Gets the column types for the table.
Record.RecordSetter<RecordData>	getRecordSetter()	Gets a record setter for an entry.
<T extends RecordData> Record.RecordSetter<T>	getRecordSetter(Class<T> tClass)	Gets a typed record setter for an entry.
BinaryRecordV2	getRecordWriter()	Get new writer for Record entries.
BinaryRowV2	getRowWriter()	Get a writer for a Row entries.
RowSetter	getSetter(String name)	Gets a setter for a column.
protected void	init()
static BinaryStoreWriterV2	makeCurrentDatePartitionedBinaryStoreWriter(com.fishlib.io.logger.Logger log, String logDir, String namespace, String tablename, TableIdentifier.NamespaceSet namespaceSet, String internalPartition, String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, Map<String,Object> constantColumnValues, int applicationVersion, ZoneId zoneId)	Make a BinaryStoreWriterV2 instance that changes the column partition value based on the current time, by changing the name of the rolling output file.
static BinaryStoreWriterV2	makeMultiPartitionFileManagerBinaryStoreWriter(com.fishlib.io.logger.Logger log, String logDir, String namespace, String tablename, TableIdentifier.NamespaceSet namespaceSet, String internalPartition, String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, Map<String,Object> constantColumnValues, int applicationVersion)	Make a BinaryStoreWriterV2 instance that uses dynamic (per-row) column partition values.
static BinaryStoreWriterV2	makeSpecifiedPartitionBinaryStoreWriter(com.fishlib.io.logger.Logger log, String logDir, String namespace, String tablename, TableIdentifier.NamespaceSet namespaceSet, String internalPartition, String columnPartition, String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, Map<String,Object> constantColumnValues, int applicationVersion, ZoneId zoneId)	Make a BinaryStoreWriterV2 with the specified values, including an unchanging column partition value.
void	setColumnPartitionValue(String columnPartitionValue)	For rows that are to be used with file managers that allow dynamic column partition selection, set the column partition value.
void	setFlags(Row.Flags flags)
void	setRecordData(RecordData recordData)
void	writeDeferredRow(WritableRowContainer<BinaryRowV2> row)	Write the row to disk.
protected void	writeMulti(ByteBuffer[] buffs, int count, GatheringByteChannel channel)	Perform a gathering write to the writer's channel.
void	writeRecord(WritableByteChannel channel)	Write a record entry to the provided output channel.
void	writeRow()	Writes out a new row (values set using setters).

Methods inherited from class com.illumon.iris.binarystore.AbstractBinaryStoreWriter

close, flush, setChannelManager, supportAllTypes

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.illumon.iris.binarystore.TableWriter

getSetter

Field Details

primaryRowWriter

protected Row primaryRowWriter

primaryRecordWriter

protected BinaryRecordV2 primaryRecordWriter

log

protected final com.fishlib.io.logger.Logger log

Constructor Details

BinaryStoreWriterV2

public BinaryStoreWriterV2(String path, String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, com.fishlib.io.logger.Logger log) throws IOException

Throws:

IOException

BinaryStoreWriterV2

public BinaryStoreWriterV2(String path, String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, DateFormat rollFormat, com.fishlib.io.logger.Logger log) throws IOException

Throws:

IOException

BinaryStoreWriterV2

public BinaryStoreWriterV2(String path, String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, Map<String,Object> constantColumnValues, com.fishlib.io.logger.Logger log) throws IOException

Throws:

IOException

BinaryStoreWriterV2

public BinaryStoreWriterV2(String path, String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, Map<String,Object> constantColumnValues, int applicationVersion, com.fishlib.io.logger.Logger log) throws IOException

Throws:

IOException

BinaryStoreWriterV2

public BinaryStoreWriterV2(String path, String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, DateFormat rollFormat, Map<String,Object> constantColumnValues, com.fishlib.io.logger.Logger log) throws IOException

Throws:

IOException

BinaryStoreWriterV2

public BinaryStoreWriterV2(String path, String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, DateFormat rollFormat, Map<String,Object> constantColumnValues, int applicationVersion, com.fishlib.io.logger.Logger log) throws IOException

Throws:

IOException

BinaryStoreWriterV2

public BinaryStoreWriterV2(String path, String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, DateFormat rollFormat, Map<String,Object> constantColumnValues, int applicationVersion, int targetWriteSize, com.fishlib.io.logger.Logger log) throws IOException

Throws:

IOException

BinaryStoreWriterV2

protected BinaryStoreWriterV2(String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, Map<String,Object> constantColumnValues, int applicationVersion, com.fishlib.io.logger.Logger log)

Note: this is for internal use, and does not call setChannelManager or init().

BinaryStoreWriterV2

protected BinaryStoreWriterV2(String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, Map<String,Object> constantColumnValues, int applicationVersion, int targetWriteSize, com.fishlib.io.logger.Logger log)

Method Details

computeHeaderBuffer

public ByteBuffer computeHeaderBuffer() throws IOException

Compute the header buffer for this writer and return it in a new buffer (direct). Callers of this method should cache the result if possible.

Returns:

a new buffer containing the binary header.

Throws:

IOException - pass through

makeCurrentDatePartitionedBinaryStoreWriter

public static BinaryStoreWriterV2 makeCurrentDatePartitionedBinaryStoreWriter(com.fishlib.io.logger.Logger log, String logDir, String namespace, String tablename, TableIdentifier.NamespaceSet namespaceSet, String internalPartition, String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, Map<String,Object> constantColumnValues, int applicationVersion, ZoneId zoneId) throws IOException

Make a BinaryStoreWriterV2 instance that changes the column partition value based on the current time, by changing the name of the rolling output file.

Parameters:

log - the Logger instance

logDir - the directory into which binary logs will be written

namespace - the namespace

tablename - the table name

namespaceSet - the NamespaceSet

internalPartition - the internal partition value

columnNames - the names of the columns being logged

columnTypes - the types of the columns being logged

typeMetadata - the type metadata

constantColumnValues - any constant column values

applicationVersion - the application logger version

zoneId - the time zone ZoneId

Throws:

IOException - from the initialization

makeSpecifiedPartitionBinaryStoreWriter

public static BinaryStoreWriterV2 makeSpecifiedPartitionBinaryStoreWriter(com.fishlib.io.logger.Logger log, String logDir, String namespace, String tablename, TableIdentifier.NamespaceSet namespaceSet, String internalPartition, String columnPartition, String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, Map<String,Object> constantColumnValues, int applicationVersion, ZoneId zoneId) throws IOException

Make a BinaryStoreWriterV2 with the specified values, including an unchanging column partition value.

Parameters:

log - the Logger instance

logDir - the directory in which the logs will be written

namespace - the namespace (used to construct the file name)

tablename - the table name (used to construct the file name)

namespaceSet - the namespaceSet (used to construct the file name)

internalPartition - the internal partition value (used to construct the file name)

columnPartition - the column partition value (used to construct the file name)

columnNames - the names of the columns being logged

columnTypes - the types of the columns being logged

typeMetadata - the type metadata

constantColumnValues - any constant column values

applicationVersion - the application logger version

zoneId - the time zone, used to generate the date-time stamp at the end of the file name

Returns:

the BinaryStoreWriterV2 instance

Throws:

IOException - from the initialization

makeMultiPartitionFileManagerBinaryStoreWriter

public static BinaryStoreWriterV2 makeMultiPartitionFileManagerBinaryStoreWriter(com.fishlib.io.logger.Logger log, String logDir, String namespace, String tablename, TableIdentifier.NamespaceSet namespaceSet, String internalPartition, String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata, Map<String,Object> constantColumnValues, int applicationVersion) throws IOException

Make a BinaryStoreWriterV2 instance that uses dynamic (per-row) column partition values.

Parameters:

log - the Logger instance

logDir - the directory into which binary logs will be written

namespace - the namespace

tablename - the table name

namespaceSet - the NamespaceSet

internalPartition - the internal partition value

columnNames - the names of the columns being logged

columnTypes - the types of the columns being logged

typeMetadata - the type metadata

constantColumnValues - any constant column values

applicationVersion - the application logger version

Throws:

IOException - from the initialization

init

protected void init() throws IOException

Throws:

IOException

getRowWriter

public BinaryRowV2 getRowWriter()

Description copied from interface: TableWriter

Get a writer for a Row entries. This is likely to be newly created, so callers should cache this value. In practice, TableWriter implementations generally cache the result of the first call to this method as a primary writer.

Returns:

a Row, likely newly created

createBinaryRow

protected BinaryRowV2 createBinaryRow(String[] columnNames, SupportedType[] columnTypes, Object[] typeMetadata)

getSetter

public RowSetter getSetter(String name)

Description copied from interface: TableWriter

Gets a setter for a column.

The implementation is likely to delegate to Row.getSetter(String) in a default Row instance.

Parameters:

name - column name

Returns:

setter for the column.

setFlags

public void setFlags(Row.Flags flags)

Description copied from interface: TableWriter

The implementation is likely to delegate to Row.setFlags(Flags) in a default Row instance.

writeRow

public void writeRow() throws IOException

Description copied from interface: TableWriter

Writes out a new row (values set using setters).

The implementation is likely to delegate to Row.writeRow() in a default Row instance.

Throws:

IOException - problem writing the row

setColumnPartitionValue

public void setColumnPartitionValue(String columnPartitionValue)

Description copied from interface: Row

For rows that are to be used with file managers that allow dynamic column partition selection, set the column partition value.

Parameters:

columnPartitionValue - the column partition value

getColumnPartitionValue

public String getColumnPartitionValue()

Description copied from interface: Row

For rows that are to be used with file managers that allow dynamic column partition selection, retrieve the column partition value.

Returns:

the previously-set column partition value

writeDeferredRow

public void writeDeferredRow(@NotNull WritableRowContainer<BinaryRowV2> row) throws IOException

Description copied from interface: TableWriter

Write the row to disk.

Implementations of this method may choose to buffer rows for more efficient use of the underlying storage. Users should invoke TableWriter.flushDeferredRows() to tell the implementation to write any buffered rows.

Parameters:

row - The row to write (or potentially buffer)

Throws:

IOException - if an error occurs during write.

flushDeferredRows

protected void flushDeferredRows(BinaryStoreWriterV2.DeferrableRowPartition part) throws IOException

Flush all of the deferred rows in the input collection to the underlying stream.

Parameters:

part - The collection of deferred rows for the partitionVlaue

Throws:

IOException - if an error occurs during writing. Rows may be partially written, however there are no guarantees that the rows were written completely.

flushDeferredRows

public void flushDeferredRows() throws IOException

Description copied from interface: TableWriter

Write any rows buffered by TableWriter.writeDeferredRow(WritableRowContainer) to storage.

Throws:

IOException - if an error occurs during write. Implementations that may write to more than one channel will write through to each channel and throw a MultiException wrapped in an IOException that contains the details on errors that occurred per channel.

writeMulti

protected void writeMulti(ByteBuffer[] buffs, int count, GatheringByteChannel channel) throws IOException

Perform a gathering write to the writer's channel. This method will not return until all buffers have been written or an error occurs.

Parameters:

buffs - The buffers to write

count - The number of buffers to write.

Throws:

IOException

getColumnTypes

public Class[] getColumnTypes()

Description copied from interface: TableWriter

Gets the column types for the table.

Returns:

column types for the table.

getColumnNames

public String[] getColumnNames()

Description copied from interface: TableWriter

Gets the column names for the table.

Returns:

column names for the table.

setRecordData

public void setRecordData(RecordData recordData)

getRecordSetter

public <T extends RecordData> Record.RecordSetter<T> getRecordSetter(Class<T> tClass)

Description copied from interface: TableWriter

Gets a typed record setter for an entry.

The implementation is likely to delegate to Record.getRecordSetter(Class) in a default Record instance.

Returns:

setter for the record.

getRecordSetter

public Record.RecordSetter<RecordData> getRecordSetter()

Description copied from interface: TableWriter

Gets a record setter for an entry.

The implementation is likely to delegate to Record.getRecordSetter() in a default Record instance.

Returns:

setter for the record.

writeRecord

public void writeRecord(WritableByteChannel channel) throws IOException

Description copied from interface: TableWriter

Write a record entry to the provided output channel. The implementor is responsible for providing the data buffer.

The TableWriter implementation is likely to delegate to Record.writeRecord(WritableByteChannel) in a default Record instance.

Throws:

IOException

getRecordWriter

public BinaryRecordV2 getRecordWriter()

Description copied from interface: TableWriter

Get new writer for Record entries. This is likely to be newly created, so callers should cache this value. In practice, implementers generally cache the result of the first call to this method as a primary writer.

Returns:

a Record, likely newly created

createBinaryRecord

protected BinaryRecordV2 createBinaryRecord()

Implementation point for creating a binary record writer (BinaryRecordV2). Derived classes can override to change the details or type.

Returns:

a new BinaryRecordV2 record writer