twitter / elephant-birdFile Size

Intro

File size measurements show the distribution of size of files.
Files are classified in four categories based on their size (lines of code): 1-100 (very small files), 101-200 (small files), 201-500 (medium size files), 501-1000 (long files), 1001+(very long files).
It is a good practice to keep files small. Long files may become "bloaters", code that have increased to such gargantuan proportions that they are hard to work with.

Learn more...

File Size Overall

There are 250 files with 16,140 lines of code.

0 very long files (0 lines of code)
0 long files (0 lines of code)
11 medium size files (3,344 lines of codeclsfd_ftr_w_mp_ins)
40 small files (5,522 lines of code)
199 very small files (7,274 lines of code)

Legend:

1001+

501-1000

201-500

101-200

1-100

explore: grouped by folders | grouped by size | sunburst | 3D view

File Size per Extension

1001+

501-1000

201-500

101-200

1-100

File Size per Logical Decomposition

primary

1001+

501-1000

201-500

101-200

1-100

Longest Files (Top 50)

File	# lines	# units
ThriftToDynamicProto.java in core/src/main/java/com/twitter/elephantbird/util	408	25
Protobufs.java in core/src/main/java/com/twitter/elephantbird/util	356	31
ThriftToPig.java in pig/src/main/java/com/twitter/elephantbird/pig/util	349	21
SplitUtil.java in core/src/main/java/com/twitter/elephantbird/util	349	27
VectorWritableConverter.java in mahout/src/main/java/com/twitter/elephantbird/pig/mahout	333	21
DeprecatedInputFormatWrapper.java in core/src/main/java/com/twitter/elephantbird/mapred/input	308	30
ProtobufToPig.java in pig/src/main/java/com/twitter/elephantbird/pig/util	284	17
ThriftUtils.java in core/src/main/java/com/twitter/elephantbird/util	267	11
HadoopCompat.java in hadoop-compat/src/main/java/com/twitter/elephantbird/util	249	21
TStructDescriptor.java in core/src/main/java/com/twitter/elephantbird/thrift	236	31
MapReduceInputFormatWrapper.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input	205	20
Invoker.java in pig/src/main/java/com/twitter/elephantbird/pig/piggybank	200	8
ThriftProtocolWrapper.java in core/src/main/java/com/twitter/elephantbird/thrift	198	44
LuceneIndexInputFormat.java in lucene/src/main/java/com/twitter/elephantbird/mapreduce/input	197	22
SequenceFileStorage.java in pig/src/main/java/com/twitter/elephantbird/pig/store	186	15
PigToThrift.java in pig/src/main/java/com/twitter/elephantbird/pig/util	184	11
PigToProtobuf.java in pig/src/main/java/com/twitter/elephantbird/pig/util	177	11
MultiInputFormat.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input	163	8
ProtobufStructObjectInspector.java in hive/src/main/java/com/twitter/elephantbird/hive/serde	163	15
RCFileThriftOutputFormat.java in rcfile/src/main/java/com/twitter/elephantbird/mapreduce/output	162	9
SequenceFileLoader.java in pig/src/main/java/com/twitter/elephantbird/pig/load	161	13
LzoInputFormat.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input	160	10
RCFilePigStorage.java in rcfile/src/main/java/com/twitter/elephantbird/pig/store	154	10
JsonLoader.java in pig/src/main/java/com/twitter/elephantbird/pig/load	153	9
HadoopUtils.java in core/src/main/java/com/twitter/elephantbird/util	146	10
RCFileThriftInputFormat.java in rcfile/src/main/java/com/twitter/elephantbird/mapreduce/input	139	9
LuceneIndexOutputFormat.java in lucene/src/main/java/com/twitter/elephantbird/mapreduce/output	139	10
LuceneHdfsDirectory.java in lucene/src/main/java/com/twitter/elephantbird/mapreduce/input	136	17
RCFileProtobufInputFormat.java in rcfile/src/main/java/com/twitter/elephantbird/mapreduce/input	134	9
Inflection.java in core/src/main/java/com/twitter/elephantbird/util	132	13
Base64Codec.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input	132	1
ExecuteOnClusterTool.java in core/src/main/java/com/twitter/elephantbird/util	130	14
BinaryBlockReader.java in core/src/main/java/com/twitter/elephantbird/mapreduce/io	128	13
LzoRecordReader.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input	127	9
CompositeRecordReader.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input/combine	126	10
BinaryWritable.java in core/src/main/java/com/twitter/elephantbird/mapreduce/io	126	13
CompositeInputSplit.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input/combine	124	13
SequenceFileConfig.java in pig/src/main/java/com/twitter/elephantbird/pig/util	120	8
LzoBinaryB64LineRecordReader.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input	118	9
LzoBaseLoadFunc.java in pig/src/main/java/com/twitter/elephantbird/pig/load	117	15
AbstractLazyTuple.java in pig/src/main/java/com/twitter/elephantbird/pig/util	115	17
LuceneIndexRecordReader.java in lucene/src/main/java/com/twitter/elephantbird/mapreduce/input	114	9
LzoBinaryBlockRecordReader.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input	112	9
Strings.java in core/src/main/java/com/twitter/elephantbird/util	110	15
RCFileUtil.java in rcfile/src/main/java/com/twitter/elephantbird/util	108	3
FilterLoadFunc.java in pig/src/main/java/com/twitter/elephantbird/pig/load	107	17
RawSequenceFileRecordReader.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input	107	6
LocationAsTuple.java in pig/src/main/java/com/twitter/elephantbird/pig/load	105	12
LuceneIndexLoader.java in pig-lucene/src/main/java/com/twitter/elephantbird/pig/load	105	7
WritableLoadCaster.java in pig/src/main/java/com/twitter/elephantbird/pig/util	104	25

Files With Most Units (Top 50)

File	# lines	# units
ThriftProtocolWrapper.java in core/src/main/java/com/twitter/elephantbird/thrift	198	44
Protobufs.java in core/src/main/java/com/twitter/elephantbird/util	356	31
TStructDescriptor.java in core/src/main/java/com/twitter/elephantbird/thrift	236	31
DeprecatedInputFormatWrapper.java in core/src/main/java/com/twitter/elephantbird/mapred/input	308	30
SplitUtil.java in core/src/main/java/com/twitter/elephantbird/util	349	27
WritableLoadCaster.java in pig/src/main/java/com/twitter/elephantbird/pig/util	104	25
ThriftToDynamicProto.java in core/src/main/java/com/twitter/elephantbird/util	408	25
LuceneIndexInputFormat.java in lucene/src/main/java/com/twitter/elephantbird/mapreduce/input	197	22
ThriftToPig.java in pig/src/main/java/com/twitter/elephantbird/pig/util	349	21
HadoopCompat.java in hadoop-compat/src/main/java/com/twitter/elephantbird/util	249	21
VectorWritableConverter.java in mahout/src/main/java/com/twitter/elephantbird/pig/mahout	333	21
MapReduceInputFormatWrapper.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input	205	20
TextConverter.java in pig/src/main/java/com/twitter/elephantbird/pig/util	81	17
ProtobufToPig.java in pig/src/main/java/com/twitter/elephantbird/pig/util	284	17
AbstractLazyTuple.java in pig/src/main/java/com/twitter/elephantbird/pig/util	115	17
FilterLoadFunc.java in pig/src/main/java/com/twitter/elephantbird/pig/load	107	17
LuceneHdfsDirectory.java in lucene/src/main/java/com/twitter/elephantbird/mapreduce/input	136	17
LzoBaseLoadFunc.java in pig/src/main/java/com/twitter/elephantbird/pig/load	117	15
SequenceFileStorage.java in pig/src/main/java/com/twitter/elephantbird/pig/store	186	15
Strings.java in core/src/main/java/com/twitter/elephantbird/util	110	15
ProtobufStructObjectInspector.java in hive/src/main/java/com/twitter/elephantbird/hive/serde	163	15
IntWritableConverter.java in pig/src/main/java/com/twitter/elephantbird/pig/util	74	14
LongWritableConverter.java in pig/src/main/java/com/twitter/elephantbird/pig/util	74	14
ExecuteOnClusterTool.java in core/src/main/java/com/twitter/elephantbird/util	130	14
WritableStoreCaster.java in pig/src/main/java/com/twitter/elephantbird/pig/util	103	13
SequenceFileLoader.java in pig/src/main/java/com/twitter/elephantbird/pig/load	161	13
Inflection.java in core/src/main/java/com/twitter/elephantbird/util	132	13
CompositeInputSplit.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input/combine	124	13
BinaryBlockReader.java in core/src/main/java/com/twitter/elephantbird/mapreduce/io	128	13
BinaryWritable.java in core/src/main/java/com/twitter/elephantbird/mapreduce/io	126	13
LzoTextDelimited.java in cascading3/src/main/java/com/twitter/elephantbird/cascading3/scheme	61	13
LzoTextDelimited.java in cascading2/src/main/java/com/twitter/elephantbird/cascading2/scheme	58	13
EBTypes.java in crunch/src/main/java/com/twitter/elephantbird/crunch	89	13
LocationAsTuple.java in pig/src/main/java/com/twitter/elephantbird/pig/load	105	12
PigToProtobuf.java in pig/src/main/java/com/twitter/elephantbird/pig/util	177	11
PigToThrift.java in pig/src/main/java/com/twitter/elephantbird/pig/util	184	11
PathFilters.java in core/src/main/java/com/twitter/elephantbird/util	76	11
ThriftUtils.java in core/src/main/java/com/twitter/elephantbird/util	267	11
DelegateCombineFileInputFormat.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input/combine	94	11
RCFilePigStorage.java in rcfile/src/main/java/com/twitter/elephantbird/pig/store	154	10
ColumnarMetadata.java in rcfile/src/main/java/com/twitter/elephantbird/util	90	10
LuceneIndexOutputFormat.java in lucene/src/main/java/com/twitter/elephantbird/mapreduce/output	139	10
HadoopUtils.java in core/src/main/java/com/twitter/elephantbird/util	146	10
HdfsUtils.java in core/src/main/java/com/twitter/elephantbird/util	96	10
CompositeRecordReader.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input/combine	126	10
LzoInputFormat.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input	160	10
LzoW3CLogRecordReader.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input	99	10
RCFileProtobufInputFormat.java in rcfile/src/main/java/com/twitter/elephantbird/mapreduce/input	134	9
RCFileThriftInputFormat.java in rcfile/src/main/java/com/twitter/elephantbird/mapreduce/input	139	9
RCFileThriftOutputFormat.java in rcfile/src/main/java/com/twitter/elephantbird/mapreduce/output	162	9

Files With Long Lines (Top 34)

There are 34 files with lines longer than 120 characters. In total, there are 67 long lines.

File	# lines	# units	# long lines
ProtobufToPig.java in pig/src/main/java/com/twitter/elephantbird/pig/util	284	17	14
ThriftToPig.java in pig/src/main/java/com/twitter/elephantbird/pig/util	349	21	3
PigUtil.java in pig/src/main/java/com/twitter/elephantbird/pig/util	80	8	3
ProtobufPigLoader.java in pig/src/main/java/com/twitter/elephantbird/pig/load	54	4	2
InvokeForLong.java in pig/src/main/java/com/twitter/elephantbird/pig/piggybank	15	4	2
InvokeForString.java in pig/src/main/java/com/twitter/elephantbird/pig/piggybank	15	4	2
InvokeForDouble.java in pig/src/main/java/com/twitter/elephantbird/pig/piggybank	15	4	2
InvokeForFloat.java in pig/src/main/java/com/twitter/elephantbird/pig/piggybank	15	4	2
InvokeForInt.java in pig/src/main/java/com/twitter/elephantbird/pig/piggybank	15	4	2
LzoProtobufBlockPigStorage.java in pig/src/main/java/com/twitter/elephantbird/pig/store	49	4	2
LzoProtobufB64LinePigStorage.java in pig/src/main/java/com/twitter/elephantbird/pig/store	49	4	2
Protobufs.java in core/src/main/java/com/twitter/elephantbird/util	356	31	2
LzoThriftScheme.java in cascading3/src/main/java/com/twitter/elephantbird/cascading3/scheme	38	4	2
LzoTextLine.java in cascading3/src/main/java/com/twitter/elephantbird/cascading3/scheme	42	8	2
LzoTextDelimited.java in cascading3/src/main/java/com/twitter/elephantbird/cascading3/scheme	61	13	2
LzoProtobufScheme.java in cascading3/src/main/java/com/twitter/elephantbird/cascading3/scheme	38	4	2
LzoTextLine.java in cascading2/src/main/java/com/twitter/elephantbird/cascading2/scheme	39	8	2
LzoTextDelimited.java in cascading2/src/main/java/com/twitter/elephantbird/cascading2/scheme	58	13	2
CombinedWritableSequenceFile.java in cascading2/src/main/java/com/twitter/elephantbird/cascading2/scheme	22	3	2
PigToProtobuf.java in pig/src/main/java/com/twitter/elephantbird/pig/util	177	11	1
JsonStringToMap.java in pig/src/main/java/com/twitter/elephantbird/pig/piggybank	63	3	1
DeprecatedInputFormatWrapper.java in core/src/main/java/com/twitter/elephantbird/mapred/input	308	30	1
DeprecatedOutputFormatWrapper.java in core/src/main/java/com/twitter/elephantbird/mapred/output	95	9	1
CompositeRecordReader.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input/combine	126	10	1
CompositeInputSplit.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input/combine	124	13	1
LzoProtobufBlockRecordReader.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input	14	1	1
LzoProtobufB64LineRecordReader.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input	14	1	1
LzoThriftB64LineRecordReader.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input	14	1	1
LzoThriftBlockRecordReader.java in core/src/main/java/com/twitter/elephantbird/mapreduce/input	14	1	1
BinaryBlockWriter.java in core/src/main/java/com/twitter/elephantbird/mapreduce/io	62	7	1
LzoProtobufB64LineOutputFormat.java in core/src/main/java/com/twitter/elephantbird/mapreduce/output	35	4	1
LzoThriftB64LineOutputFormat.java in core/src/main/java/com/twitter/elephantbird/mapreduce/output	33	3	1
CombinedSequenceFile.java in cascading3/src/main/java/com/twitter/elephantbird/cascading3/scheme	52	3	1
CombinedSequenceFile.java in cascading2/src/main/java/com/twitter/elephantbird/cascading2/scheme	54	4	1