spotify / beam
Duplication

Places in code with 6 or more lines that are exactly the same.

Intro
Learn more...
Duplication Overall
system11% (45,373 lines)
dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...
Duplication per Extension
java12% (29,239 lines)
py8% (6,166 lines)
go7% (3,094 lines)
yaml30% (2,184 lines)
html45% (1,249 lines)
sql22% (990 lines)
scss10% (968 lines)
proto12% (585 lines)
groovy17% (446 lines)
kt37% (293 lines)
js2% (58 lines)
xml7% (47 lines)
pyx7% (26 lines)
sass3% (16 lines)
in30% (6 lines)
toml6% (6 lines)
Duplication per Component (primary)
sdks10% (26,819 lines)
runners13% (12,499 lines)
website23% (4,095 lines)
learning20% (1,036 lines)
model11% (478 lines)
buildSrc19% (374 lines)
release7% (72 lines)
ownership0% (0 lines)
ROOT0% (0 lines)

Duplication Between Components (50+ lines)

G runners runners sdks sdks runners--sdks 2838 model model model--runners 255 model--sdks 269

Download: SVG DOT (open online Graphviz editor)

Open 2D force graph... Open 3D force graph...

Show more details on duplication between components...
Longest Duplicates
The list of 50 longest duplicates.
See data for all 32,509 duplicates...
Size#FoldersFilesLinesCode
190 x 2 runners/google-cloud-dat...runners/dataflow/worker
runners/google-cloud-dat...runners/dataflow/worker
553:796 (36%)
160:403 (80%)
view
178 x 2 sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
48:342 (100%)
48:342 (100%)
view
127 x 2 sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
35:220 (100%)
35:220 (100%)
view
118 x 2 runners/google-cloud-dat...rs/dataflow/worker/util
sdks/java/harness/src/ma.../beam/fn/harness/status
334:565 (38%)
334:565 (39%)
view
96 x 2 sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
33:162 (100%)
33:162 (100%)
view
89 x 2 sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
45:178 (100%)
45:178 (100%)
view
87 x 2 runners/spark/2/src/main...cturedstreaming/metrics
runners/spark/src/main/j...m/runners/spark/metrics
50:180 (100%)
50:180 (100%)
view
77 x 2 website/www/site/layouts/commercial
website/www/site/layouts/open-source
17:111 (83%)
17:111 (83%)
view
76 x 2 sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
37:136 (89%)
37:136 (89%)
view
64 x 2 sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
29:148 (100%)
29:148 (100%)
view
62 x 2 runners/spark/2/src/main...slation/batch/functions
runners/spark/src/main/j...nners/spark/translation
85:170 (72%)
78:163 (74%)
view
53 x 2 website/www/site/data
website/www/site/data
525:577 (3%)
769:821 (3%)
view
47 x 2 sdks/java/extensions/sql...extensions/sql/impl/rel
sdks/java/extensions/sql...ions/sql/zetasql/unnest
74:145 (55%)
81:152 (66%)
view
46 x 2 buildSrc/src/main/groovy/org/apache/beam/gradle
buildSrc/src/main/groovy/org/apache/beam/gradle
1322:1380 (3%)
222:280 (23%)
view
46 x 2 website/www/site/data
website/www/site/data
725:770 (2%)
819:864 (2%)
view
45 x 2 runners/google-cloud-dat...rker/util/common/worker
sdks/java/harness/src/ma.../apache/beam/fn/harness
213:283 (15%)
367:436 (14%)
view
44 x 2 sdks/java/io/amazon-web-...ache/beam/sdk/io/aws/s3
sdks/java/io/amazon-web-...che/beam/sdk/io/aws2/s3
134:203 (41%)
134:203 (41%)
view
44 x 2 runners/google-cloud-dat...runners/dataflow/worker
sdks/java/io/google-clou...beam/sdk/io/gcp/spanner
77:166 (13%)
79:166 (12%)
view
42 x 2 sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
27:99 (100%)
27:99 (100%)
view
42 x 2 runners/google-cloud-dat...s/dataflow/worker/graph
runners/google-cloud-dat...s/dataflow/worker/graph
524:574 (11%)
473:523 (10%)
view
42 x 2 sdks/java/core/src/main/.../beam/sdk/schemas/utils
sdks/java/core/src/main/.../beam/sdk/schemas/utils
258:301 (14%)
175:219 (11%)
view
42 x 2 runners/spark/2/src/main...aming/translation/utils
runners/spark/src/main/j...beam/runners/spark/util
32:104 (100%)
32:104 (100%)
view
42 x 2 runners/java-fn-executio...unners/fnexecution/wire
sdks/java/extensions/pro...sdk/extensions/protobuf
38:122 (97%)
37:121 (100%)
view
41 x 2 sdks/python/apache_beam/io/gcp
sdks/python/apache_beam/io/gcp
381:423 (11%)
433:475 (11%)
view
41 x 2 sdks/python/apache_beam/io/gcp
sdks/python/apache_beam/io/gcp
381:423 (11%)
485:527 (11%)
view
41 x 2 sdks/python/apache_beam/io/gcp
sdks/python/apache_beam/io/gcp
381:423 (11%)
537:579 (11%)
view
41 x 2 sdks/python/apache_beam/io/gcp
sdks/python/apache_beam/io/gcp
433:475 (11%)
485:527 (11%)
view
41 x 2 sdks/python/apache_beam/io/gcp
sdks/python/apache_beam/io/gcp
433:475 (11%)
537:579 (11%)
view
41 x 2 website/www/site/data
website/www/site/data
584:624 (2%)
828:868 (2%)
view
41 x 2 runners/google-cloud-dat...runners/dataflow/worker
sdks/java/io/google-clou...beam/sdk/io/gcp/spanner
446:510 (12%)
539:603 (11%)
view
41 x 2 sdks/python/apache_beam/io/gcp
sdks/python/apache_beam/io/gcp
485:527 (11%)
537:579 (11%)
view
40 x 2 website/www/site/data
website/www/site/data
622:661 (2%)
669:708 (2%)
view
40 x 2 runners/google-cloud-dat...s/dataflow/worker/graph
runners/google-cloud-dat...s/dataflow/worker/graph
393:449 (11%)
358:413 (10%)
view
40 x 2 learning/katas/java/Wind...g/addingtimestamp/pardo
learning/katas/java/Wind...imestamp/withtimestamps
25:89 (100%)
25:89 (100%)
view
39 x 2 sdks/python/apache_beam/io/gcp
sdks/python/apache_beam/io/gcp
537:575 (10%)
589:627 (10%)
view
39 x 2 sdks/python/apache_beam/io/gcp
sdks/python/apache_beam/io/gcp
381:419 (10%)
589:627 (10%)
view
39 x 2 sdks/python/apache_beam/io/gcp
sdks/python/apache_beam/io/gcp
433:471 (10%)
589:627 (10%)
view
39 x 2 runners/google-cloud-dat...rker/util/common/worker
sdks/java/harness/src/ma.../apache/beam/fn/harness
438:488 (13%)
479:529 (12%)
view
39 x 2 website/www/site/data
website/www/site/data
481:519 (2%)
772:810 (2%)
view
39 x 2 website/www/site/data
website/www/site/data
481:519 (2%)
528:566 (2%)
view
39 x 2 sdks/python/apache_beam/io/gcp
sdks/python/apache_beam/io/gcp
485:523 (10%)
589:627 (10%)
view
38 x 2 website/www/site/data
website/www/site/data
486:523 (2%)
580:617 (2%)
view
38 x 2 website/www/site/data
website/www/site/data
677:714 (2%)
827:864 (2%)
view
38 x 2 website/www/site/data
website/www/site/data
677:714 (2%)
733:770 (2%)
view
38 x 2 sdks/python/apache_beam/transforms
sdks/python/apache_beam/transforms
155:200 (6%)
677:722 (6%)
view
37 x 2 runners/spark/2/src/main...aming/translation/utils
runners/spark/src/main/j...beam/runners/spark/util
33:91 (100%)
33:91 (100%)
view
37 x 2 website/www/site/data
website/www/site/data
584:620 (2%)
678:714 (2%)
view
37 x 2 website/www/site/data
website/www/site/data
584:620 (2%)
734:770 (2%)
view
37 x 2 website/www/site/data
website/www/site/data
584:620 (2%)
922:958 (2%)
view
37 x 2 website/www/site/data
website/www/site/data
1048:1084 (2%)
1087:1123 (2%)
view
Duplicated Units
The list of top 50 duplicated units.
See data for all 160 unit duplicates...
Size#FoldersFilesLinesCode
44 x 2 sdks/java/harness/src/ma.../beam/fn/harness/status
runners/google-cloud-dat...rs/dataflow/worker/util
475:534 
475:534 
view
42 x 2 sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
130:181 
130:181 
view
39 x 2 runners/google-cloud-dat...runners/dataflow/worker
runners/google-cloud-dat...runners/dataflow/worker
553:594 
160:201 
view
39 x 2 runners/google-cloud-dat...runners/dataflow/worker
runners/google-cloud-dat...runners/dataflow/worker
677:722 
284:329 
view
36 x 2 sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
171:212 
171:212 
view
30 x 2 sdks/java/core/src/main/.../beam/sdk/schemas/utils
sdks/java/core/src/main/.../beam/sdk/schemas/utils
190:222 
273:304 
view
30 x 2 sdks/java/harness/src/ma.../apache/beam/fn/harness
runners/google-cloud-dat...rker/util/common/worker
328:364 
359:395 
view
30 x 2 runners/google-cloud-dat...runners/dataflow/worker
runners/google-cloud-dat...runners/dataflow/worker
637:671 
244:278 
view
27 x 2 sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
79:108 
79:108 
view
25 x 2 sdks/java/harness/src/ma.../apache/beam/fn/harness
runners/google-cloud-dat...rker/util/common/worker
380:408 
226:254 
view
23 x 2 sdks/java/harness/src/ma.../beam/fn/harness/status
runners/google-cloud-dat...rs/dataflow/worker/util
537:562 
537:562 
view
23 x 2 runners/google-cloud-dat...runners/dataflow/worker
runners/google-cloud-dat...runners/dataflow/worker
733:759 
340:366 
view
21 x 2 sdks/java/io/google-clou...beam/sdk/io/gcp/spanner
runners/google-cloud-dat...runners/dataflow/worker
302:326 
257:281 
view
21 x 2 runners/spark/2/src/main...slation/batch/functions
runners/spark/src/main/j...nners/spark/translation
122:144 
115:137 
view
20 x 2 sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
109:131 
109:131 
view
18 x 2 sdks/java/testing/tpcds/...g/apache/beam/sdk/tpcds
sdks/java/testing/tpcds/...g/apache/beam/sdk/tpcds
118:139 
129:149 
view
17 x 2 sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
57:75 
57:75 
view
17 x 2 sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
69:87 
69:87 
view
17 x 2 sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
94:114 
94:114 
view
17 x 2 runners/google-cloud-dat...runners/dataflow/worker
runners/google-cloud-dat...runners/dataflow/worker
596:616 
203:223 
view
17 x 2 runners/spark/2/src/main...cturedstreaming/metrics
runners/spark/src/main/j...m/runners/spark/metrics
98:123 
98:123 
view
17 x 2 runners/spark/2/src/main...edstreaming/translation
runners/spark/src/main/j...ache/beam/runners/spark
35:54 
35:54 
view
16 x 2 sdks/go/pkg/beam/io/textio
sdks/go/pkg/beam/io/avroio
61:80 
57:76 
view
16 x 2 sdks/java/core/src/main/...dk/transforms/windowing
runners/core-java/src/ma...m/runners/core/triggers
155:175 
354:374 
view
16 x 2 sdks/java/io/amazon-web-...che/beam/sdk/io/aws2/s3
sdks/java/io/amazon-web-...ache/beam/sdk/io/aws/s3
120:141 
114:135 
view
16 x 2 sdks/java/harness/src/ma.../beam/fn/harness/status
runners/google-cloud-dat...rs/dataflow/worker/util
254:272 
254:272 
view
16 x 2 sdks/java/harness/src/ma.../beam/fn/harness/status
runners/google-cloud-dat...rs/dataflow/worker/util
276:294 
276:294 
view
16 x 2 runners/spark/2/src/main...cturedstreaming/metrics
runners/spark/src/main/j...m/runners/spark/metrics
67:88 
81:102 
view
15 x 2 sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
76:93 
76:93 
view
15 x 2 sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
31:47 
31:47 
view
15 x 2 sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
215:232 
215:232 
view
15 x 2 learning/katas/java/Wind...imestamp/withtimestamps
learning/katas/java/Wind...g/addingtimestamp/pardo
32:52 
33:53 
view
14 x 2 sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
122:141 
122:141 
view
14 x 2 learning/katas/kotlin/Wi...imestamp/withtimestamps
learning/katas/kotlin/Wi...g/addingtimestamp/pardo
30:49 
31:50 
view
14 x 2 runners/java-fn-executio...ers/fnexecution/control
runners/google-cloud-dat...ners/dataflow/worker/fn
116:131 
104:119 
view
14 x 2 runners/samza/src/main/j...m/runners/samza/runtime
runners/samza/src/main/j...m/runners/samza/runtime
222:239 
199:216 
view
13 x 2 sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
95:109 
95:109 
view
13 x 2 sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
307:322 
307:322 
view
13 x 2 sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
324:338 
324:338 
view
13 x 2 sdks/java/harness/src/ma.../beam/fn/harness/status
runners/google-cloud-dat...rs/dataflow/worker/util
620:634 
625:639 
view
13 x 2 runners/core-java/src/ma...pache/beam/runners/core
runners/google-cloud-dat...runners/dataflow/worker
97:112 
79:94 
view
13 x 2 runners/samza/src/main/j...nners/samza/translation
runners/samza/src/main/j...nners/samza/translation
95:110 
79:94 
view
13 x 2 runners/google-cloud-dat...runners/dataflow/worker
runners/google-cloud-dat...runners/dataflow/worker
611:625 
408:422 
view
13 x 2 runners/google-cloud-dat...runners/dataflow/worker
runners/google-cloud-dat...runners/dataflow/worker
618:633 
225:240 
view
12 x 3 sdks/java/core/src/main/...rg/apache/beam/sdk/util
sdks/java/core/src/main/...rg/apache/beam/sdk/util
sdks/java/core/src/main/...rg/apache/beam/sdk/util
115:128 
130:143 
165:178 
view
12 x 2 sdks/java/core/src/main/.../beam/sdk/schemas/utils
sdks/java/core/src/main/.../beam/sdk/schemas/utils
175:188 
258:271 
view
12 x 2 sdks/java/core/src/main/.../beam/sdk/schemas/utils
sdks/java/core/src/main/.../beam/sdk/schemas/utils
332:346 
199:212 
view
12 x 2 sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
108:123 
108:123 
view
12 x 2 sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
103:118 
103:118 
view
12 x 2 sdks/java/io/kinesis/src...che/beam/sdk/io/kinesis
sdks/java/io/amazon-web-...eam/sdk/io/aws2/kinesis
183:199 
183:199 
view