twitter / the-algorithm
Duplication

Places in code with 6 or more lines that are exactly the same.

Intro
Learn more...
Duplication Overall
system11% (37,548 lines)
dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...
Duplication per Extension
scala12% (28,190 lines)
java5% (3,532 lines)
cpp22% (1,652 lines)
xml79% (998 lines)
workflow73% (962 lines)
py5% (808 lines)
rs16% (566 lines)
proto3% (307 lines)
thrift4% (264 lines)
sql21% (230 lines)
h2% (39 lines)
Duplication per Component (primary)
src10% (10,855 lines)
visibilitylib15% (3,486 lines)
cr-mixer20% (3,222 lines)
product-mixer11% (2,940 lines)
home-mixer13% (2,443 lines)
unified_user_actions33% (2,225 lines)
twml12% (2,129 lines)
pushservice7% (2,092 lines)
ann10% (1,582 lines)
follow-recommendations-service8% (1,245 lines)
timelineranker11% (913 lines)
tweetypie2% (905 lines)
navi7% (873 lines)
representation-manager34% (673 lines)
science90% (416 lines)
user-signal-service25% (395 lines)
recos-injector12% (237 lines)
topic-social-proof11% (222 lines)
representation-scorer15% (204 lines)
simclusters-ann21% (201 lines)
timelines4% (150 lines)
graph-feature-service5% (74 lines)
trust_and_safety_models3% (66 lines)

Duplication Between Components (50+ lines)

G home-mixer home-mixer timelineranker timelineranker home-mixer--timelineranker 536 product-mixer product-mixer home-mixer--product-mixer 437 src src home-mixer--src 58 product-mixer--timelineranker 115 representation-manager representation-manager representation-scorer representation-scorer representation-manager--representation-scorer 216 simclusters-ann simclusters-ann representation-manager--simclusters-ann 168 topic-social-proof topic-social-proof representation-scorer--topic-social-proof 100 representation-scorer--simclusters-ann 168 user-signal-service user-signal-service topic-social-proof--user-signal-service 184 simclusters-ann--src 52 cr-mixer cr-mixer cr-mixer--home-mixer 112 cr-mixer--product-mixer 66 cr-mixer--representation-manager 134 cr-mixer--representation-scorer 134 cr-mixer--topic-social-proof 164 cr-mixer--simclusters-ann 148 follow-recommendations-service follow-recommendations-service cr-mixer--follow-recommendations-service 122 follow-recommendations-service--product-mixer 80 follow-recommendations-service--representation-manager 68 follow-recommendations-service--representation-scorer 68 follow-recommendations-service--simclusters-ann 80 twml twml src--twml 76

Download: SVG DOT (open online Graphviz editor)

Open 2D force graph... Open 3D force graph...

Show more details on duplication between components...
Longest Duplicates
The list of 50 longest duplicates.
See data for all 18,092 duplicates...
Size#FoldersFilesLinesCode
86 x 2 topic-social-proof/server/src/main/resources
user-signal-service/server/src/main/resources
1:151 (100%)
1:151 (100%)
view
84 x 2 representation-manager/server/src/main/resources
representation-scorer/server/src/main/resources
1:162 (100%)
1:162 (100%)
view
78 x 2 representation-scorer/server/src/main/resources
simclusters-ann/server/src/main/resources
1:149 (92%)
1:149 (91%)
view
78 x 2 representation-manager/server/src/main/resources
simclusters-ann/server/src/main/resources
1:149 (92%)
1:149 (91%)
view
76 x 2 src/scala/com/twitter/recos/user_tweet_graph
src/scala/com/twitter/recos/user_video_graph
130:217 (41%)
132:220 (41%)
view
73 x 2 cr-mixer/server/src/main...mixer/similarity_engine
cr-mixer/server/src/main...mixer/similarity_engine
157:239 (14%)
234:313 (9%)
view
70 x 2 home-mixer/server/src/ma.../util/tweetypie/content
timelineranker/server/sr...ter/timelineranker/util
38:112 (31%)
19:93 (33%)
view
69 x 2 science/search/ingester/config
science/search/ingester/config
46:147 (49%)
45:145 (50%)
view
66 x 2 src/scala/com/twitter/recos/user_tweet_graph
src/scala/com/twitter/recos/user_user_graph
109:189 (36%)
79:159 (40%)
view
64 x 2 follow-recommendations-s...ature_hydration/sources
follow-recommendations-s...ature_hydration/sources
74:147 (56%)
74:147 (56%)
view
64 x 2 follow-recommendations-s...ature_hydration/sources
follow-recommendations-s...ature_hydration/sources
74:147 (55%)
74:147 (56%)
view
64 x 2 follow-recommendations-s...ature_hydration/sources
follow-recommendations-s...ature_hydration/sources
74:147 (56%)
74:147 (55%)
view
59 x 2 src/scala/com/twitter/re...user_tweet_entity_graph
src/scala/com/twitter/recos/user_tweet_graph
75:134 (33%)
130:189 (32%)
view
59 x 2 src/scala/com/twitter/re...user_tweet_entity_graph
src/scala/com/twitter/recos/user_video_graph
75:134 (33%)
132:191 (32%)
view
59 x 2 src/scala/com/twitter/re...user_tweet_entity_graph
src/scala/com/twitter/recos/user_user_graph
75:134 (33%)
100:159 (36%)
view
59 x 2 src/scala/com/twitter/recos/user_user_graph
src/scala/com/twitter/recos/user_video_graph
100:159 (36%)
132:191 (32%)
view
52 x 2 cr-mixer/server/src/main...mixer/similarity_engine
cr-mixer/server/src/main...mixer/similarity_engine
718:772 (6%)
854:909 (6%)
view
47 x 2 src/scala/com/twitter/recos/user_tweet_graph
src/scala/com/twitter/recos/user_video_graph
57:125 (25%)
57:125 (25%)
view
44 x 2 science/search/ingester/config
science/search/ingester/config
124:187 (26%)
83:146 (31%)
view
43 x 2 src/scala/com/twitter/si...ding/embedding/producer
src/scala/com/twitter/si...ding/embedding/producer
237:286 (17%)
309:358 (17%)
view
43 x 2 src/scala/com/twitter/si...ding/embedding/producer
src/scala/com/twitter/si...ding/embedding/producer
149:198 (22%)
219:268 (22%)
view
43 x 2 science/search/ingester/config
science/search/ingester/config
85:147 (30%)
124:186 (25%)
view
38 x 2 src/scala/com/twitter/recos/hose/common
src/scala/com/twitter/recos/hose/common
83:138 (29%)
89:144 (27%)
view
36 x 2 visibilitylib/src/main/s...witter/visibility/rules
visibilitylib/src/main/s...witter/visibility/rules
2120:2155 (1%)
2255:2290 (1%)
view
35 x 2 pushservice/src/main/sca...cate/ntab_caret_fatigue
pushservice/src/main/sca...cate/ntab_caret_fatigue
508:544 (4%)
688:724 (4%)
view
35 x 2 src/thrift/com/twitter/recos/user_tweet_graph
src/thrift/com/twitter/recos/user_video_graph
114:156 (26%)
11:53 (85%)
view
34 x 2 src/scala/com/twitter/re...ph/relatedTweetHandlers
src/scala/com/twitter/re...ph/relatedTweetHandlers
23:67 (62%)
22:66 (64%)
view
33 x 2 cr-mixer/server/src/main.../twitter/cr_mixer/param
cr-mixer/server/src/main.../twitter/cr_mixer/param
102:139 (33%)
95:132 (34%)
view
32 x 2 src/scala/com/twitter/si...ding/embedding/producer
src/scala/com/twitter/si...ding/embedding/producer
161:198 (17%)
249:286 (13%)
view
32 x 2 src/scala/com/twitter/si...ding/embedding/producer
src/scala/com/twitter/si...ding/embedding/producer
161:198 (17%)
321:358 (13%)
view
32 x 2 twml/libtwml/src/lib
twml/libtwml/src/lib
15:46 (65%)
15:46 (58%)
view
32 x 2 visibilitylib/src/main/s...witter/visibility/rules
visibilitylib/src/main/s...witter/visibility/rules
1881:1912 (<1%)
1989:2020 (<1%)
view
32 x 2 src/scala/com/twitter/si...ding/embedding/producer
src/scala/com/twitter/si...ding/embedding/producer
231:268 (17%)
117:154 (33%)
view
32 x 2 src/scala/com/twitter/si...ding/embedding/producer
src/scala/com/twitter/si...ding/embedding/producer
117:154 (33%)
321:358 (13%)
view
32 x 2 src/scala/com/twitter/si...ding/embedding/producer
src/scala/com/twitter/si...ding/embedding/producer
117:154 (33%)
249:286 (13%)
view
32 x 2 twml/libtwml/src/ops
twml/libtwml/src/ops
45:82 (14%)
120:157 (14%)
view
32 x 2 src/scala/com/twitter/si...s_v2/scalding/embedding
src/scala/com/twitter/si...s_v2/scalding/embedding
38:72 (14%)
124:158 (14%)
view
32 x 2 src/scala/com/twitter/si...ding/embedding/producer
src/scala/com/twitter/si...ding/embedding/producer
161:198 (17%)
117:154 (33%)
view
32 x 2 science/search/ingester/config
science/search/ingester/config
1:44 (19%)
1:43 (23%)
view
32 x 2 science/search/ingester/config
science/search/ingester/config
1:44 (23%)
1:44 (19%)
view
32 x 2 src/scala/com/twitter/si...ding/embedding/producer
src/scala/com/twitter/si...ding/embedding/producer
231:268 (17%)
249:286 (13%)
view
32 x 2 src/scala/com/twitter/si...ding/embedding/producer
src/scala/com/twitter/si...ding/embedding/producer
231:268 (17%)
321:358 (13%)
view
32 x 2 ann/src/main/scala/com/t...ng/offline/indexbuilder
ann/src/main/scala/com/t...line/indexbuilderfrombq
38:70 (53%)
103:135 (27%)
view
32 x 2 science/search/ingester/config
science/search/ingester/config
1:44 (23%)
1:43 (23%)
view
32 x 2 src/scala/com/twitter/simclusters_v2/scalding
src/scala/com/twitter/simclusters_v2/scalding
250:287 (6%)
240:277 (13%)
view
31 x 2 user-signal-service/serv...ersignalservice/signals
user-signal-service/serv...ersignalservice/signals
23:59 (77%)
24:60 (63%)
view
31 x 2 pushservice/src/main/sca...cate/ntab_caret_fatigue
pushservice/src/main/sca...cate/ntab_caret_fatigue
627:660 (3%)
801:834 (3%)
view
30 x 2 twml/twml/contrib/calibrators
twml/twml/contrib/calibrators
34:68 (38%)
432:466 (6%)
view
30 x 2 src/scala/com/twitter/recos/user_tweet_graph
src/scala/com/twitter/recos/user_video_graph
231:265 (16%)
234:268 (16%)
view
30 x 2 unified_user_actions/ser..._actions/service/module
unified_user_actions/ser..._actions/service/module
21:53 (47%)
21:53 (47%)
view
Duplicated Units
The list of top 50 duplicated units.
See data for all 70 unit duplicates...
Size#FoldersFilesLinesCode
44 x 3 follow-recommendations-s...ature_hydration/sources
follow-recommendations-s...ature_hydration/sources
follow-recommendations-s...ature_hydration/sources
100:148 
100:148 
100:148 
view
40 x 2 src/scala/com/twitter/recos/hose/common
src/scala/com/twitter/recos/hose/common
88:131 
94:137 
view
36 x 2 twml/libtwml/src/lib
twml/libtwml/src/lib
14:51 
14:51 
view
26 x 2 twml/libtwml/src/ops
twml/libtwml/src/ops
45:79 
15:50 
view
23 x 2 visibilitylib/src/main/s...ility/interfaces/search
visibilitylib/src/main/s...lity/interfaces/blender
427:452 
377:402 
view
23 x 2 recos-injector/server/sr...ter/recosinjector/edges
recos-injector/server/sr...ter/recosinjector/edges
43:69 
115:141 
view
21 x 3 src/scala/com/twitter/si...ding/embedding/producer
src/scala/com/twitter/si...ding/embedding/producer
src/scala/com/twitter/si...ding/embedding/producer
251:274 
119:142 
163:186 
view
20 x 2 src/scala/com/twitter/recos/user_video_graph/util
src/scala/com/twitter/recos/user_tweet_graph/util
11:34 
11:34 
view
19 x 2 src/java/com/twitter/sea...mon/converter/earlybird
src/java/com/twitter/sea...mon/converter/earlybird
93:118 
44:69 
view
19 x 2 src/scala/com/twitter/si...alding/embedding/common
src/scala/com/twitter/simclusters_v2/scalding
405:425 
50:70 
view
18 x 2 twml/libtwml/src/ops
twml/libtwml/src/ops
427:451 
695:719 
view
17 x 2 src/scala/com/twitter/recos/user_video_graph/store
src/scala/com/twitter/recos/user_tweet_graph/store
19:40 
19:40 
view
16 x 2 cr-mixer/server/src/main...mixer/similarity_engine
cr-mixer/server/src/main...mixer/similarity_engine
85:102 
80:97 
view
16 x 2 cr-mixer/server/src/main...twitter/cr_mixer/filter
cr-mixer/server/src/main...twitter/cr_mixer/filter
81:98 
78:95 
view
16 x 2 src/scala/com/twitter/si.../scalding/embedding/tfg
src/scala/com/twitter/si.../scalding/embedding/tfg
80:97 
120:137 
view
15 x 2 twml/libtwml/src/ops
twml/libtwml/src/ops
473:491 
741:759 
view
15 x 3 cr-mixer/server/src/main...er/candidate_generation
cr-mixer/server/src/main...er/candidate_generation
cr-mixer/server/src/main...er/candidate_generation
117:138 
134:155 
442:462 
view
15 x 2 cr-mixer/server/src/main...mixer/similarity_engine
cr-mixer/server/src/main...mixer/similarity_engine
392:410 
315:333 
view
14 x 2 src/scala/com/twitter/recos/user_video_graph
src/scala/com/twitter/recos/user_tweet_graph
91:106 
91:106 
view
13 x 2 pushservice/src/main/sca.../pushservice/model/ibis
pushservice/src/main/sca.../pushservice/model/ibis
32:46 
28:42 
view
13 x 2 src/scala/com/twitter/re...user_tweet_entity_graph
src/scala/com/twitter/recos/user_tweet_graph/util
80:94 
62:76 
view
12 x 2 timelines/data_processin...ation_framework/metrics
timelines/data_processin...ation_framework/metrics
36:50 
14:28 
view
12 x 2 home-mixer/server/src/ma...ct/scored_tweets/filter
home-mixer/server/src/ma.../product/for_you/filter
45:60 
60:75 
view
11 x 2 pushservice/src/main/sca...e/pushservice/predicate
pushservice/src/main/sca...e/pushservice/predicate
513:525 
43:55 
view
11 x 2 unified_user_actions/ser..._actions/service/module
unified_user_actions/ser..._actions/service/module
103:121 
100:118 
view
11 x 2 src/scala/com/twitter/recos/user_video_graph/util
src/scala/com/twitter/recos/user_tweet_graph/util
43:55 
43:55 
view
10 x 4 cr-mixer/server/src/main...er/module/thrift_client
cr-mixer/server/src/main...er/module/thrift_client
cr-mixer/server/src/main...er/module/thrift_client
cr-mixer/server/src/main...er/module/thrift_client
32:45 
29:42 
32:45 
29:42 
view
10 x 2 recos-injector/server/sr...ter/recosinjector/edges
recos-injector/server/sr...ter/recosinjector/edges
27:38 
37:48 
view
10 x 2 tweetypie/server/src/mai...etypie/service/observer
tweetypie/server/src/mai...etypie/service/observer
12:24 
13:25 
view
10 x 3 src/scala/com/twitter/si...ding/embedding/producer
src/scala/com/twitter/si...ding/embedding/producer
src/scala/com/twitter/si...ding/embedding/producer
276:287 
144:155 
188:199 
view
10 x 2 src/scala/com/twitter/si...alding/embedding/common
src/scala/com/twitter/simclusters_v2/scio/common
524:535 
288:299 
view
10 x 2 src/scala/com/twitter/simclusters_v2/scalding/mbcg
src/scala/com/twitter/simclusters_v2/scalding/mbcg
66:77 
82:93 
view
10 x 2 src/scala/com/twitter/recos/user_video_graph
src/scala/com/twitter/recos/user_tweet_graph
48:59 
65:76 
view
10 x 2 src/scala/com/twitter/recos/user_video_graph
src/scala/com/twitter/recos/user_tweet_graph
61:72 
78:89 
view
9 x 2 product-mixer/core/src/m...uct_mixer/core/pipeline
product-mixer/core/src/m...uct_mixer/core/pipeline
169:180 
116:127 
view
9 x 2 visibilitylib/src/main/s...ility/interfaces/search
visibilitylib/src/main/s...lity/interfaces/blender
454:465 
404:415 
view
9 x 2 visibilitylib/src/main/s...ility/interfaces/tweets
visibilitylib/src/main/s...nterfaces/conversations
410:420 
249:259 
view
9 x 2 cr-mixer/server/src/main...mixer/similarity_engine
cr-mixer/server/src/main...mixer/similarity_engine
61:71 
62:72 
view
9 x 2 cr-mixer/server/src/main...mixer/similarity_engine
cr-mixer/server/src/main...mixer/similarity_engine
132:142 
133:143 
view
9 x 2 timelines/data_processin...ation_framework/metrics
timelines/data_processin...ation_framework/metrics
17:27 
38:48 
view
9 x 2 src/java/com/twitter/search/earlybird/partition
src/java/com/twitter/search/earlybird/partition
32:43 
30:42 
view
9 x 2 src/scala/com/twitter/re...ph/relatedTweetHandlers
src/scala/com/twitter/re...ph/relatedTweetHandlers
26:37 
27:38 
view
9 x 3 src/scala/com/twitter/recos/user_video_graph
src/scala/com/twitter/re...user_tweet_entity_graph
src/scala/com/twitter/recos/user_tweet_graph
70:80 
81:91 
76:86 
view
8 x 2 product-mixer/shared-lib...shared_library/observer
product-mixer/shared-lib...shared_library/observer
197:206 
209:218 
view
8 x 2 twml/libtwml/src/ops
twml/libtwml/src/ops
359:368 
654:663 
view
8 x 2 representation-manager/s...ation_manager/migration
representation-manager/s...sentation_manager/store
560:569 
246:255 
view
8 x 2 pushservice/src/main/sca...rigate/pushservice/util
pushservice/src/main/sca...rigate/pushservice/util
22:33 
38:49 
view
8 x 2 cr-mixer/server/src/main...mixer/similarity_engine
cr-mixer/server/src/main...mixer/similarity_engine
442:453 
336:347 
view
8 x 2 timelineranker/server/sr...ter/timelineranker/util
home-mixer/server/src/ma.../util/tweetypie/content
104:113 
118:127 
view
8 x 2 timelineranker/server/sr...ter/timelineranker/util
home-mixer/server/src/ma.../util/tweetypie/content
216:225 
231:240 
view