Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................................XX.....XXX.] duration: 23.296314259s trace: https://jaeger.dev.storj.io/trace/5d94ed047f642894 FAIL (2026-03-18 21:43:50 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:33.375064813 (197h31m36.390734419s) [upload=189706]: Starting upload Mar 18 21:42:33.375134799 (197h31m36.390804408s) [upload=189706]: Done starting upload Mar 18 21:42:33.375215794 (197h31m36.390885383s) [upload=189706]: Uploading segments... Mar 18 21:42:33.375428458 (197h31m36.391098058s) [upload=189706]: Got next segment. Inline: false Mar 18 21:42:33.375465179 (197h31m36.39113478s) [upload=189706]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:33.375471522 (197h31m36.391141123s) [upload=189706]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:42:37.664656038 (197h31m40.680325628s) [seg_pos={0 0}, upload=189706]: Begin upload segment... Mar 18 21:42:37.664715414 (197h31m40.680385017s) [seg_pos={0 0}, upload=189706]: Done begin upload segment. Mar 18 21:42:37.664794796 (197h31m40.680464405s) [noise=true , address=145.79.227.18:4009 , piece_id=LEMCFQEOUGI736IUXTJXFTMXC5NFY4YWLQQP6PGFCSVDIFCY3G3A, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.664831674 (197h31m40.680501284s) [noise=true , address=145.79.227.22:4512 , piece_id=E6VE76TFIQUMEBBIPSKCXYRBDKCS6RXGM3I6XXVQSEKC3JBZIZBQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.665521619 (197h31m40.681191233s) [noise=true , address=45.91.116.9:4295 , piece_id=XAOJZRMPTGYMLM53BFW7EMZPUJGFCPUJSS5TLBC4OOMD2V2SXY5Q, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.665986694 (197h31m40.681656306s) [noise=true , address=145.79.227.40:4054 , piece_id=OPET6CQ2SDSZ6WPN7HOOQGBACY2UFODBSP5EA5CLDHOJUNTOZWYQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666471094 (197h31m40.682140702s) [noise=true , address=77.90.31.17:4773 , piece_id=MMO6R44QVGR24QFBH6EWFC6PVO43Z6B4J57DPACXBP3LRHAIV6CA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666696625 (197h31m40.682366236s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666912302 (197h31m40.682581917s) [noise=true , address=145.79.227.4:4190 , piece_id=C3JULQB52VXJEGZO43S4XS5NCXUVFGBYA6TXCOSLLKU6KDFOWMXQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667065653 (197h31m40.682735256s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667336294 (197h31m40.683005906s) [noise=true , address=145.79.227.19:4080 , piece_id=R53ZISNJ4Z6OLBLEP6OII4V2IQAIFZP6JONYBPQEKX72F2JC4ZPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667445511 (197h31m40.683115114s) [noise=true , address=145.79.227.2:4119 , piece_id=WPOGZKI7ESW25BJTNTUNHYWLGWUQ3IMZERXZEROZEAO3J2M2QBPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667971094 (197h31m40.683640715s) [noise=true , address=145.79.227.10:4407 , piece_id=Z67GGWR7YT2BCGSLQ4YNPBYXRJ4DTK7457GWSE4WYDWRGX47D6KA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668032048 (197h31m40.683701657s) [noise=true , address=145.79.227.16:4914 , piece_id=OERWN6RSHMYDRSXQBKY6URKRYZP2JEGMSFLJOBC3YAILYMS55VDA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668428467 (197h31m40.684098073s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668456188 (197h31m40.684125799s) [noise=true , address=145.79.227.52:4446 , piece_id=BEQJEJDRCRELR2H6FYXRWAJFYXARWWU62ZHXZXS6X7FDRVUZFCTQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668783554 (197h31m40.684453157s) [noise=true , address=145.79.227.50:4377 , piece_id=INGAOSQXHOEKHJWPSQE5N6H6VKUFSG4V5QUCTA6W7OVJFOFKQQQQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668973021 (197h31m40.684642632s) [noise=true , address=77.90.31.14:4662 , piece_id=4XTFQFJVHBW73U3DTODCCTZSBQJNXAMYHSNKHFDCEF3E5YPA7MUQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:38.270031363 (197h31m41.285700966s) [upload=189706]: Next returned nil segment Mar 18 21:42:38.270051227 (197h31m41.285720845s) [upload=189706]: Waiting for error group managing segments... Mar 18 21:42:38.397659417 (197h31m41.413329018s) [noise=true , address=77.90.31.17:4773 , piece_id=MMO6R44QVGR24QFBH6EWFC6PVO43Z6B4J57DPACXBP3LRHAIV6CA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.405646099 (197h31m41.421315698s) [noise=true , address=145.79.227.22:4512 , piece_id=E6VE76TFIQUMEBBIPSKCXYRBDKCS6RXGM3I6XXVQSEKC3JBZIZBQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.408651586 (197h31m41.424321186s) [noise=true , address=145.79.227.4:4190 , piece_id=C3JULQB52VXJEGZO43S4XS5NCXUVFGBYA6TXCOSLLKU6KDFOWMXQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.409031740 (197h31m41.42470134s) [noise=true , address=145.79.227.18:4009 , piece_id=LEMCFQEOUGI736IUXTJXFTMXC5NFY4YWLQQP6PGFCSVDIFCY3G3A, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.409219281 (197h31m41.424888881s) [noise=true , address=145.79.227.19:4080 , piece_id=R53ZISNJ4Z6OLBLEP6OII4V2IQAIFZP6JONYBPQEKX72F2JC4ZPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.411270163 (197h31m41.426939768s) [noise=true , address=145.79.227.50:4377 , piece_id=INGAOSQXHOEKHJWPSQE5N6H6VKUFSG4V5QUCTA6W7OVJFOFKQQQQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.413348366 (197h31m41.42901798s) [noise=true , address=145.79.227.10:4407 , piece_id=Z67GGWR7YT2BCGSLQ4YNPBYXRJ4DTK7457GWSE4WYDWRGX47D6KA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.415061924 (197h31m41.430731524s) [noise=true , address=145.79.227.2:4119 , piece_id=WPOGZKI7ESW25BJTNTUNHYWLGWUQ3IMZERXZEROZEAO3J2M2QBPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.415632963 (197h31m41.431302563s) [noise=true , address=145.79.227.52:4446 , piece_id=BEQJEJDRCRELR2H6FYXRWAJFYXARWWU62ZHXZXS6X7FDRVUZFCTQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.416575041 (197h31m41.432244646s) [noise=true , address=145.79.227.16:4914 , piece_id=OERWN6RSHMYDRSXQBKY6URKRYZP2JEGMSFLJOBC3YAILYMS55VDA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.418196668 (197h31m41.433866268s) [noise=true , address=77.90.31.14:4662 , piece_id=4XTFQFJVHBW73U3DTODCCTZSBQJNXAMYHSNKHFDCEF3E5YPA7MUQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.422831388 (197h31m41.438501023s) [noise=true , address=145.79.227.40:4054 , piece_id=OPET6CQ2SDSZ6WPN7HOOQGBACY2UFODBSP5EA5CLDHOJUNTOZWYQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.425116704 (197h31m41.440786312s) [noise=true , address=45.91.116.9:4295 , piece_id=XAOJZRMPTGYMLM53BFW7EMZPUJGFCPUJSS5TLBC4OOMD2V2SXY5Q, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.425145914 (197h31m41.440815526s) [seg_pos={0 0}, upload=189706]: Satisfied cohort requirements after 13 pieces Mar 18 21:42:38.425508871 (197h31m41.44117848s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u6e2ZjrPg2q3EDN2ZjDj9hzEmCAoRvt39HMeM79gcVmMVm53j): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425521836 (197h31m41.441191439s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425542587 (197h31m41.441212198s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aK78a6w6pX4fS7DTuJ7uHHvQfPvQjNaVVGgFc2uLGyDzbNXGG): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425554283 (197h31m41.441223894s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425688461 (197h31m41.441358071s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425697596 (197h31m41.441367211s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425723964 (197h31m41.441393558s) [seg_pos={0 0}, upload=189706]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:42:38.425821652 (197h31m41.441491264s) [upload=189706]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:42:38.425827934 (197h31m41.441497538s) [upload=189706]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:55.167706761 (197h31m58.183376353s) [upload=189706]: Done uploading segments... Mar 18 21:42:55.167730107 (197h31m58.183399698s) [upload=189706]: Upload finished. err: <nil> Duration: 23.296357598s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 47 out of 60 is failed (threshold=6) [......XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXXX] duration: 5.001107536s trace: https://jaeger.dev.storj.io/trace/88d5be3b8324a17 FAIL (2026-03-18 21:43:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001130748s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 7 out of 60 is failed (threshold=3) [.................................................XX....XXXXX] duration: 30.001180974s trace: https://jaeger.dev.storj.io/trace/f1b7d8417d4adb6 FAIL (2026-03-18 21:43:40 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.001193012s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................X....X.X] duration: 30.004475441s trace: https://jaeger.dev.storj.io/trace/3b96ae06fd84953d FAIL (2026-03-18 21:43:37 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.004498817s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XXX..] duration: 25.69905438s trace: https://jaeger.dev.storj.io/trace/78ef6f66bd16ad91 FAIL (2026-03-18 21:43:27 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:10.256027246 (197h31m13.27169685s) [upload=189700]: Starting upload Mar 18 21:42:10.256175294 (197h31m13.271844905s) [upload=189700]: Done starting upload Mar 18 21:42:10.256328593 (197h31m13.271998185s) [upload=189700]: Uploading segments... Mar 18 21:42:10.256454128 (197h31m13.272123732s) [upload=189700]: Got next segment. Inline: true Mar 18 21:42:10.256463382 (197h31m13.272132982s) [upload=189700]: Waiting for error group managing segments... Mar 18 21:42:10.256569091 (197h31m13.272238699s) [upload=189700]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:10.256598984 (197h31m13.272268587s) [upload=189700]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:42:10.256606889 (197h31m13.272276503s) [upload=189700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:39.082934610 (197h31m42.098604216s) [upload=189700]: Done uploading segments... Mar 18 21:42:39.082971404 (197h31m42.098641014s) [upload=189700]: Upload finished. err: <nil> Duration:* 25.699084689s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 6 out of 60 is failed (threshold=3) [..................................................XX....XXXX] duration: 30.001537599s trace: https://jaeger.dev.storj.io/trace/48ca5adafc30ca5f FAIL (2026-03-18 21:43:10 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.001563183s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................................XX.....XXX] duration: 30.0038269s trace: https://jaeger.dev.storj.io/trace/5d94ed047f642894 FAIL (2026-03-18 21:43:03 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:33.375064813 (197h31m36.390734419s) [upload=189706]: Starting upload Mar 18 21:42:33.375134799 (197h31m36.390804408s) [upload=189706]: Done starting upload Mar 18 21:42:33.375215794 (197h31m36.390885383s) [upload=189706]: Uploading segments... Mar 18 21:42:33.375428458 (197h31m36.391098058s) [upload=189706]: Got next segment. Inline: false Mar 18 21:42:33.375465179 (197h31m36.39113478s) [upload=189706]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:33.375471522 (197h31m36.391141123s) [upload=189706]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:42:37.664656038 (197h31m40.680325628s) [seg_pos={0 0}, upload=189706]: Begin upload segment... Mar 18 21:42:37.664715414 (197h31m40.680385017s) [seg_pos={0 0}, upload=189706]: Done begin upload segment. Mar 18 21:42:37.664794796 (197h31m40.680464405s) [noise=true , address=145.79.227.18:4009 , piece_id=LEMCFQEOUGI736IUXTJXFTMXC5NFY4YWLQQP6PGFCSVDIFCY3G3A, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.664831674 (197h31m40.680501284s) [noise=true , address=145.79.227.22:4512 , piece_id=E6VE76TFIQUMEBBIPSKCXYRBDKCS6RXGM3I6XXVQSEKC3JBZIZBQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.665521619 (197h31m40.681191233s) [noise=true , address=45.91.116.9:4295 , piece_id=XAOJZRMPTGYMLM53BFW7EMZPUJGFCPUJSS5TLBC4OOMD2V2SXY5Q, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.665986694 (197h31m40.681656306s) [noise=true , address=145.79.227.40:4054 , piece_id=OPET6CQ2SDSZ6WPN7HOOQGBACY2UFODBSP5EA5CLDHOJUNTOZWYQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666471094 (197h31m40.682140702s) [noise=true , address=77.90.31.17:4773 , piece_id=MMO6R44QVGR24QFBH6EWFC6PVO43Z6B4J57DPACXBP3LRHAIV6CA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666696625 (197h31m40.682366236s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.666912302 (197h31m40.682581917s) [noise=true , address=145.79.227.4:4190 , piece_id=C3JULQB52VXJEGZO43S4XS5NCXUVFGBYA6TXCOSLLKU6KDFOWMXQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667065653 (197h31m40.682735256s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667336294 (197h31m40.683005906s) [noise=true , address=145.79.227.19:4080 , piece_id=R53ZISNJ4Z6OLBLEP6OII4V2IQAIFZP6JONYBPQEKX72F2JC4ZPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667445511 (197h31m40.683115114s) [noise=true , address=145.79.227.2:4119 , piece_id=WPOGZKI7ESW25BJTNTUNHYWLGWUQ3IMZERXZEROZEAO3J2M2QBPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.667971094 (197h31m40.683640715s) [noise=true , address=145.79.227.10:4407 , piece_id=Z67GGWR7YT2BCGSLQ4YNPBYXRJ4DTK7457GWSE4WYDWRGX47D6KA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668032048 (197h31m40.683701657s) [noise=true , address=145.79.227.16:4914 , piece_id=OERWN6RSHMYDRSXQBKY6URKRYZP2JEGMSFLJOBC3YAILYMS55VDA, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668428467 (197h31m40.684098073s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668456188 (197h31m40.684125799s) [noise=true , address=145.79.227.52:4446 , piece_id=BEQJEJDRCRELR2H6FYXRWAJFYXARWWU62ZHXZXS6X7FDRVUZFCTQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668783554 (197h31m40.684453157s) [noise=true , address=145.79.227.50:4377 , piece_id=INGAOSQXHOEKHJWPSQE5N6H6VKUFSG4V5QUCTA6W7OVJFOFKQQQQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:37.668973021 (197h31m40.684642632s) [noise=true , address=77.90.31.14:4662 , piece_id=4XTFQFJVHBW73U3DTODCCTZSBQJNXAMYHSNKHFDCEF3E5YPA7MUQ, seg_pos={0 0}, upload=189706]: Uploading piece... Mar 18 21:42:38.270031363 (197h31m41.285700966s) [upload=189706]: Next returned nil segment Mar 18 21:42:38.270051227 (197h31m41.285720845s) [upload=189706]: Waiting for error group managing segments... Mar 18 21:42:38.397659417 (197h31m41.413329018s) [noise=true , address=77.90.31.17:4773 , piece_id=MMO6R44QVGR24QFBH6EWFC6PVO43Z6B4J57DPACXBP3LRHAIV6CA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.405646099 (197h31m41.421315698s) [noise=true , address=145.79.227.22:4512 , piece_id=E6VE76TFIQUMEBBIPSKCXYRBDKCS6RXGM3I6XXVQSEKC3JBZIZBQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.408651586 (197h31m41.424321186s) [noise=true , address=145.79.227.4:4190 , piece_id=C3JULQB52VXJEGZO43S4XS5NCXUVFGBYA6TXCOSLLKU6KDFOWMXQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.409031740 (197h31m41.42470134s) [noise=true , address=145.79.227.18:4009 , piece_id=LEMCFQEOUGI736IUXTJXFTMXC5NFY4YWLQQP6PGFCSVDIFCY3G3A, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.409219281 (197h31m41.424888881s) [noise=true , address=145.79.227.19:4080 , piece_id=R53ZISNJ4Z6OLBLEP6OII4V2IQAIFZP6JONYBPQEKX72F2JC4ZPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.411270163 (197h31m41.426939768s) [noise=true , address=145.79.227.50:4377 , piece_id=INGAOSQXHOEKHJWPSQE5N6H6VKUFSG4V5QUCTA6W7OVJFOFKQQQQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.413348366 (197h31m41.42901798s) [noise=true , address=145.79.227.10:4407 , piece_id=Z67GGWR7YT2BCGSLQ4YNPBYXRJ4DTK7457GWSE4WYDWRGX47D6KA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.415061924 (197h31m41.430731524s) [noise=true , address=145.79.227.2:4119 , piece_id=WPOGZKI7ESW25BJTNTUNHYWLGWUQ3IMZERXZEROZEAO3J2M2QBPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.415632963 (197h31m41.431302563s) [noise=true , address=145.79.227.52:4446 , piece_id=BEQJEJDRCRELR2H6FYXRWAJFYXARWWU62ZHXZXS6X7FDRVUZFCTQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.416575041 (197h31m41.432244646s) [noise=true , address=145.79.227.16:4914 , piece_id=OERWN6RSHMYDRSXQBKY6URKRYZP2JEGMSFLJOBC3YAILYMS55VDA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.418196668 (197h31m41.433866268s) [noise=true , address=77.90.31.14:4662 , piece_id=4XTFQFJVHBW73U3DTODCCTZSBQJNXAMYHSNKHFDCEF3E5YPA7MUQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.422831388 (197h31m41.438501023s) [noise=true , address=145.79.227.40:4054 , piece_id=OPET6CQ2SDSZ6WPN7HOOQGBACY2UFODBSP5EA5CLDHOJUNTOZWYQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.425116704 (197h31m41.440786312s) [noise=true , address=45.91.116.9:4295 , piece_id=XAOJZRMPTGYMLM53BFW7EMZPUJGFCPUJSS5TLBC4OOMD2V2SXY5Q, seg_pos={0 0}, upload=189706]: Done uploading piece. err: <nil> Mar 18 21:42:38.425145914 (197h31m41.440815526s) [seg_pos={0 0}, upload=189706]: Satisfied cohort requirements after 13 pieces Mar 18 21:42:38.425508871 (197h31m41.44117848s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1u6e2ZjrPg2q3EDN2ZjDj9hzEmCAoRvt39HMeM79gcVmMVm53j): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425521836 (197h31m41.441191439s) [noise=true , address=145.79.227.26:4656 , piece_id=X66XSA75Y2HZX3RVYHV3VDIAVZX6PNM2ZUOX3EYJEN35D7YRKTPQ, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425542587 (197h31m41.441212198s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aK78a6w6pX4fS7DTuJ7uHHvQfPvQjNaVVGgFc2uLGyDzbNXGG): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425554283 (197h31m41.441223894s) [noise=true , address=145.79.227.48:4304 , piece_id=UHWNG5OQWPDAKNHL6IRMTWFVUFIN4AHP7AZQS5NSTQDJQGC6JV7A, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425688461 (197h31m41.441358071s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:38.425697596 (197h31m41.441367211s) [noise=true , address=145.79.227.33:4869 , piece_id=B3FCIWULOIP4UCZQVRRT7FKG2VETLBCB2NGUYFQRRMPTUJQIL7ZA, seg_pos={0 0}, upload=189706]: Long tail cancellation due to optimal threshold Mar 18 21:42:38.425723964 (197h31m41.441393558s) [seg_pos={0 0}, upload=189706]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:42:38.425821652 (197h31m41.441491264s) [upload=189706]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:42:38.425827934 (197h31m41.441497538s) [upload=189706]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:55.167706761 (197h31m58.183376353s) [upload=189706]: Done uploading segments... Mar 18 21:42:55.167730107 (197h31m58.183399698s) [upload=189706]: Upload finished. err: <nil> Duration: 30.003852084s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XXX.] duration: 21.428238575s trace: https://jaeger.dev.storj.io/trace/78ef6f66bd16ad91 FAIL (2026-03-18 21:43:01 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:10.256027246 (197h31m13.27169685s) [upload=189700]: Starting upload Mar 18 21:42:10.256175294 (197h31m13.271844905s) [upload=189700]: Done starting upload Mar 18 21:42:10.256328593 (197h31m13.271998185s) [upload=189700]: Uploading segments... Mar 18 21:42:10.256454128 (197h31m13.272123732s) [upload=189700]: Got next segment. Inline: true Mar 18 21:42:10.256463382 (197h31m13.272132982s) [upload=189700]: Waiting for error group managing segments... Mar 18 21:42:10.256569091 (197h31m13.272238699s) [upload=189700]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:10.256598984 (197h31m13.272268587s) [upload=189700]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:42:10.256606889 (197h31m13.272276503s) [upload=189700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:39.082934610 (197h31m42.098604216s) [upload=189700]: Done uploading segments... Mar 18 21:42:39.082971404 (197h31m42.098641014s) [upload=189700]: Upload finished. err: <nil> Duration:* 21.428257508s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 46 out of 60 is failed (threshold=6) [.......XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXXX] duration: 5.001548723s trace: https://jaeger.dev.storj.io/trace/66e6701df5ebc343 FAIL (2026-03-18 21:43:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001636972s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X....X.] duration: 22.883333976s trace: https://jaeger.dev.storj.io/trace/3eb81f5e79d77711 FAIL (2026-03-18 21:42:57 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 22.883365633s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 30.001753067s trace: https://jaeger.dev.storj.io/trace/78ef6f66bd16ad91 FAIL (2026-03-18 21:42:39 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:10.256027246 (197h31m13.27169685s) [upload=189700]: Starting upload Mar 18 21:42:10.256175294 (197h31m13.271844905s) [upload=189700]: Done starting upload Mar 18 21:42:10.256328593 (197h31m13.271998185s) [upload=189700]: Uploading segments... Mar 18 21:42:10.256454128 (197h31m13.272123732s) [upload=189700]: Got next segment. Inline: true Mar 18 21:42:10.256463382 (197h31m13.272132982s) [upload=189700]: Waiting for error group managing segments... Mar 18 21:42:10.256569091 (197h31m13.272238699s) [upload=189700]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:10.256598984 (197h31m13.272268587s) [upload=189700]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:42:10.256606889 (197h31m13.272276503s) [upload=189700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:39.082934610 (197h31m42.098604216s) [upload=189700]: Done uploading segments... Mar 18 21:42:39.082971404 (197h31m42.098641014s) [upload=189700]: Upload finished. err: <nil> Duration:* 30.001768186s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................................XX.....XX] duration: 30.00213144s trace: https://jaeger.dev.storj.io/trace/60e12ac05cfc6cad FAIL (2026-03-18 21:42:32 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:42:02.417656046 (197h31m5.433325652s) [upload=189697]: Starting upload Mar 18 21:42:02.417789276 (197h31m5.433458885s) [upload=189697]: Done starting upload Mar 18 21:42:02.418076625 (197h31m5.433746218s) [upload=189697]: Uploading segments... Mar 18 21:42:02.418408602 (197h31m5.434078206s) [upload=189697]: Got next segment. Inline: false Mar 18 21:42:02.418533265 (197h31m5.43420287s) [upload=189697]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:42:02.418540334 (197h31m5.434209936s) [upload=189697]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:42:18.695643535 (197h31m21.711313155s) [seg_pos={0 0}, upload=189697]: Begin upload segment... Mar 18 21:42:18.695691242 (197h31m21.711360841s) [seg_pos={0 0}, upload=189697]: Done begin upload segment. Mar 18 21:42:18.695716852 (197h31m21.711386465s) [noise=true , address=145.79.227.40:4050 , piece_id=7YJNFY4QRMLQVSCDCFI35SLRJ2O3SGSPXKWUVGE6CCBD5WZI37GQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.695876691 (197h31m21.711546302s) [noise=true , address=145.79.227.20:4439 , piece_id=SRARUVJRMKROJQ6LU3AK75M63WFRGSFATQHUVI5KJPTCHFU4MSWQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.696525961 (197h31m21.712195573s) [noise=true , address=45.91.116.11:4152 , piece_id=KKGWY2V3OPHHSNVQ4WPVOWEUDCM2HCOEPYTR7HIANWQGUMLQJ6EQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.696678593 (197h31m21.712348194s) [noise=true , address=145.79.227.36:4517 , piece_id=KPOYF4N3MOMXXKKASLPUUEE2YHZWN4H2RTPRLROW7OU7MMC5AKZA, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.696993890 (197h31m21.712663491s) [noise=true , address=145.79.227.50:4377 , piece_id=JNUZIREIYZF7HVQFGFHHD6AUZPGWGHLMLUOCZ65HKG4ZE4UOUU5A, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.697114858 (197h31m21.712784458s) [noise=true , address=145.79.227.29:4947 , piece_id=AKXVS5QHKW4TZOQ5BR4HW2LPGIBDK53Q7NWCJYOMALK7V3JOUX2Q, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.697376505 (197h31m21.713046115s) [noise=true , address=77.90.31.15:4700 , piece_id=H277NWAXFIAVY25DGABAGCO3E34ALPZQ5SCPBVR7PVHRJ2YSB2HA, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.697542371 (197h31m21.713211971s) [noise=true , address=145.79.227.43:4161 , piece_id=BKPW7U55KWTYGLRXNSJ66ZYYEBJGZQFQC44WNZVLA5VTLLINHZJQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.697695110 (197h31m21.713364709s) [noise=true , address=145.79.227.33:4875 , piece_id=XESVVZO7XM6HCLKFOGFTC74CSIT4Z55V4OJQNGJKOWDDBX27JWWA, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.698022818 (197h31m21.713692421s) [noise=true , address=77.90.31.18:4810 , piece_id=YKOJR7RINILGS26VNUVQ7VSCSB7Q7KGC6SIHLRH5QSXKMI2G3VQQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.698037811 (197h31m21.713707412s) [noise=true , address=145.79.227.27:4694 , piece_id=SR4R4YFAAR24NMSZTEFZWYI33KO6DPLSMDC35WUORAVYINGZJ5KQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.698293088 (197h31m21.713962674s) [noise=true , address=45.91.116.5:4299 , piece_id=FNC7UX2TOZQJUA4DSZGTOXO3JK5UKY77JQSYYMDIARXLEAO72PVQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.698441877 (197h31m21.714111478s) [noise=true , address=145.79.227.2:4115 , piece_id=BDWEGOE6QQO23MZ2AO3QZTTQCA3IC44GUYRVR3AOB3PKTI7PDS3A, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.698581463 (197h31m21.71425105s) [noise=true , address=145.79.227.37:4570 , piece_id=27RDO3IG6WCZE4RXW53UVSJ4OIBTTNTBNRORP4PEVTNBXF4I2URA, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.698872122 (197h31m21.714541731s) [noise=true , address=145.79.227.47:4265 , piece_id=O7VMOCF2PK6CW4Y2QZEIIQYNMHMVFZPCXED25ATCVPDV7IWBPB2A, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:18.700720459 (197h31m21.716390053s) [noise=true , address=145.79.227.11:4733 , piece_id=DRYA2AX5UREDVDRL4KZOB5BNEXYNZFWSPNMC6D62WB7W5QXYECZQ, seg_pos={0 0}, upload=189697]: Uploading piece... Mar 18 21:42:19.287654268 (197h31m22.30332387s) [upload=189697]: Next returned nil segment Mar 18 21:42:19.287679198 (197h31m22.303348808s) [upload=189697]: Waiting for error group managing segments... Mar 18 21:42:19.424763134 (197h31m22.440432743s) [noise=true , address=77.90.31.18:4810 , piece_id=YKOJR7RINILGS26VNUVQ7VSCSB7Q7KGC6SIHLRH5QSXKMI2G3VQQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.427461527 (197h31m22.44313113s) [noise=true , address=77.90.31.15:4700 , piece_id=H277NWAXFIAVY25DGABAGCO3E34ALPZQ5SCPBVR7PVHRJ2YSB2HA, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.432400301 (197h31m22.448069901s) [noise=true , address=145.79.227.50:4377 , piece_id=JNUZIREIYZF7HVQFGFHHD6AUZPGWGHLMLUOCZ65HKG4ZE4UOUU5A, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.436191214 (197h31m22.451860821s) [noise=true , address=145.79.227.20:4439 , piece_id=SRARUVJRMKROJQ6LU3AK75M63WFRGSFATQHUVI5KJPTCHFU4MSWQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.436543389 (197h31m22.45221299s) [noise=true , address=45.91.116.11:4152 , piece_id=KKGWY2V3OPHHSNVQ4WPVOWEUDCM2HCOEPYTR7HIANWQGUMLQJ6EQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.438064568 (197h31m22.453734168s) [noise=true , address=145.79.227.27:4694 , piece_id=SR4R4YFAAR24NMSZTEFZWYI33KO6DPLSMDC35WUORAVYINGZJ5KQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.441153829 (197h31m22.45682343s) [noise=true , address=145.79.227.40:4050 , piece_id=7YJNFY4QRMLQVSCDCFI35SLRJ2O3SGSPXKWUVGE6CCBD5WZI37GQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.445617792 (197h31m22.461287398s) [noise=true , address=145.79.227.2:4115 , piece_id=BDWEGOE6QQO23MZ2AO3QZTTQCA3IC44GUYRVR3AOB3PKTI7PDS3A, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.445982446 (197h31m22.461652048s) [noise=true , address=145.79.227.33:4875 , piece_id=XESVVZO7XM6HCLKFOGFTC74CSIT4Z55V4OJQNGJKOWDDBX27JWWA, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.446667573 (197h31m22.46233718s) [noise=true , address=45.91.116.5:4299 , piece_id=FNC7UX2TOZQJUA4DSZGTOXO3JK5UKY77JQSYYMDIARXLEAO72PVQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.449617438 (197h31m22.465287043s) [noise=true , address=145.79.227.47:4265 , piece_id=O7VMOCF2PK6CW4Y2QZEIIQYNMHMVFZPCXED25ATCVPDV7IWBPB2A, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.451584770 (197h31m22.467254371s) [noise=true , address=145.79.227.29:4947 , piece_id=AKXVS5QHKW4TZOQ5BR4HW2LPGIBDK53Q7NWCJYOMALK7V3JOUX2Q, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.453078554 (197h31m22.468748158s) [noise=true , address=145.79.227.11:4733 , piece_id=DRYA2AX5UREDVDRL4KZOB5BNEXYNZFWSPNMC6D62WB7W5QXYECZQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: <nil> Mar 18 21:42:19.453096496 (197h31m22.468766097s) [seg_pos={0 0}, upload=189697]: Satisfied cohort requirements after 13 pieces Mar 18 21:42:19.453377163 (197h31m22.469046772s) [noise=true , address=145.79.227.37:4570 , piece_id=27RDO3IG6WCZE4RXW53UVSJ4OIBTTNTBNRORP4PEVTNBXF4I2URA, seg_pos={0 0}, upload=189697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1q5evYAfULsn19ddsAvSzJwxiDd8wsWbjP6gpcRkuU1b1q57eJ): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:19.453384477 (197h31m22.46905408s) [noise=true , address=145.79.227.37:4570 , piece_id=27RDO3IG6WCZE4RXW53UVSJ4OIBTTNTBNRORP4PEVTNBXF4I2URA, seg_pos={0 0}, upload=189697]: Long tail cancellation due to optimal threshold Mar 18 21:42:19.453527183 (197h31m22.46919679s) [noise=true , address=145.79.227.43:4161 , piece_id=BKPW7U55KWTYGLRXNSJ66ZYYEBJGZQFQC44WNZVLA5VTLLINHZJQ, seg_pos={0 0}, upload=189697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyhMcYjL5b9posZurgKXBnCHaMhaEjL3q3YWyc3aWkcYwbvTD): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:19.453533387 (197h31m22.469202987s) [noise=true , address=145.79.227.43:4161 , piece_id=BKPW7U55KWTYGLRXNSJ66ZYYEBJGZQFQC44WNZVLA5VTLLINHZJQ, seg_pos={0 0}, upload=189697]: Long tail cancellation due to optimal threshold Mar 18 21:42:19.453677790 (197h31m22.469347399s) [noise=true , address=145.79.227.36:4517 , piece_id=KPOYF4N3MOMXXKKASLPUUEE2YHZWN4H2RTPRLROW7OU7MMC5AKZA, seg_pos={0 0}, upload=189697]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS): protocol: expected piece hash; context canceled; EOF Mar 18 21:42:19.453690226 (197h31m22.469359826s) [noise=true , address=145.79.227.36:4517 , piece_id=KPOYF4N3MOMXXKKASLPUUEE2YHZWN4H2RTPRLROW7OU7MMC5AKZA, seg_pos={0 0}, upload=189697]: Long tail cancellation due to optimal threshold Mar 18 21:42:19.453711762 (197h31m22.469381354s) [seg_pos={0 0}, upload=189697]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:42:19.453804025 (197h31m22.469473629s) [upload=189697]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:42:19.453812865 (197h31m22.46948248s) [upload=189697]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:27.988758516 (197h31m31.00442812s) [upload=189697]: Done uploading segments... Mar 18 21:42:27.988780210 (197h31m31.004449811s) [upload=189697]: Upload finished. err: <nil> Duration: 30.002157397s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.001367944s trace: https://jaeger.dev.storj.io/trace/2091b044248e599a FAIL (2026-03-18 21:42:09 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:40.120881538 (197h30m43.136551138s) [upload=189693]: Starting upload Mar 18 21:41:40.121006332 (197h30m43.136675926s) [upload=189693]: Done starting upload Mar 18 21:41:40.121116258 (197h30m43.136785857s) [upload=189693]: Uploading segments... Mar 18 21:41:40.121231879 (197h30m43.136901483s) [upload=189693]: Got next segment. Inline: true Mar 18 21:41:40.121238612 (197h30m43.136908216s) [upload=189693]: Waiting for error group managing segments... Mar 18 21:41:40.121369082 (197h30m43.137038848s) [upload=189693]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:40.121377813 (197h30m43.137047422s) [upload=189693]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:41:40.121382320 (197h30m43.137051931s) [upload=189693]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:42:04.954963532 (197h31m7.970633133s) [upload=189693]: Done uploading segments... Mar 18 21:42:04.954984367 (197h31m7.970653961s) [upload=189693]: Upload finished. err: <nil> Duration:* 30.001393282s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 5 out of 60 is failed (threshold=3) [...................................................XX....XXX] duration: 30.001088862s trace: https://jaeger.dev.storj.io/trace/1d7f86d24ec17772 FAIL (2026-03-18 21:42:09 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.001117797s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [........XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXXX] duration: 5.000715173s trace: https://jaeger.dev.storj.io/trace/7736a03b25f2ff20 FAIL (2026-03-18 21:42:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000737863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X....X] duration: 30.005395566s trace: https://jaeger.dev.storj.io/trace/3eb81f5e79d77711 FAIL (2026-03-18 21:42:05 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.005481469s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [.........XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXXX] duration: 5.000794843s trace: https://jaeger.dev.storj.io/trace/7a3bb9f117e697f7 FAIL (2026-03-18 21:41:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000818402s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................XX.....X] duration: 30.003100164s trace: https://jaeger.dev.storj.io/trace/4fc9ef2ae31ecd22 FAIL (2026-03-18 21:41:45 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:41:15.503249526 (197h30m18.518919116s) [upload=189686]: Starting upload Mar 18 21:41:15.503318234 (197h30m18.518987825s) [upload=189686]: Done starting upload Mar 18 21:41:15.503408749 (197h30m18.519078341s) [upload=189686]: Uploading segments... Mar 18 21:41:15.503656555 (197h30m18.519326157s) [upload=189686]: Got next segment. Inline: false Mar 18 21:41:15.504053177 (197h30m18.519722786s) [upload=189686]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:41:15.504067001 (197h30m18.519736612s) [upload=189686]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:41:31.731468550 (197h30m34.747138153s) [seg_pos={0 0}, upload=189686]: Begin upload segment... Mar 18 21:41:31.731519809 (197h30m34.747189396s) [seg_pos={0 0}, upload=189686]: Done begin upload segment. Mar 18 21:41:31.731549457 (197h30m34.74721906s) [noise=true , address=145.79.227.15:4879 , piece_id=5HC4JLZPXTO6K75A2PFVRHATA3NGLCLHDHRUNDRHRX4GVGOB2YIA, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.731587350 (197h30m34.74725697s) [noise=true , address=145.79.227.50:4374 , piece_id=TCPENHS7SPJAQ6ZX4JYFYOZZOGE4ZR2ATTMJCV26ISMLK7ZAY6UA, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.732577179 (197h30m34.74824679s) [noise=true , address=45.91.116.7:4012 , piece_id=WLYVIBGTRJK7BSVGIBUZZ4X3N32WNRNK4VCOAZ5NA3KUVBNAVBRQ, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.733071808 (197h30m34.748741415s) [noise=true , address=145.79.227.35:4481 , piece_id=G7TWEZJRZIZHNJAVFXYCGCP4OKVYBDHDQTM64TIIQGMLHVJUSBRQ, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.733139183 (197h30m34.748808796s) [noise=true , address=145.79.227.43:4161 , piece_id=XPZ6PBEI5MNHCYVK5YBLTT4O4N2IYKV2WNPXAL2RK2G7ZIYWIPAA, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.733468284 (197h30m34.749137887s) [noise=true , address=145.79.227.5:4227 , piece_id=CTH3CMPJEYJDQETPHJRPIPDTBWORFHED525XIX2TKIHIVHGRUREQ, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.733532881 (197h30m34.74920249s) [noise=true , address=145.79.227.38:4981 , piece_id=CLXD5PGJBIIWOZOUMB5XPWXUCLLTBOQQEWS6INMIV2F6ZKVZS65A, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.733834983 (197h30m34.749504586s) [noise=true , address=145.79.227.8:4334 , piece_id=UB5BBUZD3F4O3A3E4MLEM2CYTPNFJTLGGFNMESULGHDQKLUXR4DA, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.734023775 (197h30m34.749693388s) [noise=true , address=145.79.227.27:4695 , piece_id=QE6NWVVRQKG26JRX2VN2IHCAU6KVXYWAGUB2ARG3BT52FKZEV6MA, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.734262300 (197h30m34.74993191s) [noise=true , address=145.79.227.14:4841 , piece_id=T5TWL2TDSDYDN3JG7PVWLYD4VNE55KEYFQHJ73G2FLBPXRAAYSZA, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.734417866 (197h30m34.750087453s) [noise=true , address=145.79.227.22:4511 , piece_id=7GXHYUPPUMRZU4ULJQN3AYHF4RLMVMNNCLO5GMZ2WJZT4V3N2DMA, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.734644759 (197h30m34.75031436s) [noise=true , address=145.79.227.23:4551 , piece_id=RJEPLWLVLFYHZWXVF3GZ2NMHEHS3VF7YDOPO5HHVHHK2Y3MQDVTQ, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.734801466 (197h30m34.750471069s) [noise=true , address=145.79.227.16:4914 , piece_id=CRDK5SD6LSTTOXXORW2IIE5VSSD72ST5T2XLCNWELPJT6PM27JCQ, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.735029769 (197h30m34.750699374s) [noise=true , address=45.91.116.4:4952 , piece_id=P6UISZSQ66K7B5VNXUSAU5LRDFNETP44RGSD7ENGENQJZUSA6M5A, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.735437298 (197h30m34.751106898s) [noise=true , address=77.90.31.12:4589 , piece_id=JURFT5XHNNZLX4VKC7D6F653KWZT47IJVWJDPYBBW34XJEVLOSGQ, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:31.735877393 (197h30m34.751547003s) [noise=true , address=145.79.227.6:4260 , piece_id=DAZBN6ZO4CK55YAKDXFJOAFQVN3EYWSWD6RM3D3PDALXAWZWRITQ, seg_pos={0 0}, upload=189686]: Uploading piece... Mar 18 21:41:32.333599746 (197h30m35.349269333s) [upload=189686]: Next returned nil segment Mar 18 21:41:32.333613110 (197h30m35.34928271s) [upload=189686]: Waiting for error group managing segments... Mar 18 21:41:32.459469423 (197h30m35.475139024s) [noise=true , address=77.90.31.12:4589 , piece_id=JURFT5XHNNZLX4VKC7D6F653KWZT47IJVWJDPYBBW34XJEVLOSGQ, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.469121284 (197h30m35.484790885s) [noise=true , address=145.79.227.50:4374 , piece_id=TCPENHS7SPJAQ6ZX4JYFYOZZOGE4ZR2ATTMJCV26ISMLK7ZAY6UA, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.476940838 (197h30m35.492610439s) [noise=true , address=145.79.227.23:4551 , piece_id=RJEPLWLVLFYHZWXVF3GZ2NMHEHS3VF7YDOPO5HHVHHK2Y3MQDVTQ, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.479745098 (197h30m35.495414699s) [noise=true , address=145.79.227.8:4334 , piece_id=UB5BBUZD3F4O3A3E4MLEM2CYTPNFJTLGGFNMESULGHDQKLUXR4DA, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.481099558 (197h30m35.49676916s) [noise=true , address=145.79.227.6:4260 , piece_id=DAZBN6ZO4CK55YAKDXFJOAFQVN3EYWSWD6RM3D3PDALXAWZWRITQ, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.482503144 (197h30m35.498172734s) [noise=true , address=145.79.227.16:4914 , piece_id=CRDK5SD6LSTTOXXORW2IIE5VSSD72ST5T2XLCNWELPJT6PM27JCQ, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.482906608 (197h30m35.498576209s) [noise=true , address=145.79.227.38:4981 , piece_id=CLXD5PGJBIIWOZOUMB5XPWXUCLLTBOQQEWS6INMIV2F6ZKVZS65A, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.483507975 (197h30m35.499177575s) [noise=true , address=145.79.227.22:4511 , piece_id=7GXHYUPPUMRZU4ULJQN3AYHF4RLMVMNNCLO5GMZ2WJZT4V3N2DMA, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.485006092 (197h30m35.500675692s) [noise=true , address=145.79.227.5:4227 , piece_id=CTH3CMPJEYJDQETPHJRPIPDTBWORFHED525XIX2TKIHIVHGRUREQ, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.485595755 (197h30m35.501265356s) [noise=true , address=145.79.227.14:4841 , piece_id=T5TWL2TDSDYDN3JG7PVWLYD4VNE55KEYFQHJ73G2FLBPXRAAYSZA, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.491892046 (197h30m35.507561646s) [noise=true , address=145.79.227.35:4481 , piece_id=G7TWEZJRZIZHNJAVFXYCGCP4OKVYBDHDQTM64TIIQGMLHVJUSBRQ, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.492004041 (197h30m35.507673642s) [noise=true , address=145.79.227.15:4879 , piece_id=5HC4JLZPXTO6K75A2PFVRHATA3NGLCLHDHRUNDRHRX4GVGOB2YIA, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.492940751 (197h30m35.508610352s) [noise=true , address=45.91.116.7:4012 , piece_id=WLYVIBGTRJK7BSVGIBUZZ4X3N32WNRNK4VCOAZ5NA3KUVBNAVBRQ, seg_pos={0 0}, upload=189686]: Done uploading piece. err: <nil> Mar 18 21:41:32.492965235 (197h30m35.508634839s) [seg_pos={0 0}, upload=189686]: Satisfied cohort requirements after 13 pieces Mar 18 21:41:32.493172636 (197h30m35.508842244s) [noise=true , address=145.79.227.27:4695 , piece_id=QE6NWVVRQKG26JRX2VN2IHCAU6KVXYWAGUB2ARG3BT52FKZEV6MA, seg_pos={0 0}, upload=189686]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1scGuKGuDJCYuUGp5nDdU1Sxr23L7EEPChWtLAhZ52i3RpiF6y): protocol: expected piece hash; context canceled; EOF Mar 18 21:41:32.493181025 (197h30m35.508850625s) [noise=true , address=145.79.227.27:4695 , piece_id=QE6NWVVRQKG26JRX2VN2IHCAU6KVXYWAGUB2ARG3BT52FKZEV6MA, seg_pos={0 0}, upload=189686]: Long tail cancellation due to optimal threshold Mar 18 21:41:32.493209444 (197h30m35.50887905s) [noise=true , address=45.91.116.4:4952 , piece_id=P6UISZSQ66K7B5VNXUSAU5LRDFNETP44RGSD7ENGENQJZUSA6M5A, seg_pos={0 0}, upload=189686]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LfpoDAb2seboZSoyGNp5zw2YZ4okmnvNZ5UQsgqgzn4MmDLJn): protocol: expected piece hash; context canceled; EOF Mar 18 21:41:32.493216579 (197h30m35.508886183s) [noise=true , address=45.91.116.4:4952 , piece_id=P6UISZSQ66K7B5VNXUSAU5LRDFNETP44RGSD7ENGENQJZUSA6M5A, seg_pos={0 0}, upload=189686]: Long tail cancellation due to optimal threshold Mar 18 21:41:32.493361946 (197h30m35.509031552s) [noise=true , address=145.79.227.43:4161 , piece_id=XPZ6PBEI5MNHCYVK5YBLTT4O4N2IYKV2WNPXAL2RK2G7ZIYWIPAA, seg_pos={0 0}, upload=189686]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pyhMcYjL5b9posZurgKXBnCHaMhaEjL3q3YWyc3aWkcYwbvTD): protocol: expected piece hash; context canceled; EOF Mar 18 21:41:32.493367612 (197h30m35.509037214s) [noise=true , address=145.79.227.43:4161 , piece_id=XPZ6PBEI5MNHCYVK5YBLTT4O4N2IYKV2WNPXAL2RK2G7ZIYWIPAA, seg_pos={0 0}, upload=189686]: Long tail cancellation due to optimal threshold Mar 18 21:41:32.493385412 (197h30m35.509054999s) [seg_pos={0 0}, upload=189686]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:41:32.493471565 (197h30m35.509141165s) [upload=189686]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:41:32.493476984 (197h30m35.509146584s) [upload=189686]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:41:41.004334124 (197h30m44.020003736s) [upload=189686]: Done uploading segments... Mar 18 21:41:41.004359380 (197h30m44.020028995s) [upload=189686]: Upload finished. err: <nil> Duration: 30.003123553s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 4 out of 60 is failed (threshold=3) [....................................................XX....XX] duration: 30.001888276s trace: https://jaeger.dev.storj.io/trace/a150b95b00beb34 FAIL (2026-03-18 21:41:31 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001927046s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [..........XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XXX] duration: 5.000911274s trace: https://jaeger.dev.storj.io/trace/52716293d774ae67 FAIL (2026-03-18 21:41:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000938911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001494239s trace: https://jaeger.dev.storj.io/trace/4ac054bffad9608 FAIL (2026-03-18 21:40:56 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:40:26.840889435 (197h29m29.856559038s) [upload=189674]: Starting upload Mar 18 21:40:26.840962647 (197h29m29.856632239s) [upload=189674]: Done starting upload Mar 18 21:40:26.841064766 (197h29m29.856734359s) [upload=189674]: Uploading segments... Mar 18 21:40:26.841123788 (197h29m29.856793395s) [upload=189674]: Got next segment. Inline: true Mar 18 21:40:26.841129712 (197h29m29.856799316s) [upload=189674]: Waiting for error group managing segments... Mar 18 21:40:26.841184594 (197h29m29.856854194s) [upload=189674]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:40:26.841190017 (197h29m29.856859618s) [upload=189674]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Mar 18 21:40:26.841193766 (197h29m29.856863358s) [upload=189674]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:40:43.578971735 (197h29m46.594641345s) [upload=189674]: Done uploading segments... Mar 18 21:40:43.579007655 (197h29m46.594677257s) [upload=189674]: Upload finished. err: <nil> Duration:* 30.001515174s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................XX.....] duration: 23.105496299s trace: https://jaeger.dev.storj.io/trace/1fea1be3553256d8 FAIL (2026-03-18 21:40:52 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:37:13.407226914 (197h26m16.422896563s) [upload=189623]: Starting upload Mar 18 21:37:13.407304870 (197h26m16.422974473s) [upload=189623]: Done starting upload Mar 18 21:37:13.407540972 (197h26m16.423210564s) [upload=189623]: Uploading segments... Mar 18 21:37:13.408026524 (197h26m16.423696147s) [upload=189623]: Got next segment. Inline: false Mar 18 21:37:13.408123553 (197h26m16.423793153s) [upload=189623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:37:13.408130704 (197h26m16.423800304s) [upload=189623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:37:21.655023605 (197h26m24.670693205s) [seg_pos={0 0}, upload=189623]: Begin upload segment... Mar 18 21:37:21.655094469 (197h26m24.670764059s) [seg_pos={0 0}, upload=189623]: Done begin upload segment. Mar 18 21:37:21.655129337 (197h26m24.67079894s) [noise=true , address=145.79.227.4:4189 , piece_id=NMJ2X5ZX7D3U7KRZZYC2MC4XAAJNPQAZAWHLLBY2HNXSZCKCQCUQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.656236260 (197h26m24.671905872s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.656865510 (197h26m24.672535113s) [noise=true , address=145.79.227.20:4439 , piece_id=3FWVMDZ7DLJV652FESYLUQS6CRTDL4ESGMHWEATRSFTBY2B6LTLA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.657306588 (197h26m24.672976189s) [noise=true , address=145.79.227.35:4486 , piece_id=RSLN2AZ2HSBPPU7XTUASMYU2Y2PFZN4YW6ZMPIOJNZTEWKWEQXJA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.657690602 (197h26m24.673360202s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.658444416 (197h26m24.674114022s) [noise=true , address=145.79.227.8:4335 , piece_id=U3ICUMTLPLHULI57IQ3V3H4VVEEAXSMQ4GKVFXZOUYWRYN6O2TYA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.659018578 (197h26m24.674688178s) [noise=true , address=145.79.227.15:4877 , piece_id=BQJN5RVMLYB7IIX3IFHYVFN3D2A5GYPV5L3Q6FWJ3VKVWH4NZWRA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.659604237 (197h26m24.675273838s) [noise=true , address=145.79.227.26:4657 , piece_id=O6UZJ7OLIA4BJQ6IRCI5ERXO3IURKV3GLJVG476KSPMRVQGY7LYA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.660277318 (197h26m24.675946921s) [noise=true , address=145.79.227.46:4601 , piece_id=3SWEFW5XZ3DZLPLKJZE3NBRRJCDZ7F7TDA5UXM7QBJUOWZTVPUPQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.660830355 (197h26m24.67649996s) [noise=true , address=45.91.116.7:4012 , piece_id=2JJXJQZO47ZTD7PFKABUENRESMEJMJNKAYCV6E5PKPAPO3IOKNOA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.661333927 (197h26m24.677003527s) [noise=true , address=145.79.227.3:4149 , piece_id=O4PG4M6HCKSRJZMO6KHNSFCRF5JRD2RWLNOMQGPGN2BS6TN2B2RQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.664022598 (197h26m24.679692201s) [noise=true , address=145.79.227.18:4011 , piece_id=NLH7AVODIX6ICBEYYN5JXGGLWLVOXZVYA3XWEOGUEQCW4HZTHEFQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.666068628 (197h26m24.681738234s) [noise=true , address=145.79.227.48:4306 , piece_id=FOUGOBCQYYPKA45WADXNNU4CRYNJDWVFVIFBF6VRKINXKAVXNBTQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.667030758 (197h26m24.682700369s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.671055812 (197h26m24.686725418s) [noise=true , address=145.79.227.39:4018 , piece_id=DBAKT7XQX5IZVLEXDY63HP4NXGZVF6RTFGTNSCYKNX6EFAWVJTKQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.677611108 (197h26m24.693280721s) [noise=true , address=145.79.227.33:4874 , piece_id=NLQ737OMRU53WHBROA6TWFT4JU2RWL6AFGAVFFNWLHHEHMHFU5LQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:22.277424525 (197h26m25.293094127s) [upload=189623]: Next returned nil segment Mar 18 21:37:22.277443392 (197h26m25.293113004s) [upload=189623]: Waiting for error group managing segments... Mar 18 21:37:22.414377107 (197h26m25.430046707s) [noise=true , address=145.79.227.48:4306 , piece_id=FOUGOBCQYYPKA45WADXNNU4CRYNJDWVFVIFBF6VRKINXKAVXNBTQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.418771029 (197h26m25.434440635s) [noise=true , address=145.79.227.39:4018 , piece_id=DBAKT7XQX5IZVLEXDY63HP4NXGZVF6RTFGTNSCYKNX6EFAWVJTKQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.420950984 (197h26m25.436620574s) [noise=true , address=145.79.227.8:4335 , piece_id=U3ICUMTLPLHULI57IQ3V3H4VVEEAXSMQ4GKVFXZOUYWRYN6O2TYA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.421003817 (197h26m25.436673414s) [noise=true , address=145.79.227.4:4189 , piece_id=NMJ2X5ZX7D3U7KRZZYC2MC4XAAJNPQAZAWHLLBY2HNXSZCKCQCUQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.421987949 (197h26m25.437657549s) [noise=true , address=145.79.227.18:4011 , piece_id=NLH7AVODIX6ICBEYYN5JXGGLWLVOXZVYA3XWEOGUEQCW4HZTHEFQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.422142469 (197h26m25.437812071s) [noise=true , address=145.79.227.26:4657 , piece_id=O6UZJ7OLIA4BJQ6IRCI5ERXO3IURKV3GLJVG476KSPMRVQGY7LYA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.425641685 (197h26m25.441311285s) [noise=true , address=145.79.227.20:4439 , piece_id=3FWVMDZ7DLJV652FESYLUQS6CRTDL4ESGMHWEATRSFTBY2B6LTLA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.426101126 (197h26m25.44177072s) [noise=true , address=145.79.227.3:4149 , piece_id=O4PG4M6HCKSRJZMO6KHNSFCRF5JRD2RWLNOMQGPGN2BS6TN2B2RQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.427039270 (197h26m25.44270887s) [noise=true , address=145.79.227.15:4877 , piece_id=BQJN5RVMLYB7IIX3IFHYVFN3D2A5GYPV5L3Q6FWJ3VKVWH4NZWRA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.427616837 (197h26m25.443286441s) [noise=true , address=145.79.227.46:4601 , piece_id=3SWEFW5XZ3DZLPLKJZE3NBRRJCDZ7F7TDA5UXM7QBJUOWZTVPUPQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.431130437 (197h26m25.446800038s) [noise=true , address=45.91.116.7:4012 , piece_id=2JJXJQZO47ZTD7PFKABUENRESMEJMJNKAYCV6E5PKPAPO3IOKNOA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.431183849 (197h26m25.446853442s) [noise=true , address=145.79.227.33:4874 , piece_id=NLQ737OMRU53WHBROA6TWFT4JU2RWL6AFGAVFFNWLHHEHMHFU5LQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.456250331 (197h26m25.471919937s) [noise=true , address=145.79.227.35:4486 , piece_id=RSLN2AZ2HSBPPU7XTUASMYU2Y2PFZN4YW6ZMPIOJNZTEWKWEQXJA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.456277546 (197h26m25.47194715s) [seg_pos={0 0}, upload=189623]: Satisfied cohort requirements after 13 pieces Mar 18 21:37:22.456632071 (197h26m25.472301676s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.456649964 (197h26m25.47231957s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.456701058 (197h26m25.472370669s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.456714788 (197h26m25.472384393s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.457052179 (197h26m25.47272179s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.457069637 (197h26m25.472739249s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.457128468 (197h26m25.472798071s) [seg_pos={0 0}, upload=189623]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:37:22.457274145 (197h26m25.472943746s) [upload=189623]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:37:22.457281197 (197h26m25.472950797s) [upload=189623]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:37:38.941520305 (197h26m41.957189918s) [upload=189623]: Done uploading segments... Mar 18 21:37:38.941544429 (197h26m41.957214028s) [upload=189623]: Upload finished. err: <nil> Duration: 23.105528533s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 22.903290305s trace: https://jaeger.dev.storj.io/trace/2e6fa05e97e090 FAIL (2026-03-18 21:40:50 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 22.903363153s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [X..........XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.XX] duration: 5.000689315s trace: https://jaeger.dev.storj.io/trace/7671635b36f95e9a FAIL (2026-03-18 21:40:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000717963s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................XX....X] duration: 30.001483491s trace: https://jaeger.dev.storj.io/trace/1dd5f2bde0ec2260 FAIL (2026-03-18 21:40:39 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.00150831s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 23.819500544s trace: https://jaeger.dev.storj.io/trace/1fea1be3553256d8 FAIL (2026-03-18 21:40:04 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:37:13.407226914 (197h26m16.422896563s) [upload=189623]: Starting upload Mar 18 21:37:13.407304870 (197h26m16.422974473s) [upload=189623]: Done starting upload Mar 18 21:37:13.407540972 (197h26m16.423210564s) [upload=189623]: Uploading segments... Mar 18 21:37:13.408026524 (197h26m16.423696147s) [upload=189623]: Got next segment. Inline: false Mar 18 21:37:13.408123553 (197h26m16.423793153s) [upload=189623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:37:13.408130704 (197h26m16.423800304s) [upload=189623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:37:21.655023605 (197h26m24.670693205s) [seg_pos={0 0}, upload=189623]: Begin upload segment... Mar 18 21:37:21.655094469 (197h26m24.670764059s) [seg_pos={0 0}, upload=189623]: Done begin upload segment. Mar 18 21:37:21.655129337 (197h26m24.67079894s) [noise=true , address=145.79.227.4:4189 , piece_id=NMJ2X5ZX7D3U7KRZZYC2MC4XAAJNPQAZAWHLLBY2HNXSZCKCQCUQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.656236260 (197h26m24.671905872s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.656865510 (197h26m24.672535113s) [noise=true , address=145.79.227.20:4439 , piece_id=3FWVMDZ7DLJV652FESYLUQS6CRTDL4ESGMHWEATRSFTBY2B6LTLA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.657306588 (197h26m24.672976189s) [noise=true , address=145.79.227.35:4486 , piece_id=RSLN2AZ2HSBPPU7XTUASMYU2Y2PFZN4YW6ZMPIOJNZTEWKWEQXJA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.657690602 (197h26m24.673360202s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.658444416 (197h26m24.674114022s) [noise=true , address=145.79.227.8:4335 , piece_id=U3ICUMTLPLHULI57IQ3V3H4VVEEAXSMQ4GKVFXZOUYWRYN6O2TYA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.659018578 (197h26m24.674688178s) [noise=true , address=145.79.227.15:4877 , piece_id=BQJN5RVMLYB7IIX3IFHYVFN3D2A5GYPV5L3Q6FWJ3VKVWH4NZWRA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.659604237 (197h26m24.675273838s) [noise=true , address=145.79.227.26:4657 , piece_id=O6UZJ7OLIA4BJQ6IRCI5ERXO3IURKV3GLJVG476KSPMRVQGY7LYA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.660277318 (197h26m24.675946921s) [noise=true , address=145.79.227.46:4601 , piece_id=3SWEFW5XZ3DZLPLKJZE3NBRRJCDZ7F7TDA5UXM7QBJUOWZTVPUPQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.660830355 (197h26m24.67649996s) [noise=true , address=45.91.116.7:4012 , piece_id=2JJXJQZO47ZTD7PFKABUENRESMEJMJNKAYCV6E5PKPAPO3IOKNOA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.661333927 (197h26m24.677003527s) [noise=true , address=145.79.227.3:4149 , piece_id=O4PG4M6HCKSRJZMO6KHNSFCRF5JRD2RWLNOMQGPGN2BS6TN2B2RQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.664022598 (197h26m24.679692201s) [noise=true , address=145.79.227.18:4011 , piece_id=NLH7AVODIX6ICBEYYN5JXGGLWLVOXZVYA3XWEOGUEQCW4HZTHEFQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.666068628 (197h26m24.681738234s) [noise=true , address=145.79.227.48:4306 , piece_id=FOUGOBCQYYPKA45WADXNNU4CRYNJDWVFVIFBF6VRKINXKAVXNBTQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.667030758 (197h26m24.682700369s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.671055812 (197h26m24.686725418s) [noise=true , address=145.79.227.39:4018 , piece_id=DBAKT7XQX5IZVLEXDY63HP4NXGZVF6RTFGTNSCYKNX6EFAWVJTKQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.677611108 (197h26m24.693280721s) [noise=true , address=145.79.227.33:4874 , piece_id=NLQ737OMRU53WHBROA6TWFT4JU2RWL6AFGAVFFNWLHHEHMHFU5LQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:22.277424525 (197h26m25.293094127s) [upload=189623]: Next returned nil segment Mar 18 21:37:22.277443392 (197h26m25.293113004s) [upload=189623]: Waiting for error group managing segments... Mar 18 21:37:22.414377107 (197h26m25.430046707s) [noise=true , address=145.79.227.48:4306 , piece_id=FOUGOBCQYYPKA45WADXNNU4CRYNJDWVFVIFBF6VRKINXKAVXNBTQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.418771029 (197h26m25.434440635s) [noise=true , address=145.79.227.39:4018 , piece_id=DBAKT7XQX5IZVLEXDY63HP4NXGZVF6RTFGTNSCYKNX6EFAWVJTKQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.420950984 (197h26m25.436620574s) [noise=true , address=145.79.227.8:4335 , piece_id=U3ICUMTLPLHULI57IQ3V3H4VVEEAXSMQ4GKVFXZOUYWRYN6O2TYA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.421003817 (197h26m25.436673414s) [noise=true , address=145.79.227.4:4189 , piece_id=NMJ2X5ZX7D3U7KRZZYC2MC4XAAJNPQAZAWHLLBY2HNXSZCKCQCUQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.421987949 (197h26m25.437657549s) [noise=true , address=145.79.227.18:4011 , piece_id=NLH7AVODIX6ICBEYYN5JXGGLWLVOXZVYA3XWEOGUEQCW4HZTHEFQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.422142469 (197h26m25.437812071s) [noise=true , address=145.79.227.26:4657 , piece_id=O6UZJ7OLIA4BJQ6IRCI5ERXO3IURKV3GLJVG476KSPMRVQGY7LYA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.425641685 (197h26m25.441311285s) [noise=true , address=145.79.227.20:4439 , piece_id=3FWVMDZ7DLJV652FESYLUQS6CRTDL4ESGMHWEATRSFTBY2B6LTLA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.426101126 (197h26m25.44177072s) [noise=true , address=145.79.227.3:4149 , piece_id=O4PG4M6HCKSRJZMO6KHNSFCRF5JRD2RWLNOMQGPGN2BS6TN2B2RQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.427039270 (197h26m25.44270887s) [noise=true , address=145.79.227.15:4877 , piece_id=BQJN5RVMLYB7IIX3IFHYVFN3D2A5GYPV5L3Q6FWJ3VKVWH4NZWRA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.427616837 (197h26m25.443286441s) [noise=true , address=145.79.227.46:4601 , piece_id=3SWEFW5XZ3DZLPLKJZE3NBRRJCDZ7F7TDA5UXM7QBJUOWZTVPUPQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.431130437 (197h26m25.446800038s) [noise=true , address=45.91.116.7:4012 , piece_id=2JJXJQZO47ZTD7PFKABUENRESMEJMJNKAYCV6E5PKPAPO3IOKNOA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.431183849 (197h26m25.446853442s) [noise=true , address=145.79.227.33:4874 , piece_id=NLQ737OMRU53WHBROA6TWFT4JU2RWL6AFGAVFFNWLHHEHMHFU5LQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.456250331 (197h26m25.471919937s) [noise=true , address=145.79.227.35:4486 , piece_id=RSLN2AZ2HSBPPU7XTUASMYU2Y2PFZN4YW6ZMPIOJNZTEWKWEQXJA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.456277546 (197h26m25.47194715s) [seg_pos={0 0}, upload=189623]: Satisfied cohort requirements after 13 pieces Mar 18 21:37:22.456632071 (197h26m25.472301676s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.456649964 (197h26m25.47231957s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.456701058 (197h26m25.472370669s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.456714788 (197h26m25.472384393s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.457052179 (197h26m25.47272179s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.457069637 (197h26m25.472739249s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.457128468 (197h26m25.472798071s) [seg_pos={0 0}, upload=189623]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:37:22.457274145 (197h26m25.472943746s) [upload=189623]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:37:22.457281197 (197h26m25.472950797s) [upload=189623]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:37:38.941520305 (197h26m41.957189918s) [upload=189623]: Done uploading segments... Mar 18 21:37:38.941544429 (197h26m41.957214028s) [upload=189623]: Upload finished. err: <nil> Duration: 23.81953123s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 12.5256823s trace: https://jaeger.dev.storj.io/trace/2e6fa05e97e090 FAIL (2026-03-18 21:39:50 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 12.525711558s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XX..........XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.X] duration: 5.001009561s trace: https://jaeger.dev.storj.io/trace/7cfa1932860578be FAIL (2026-03-18 21:39:36 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001032236s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................................................XX....] duration: 929.990023ms trace: https://jaeger.dev.storj.io/trace/94bb3ca45ba55fa FAIL (2026-03-18 21:39:13 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 930.021872ms core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [.XX..........XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX.] duration: 2.174316916s trace: https://jaeger.dev.storj.io/trace/468bbab930a6aba3 FAIL (2026-03-18 21:38:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.174352422s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 1.172979813s trace: https://jaeger.dev.storj.io/trace/94bb3ca45ba55fa FAIL (2026-03-18 21:38:43 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.173015287s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 2.497672s trace: https://jaeger.dev.storj.io/trace/1fea1be3553256d8 FAIL (2026-03-18 21:38:30 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:37:13.407226914 (197h26m16.422896563s) [upload=189623]: Starting upload Mar 18 21:37:13.407304870 (197h26m16.422974473s) [upload=189623]: Done starting upload Mar 18 21:37:13.407540972 (197h26m16.423210564s) [upload=189623]: Uploading segments... Mar 18 21:37:13.408026524 (197h26m16.423696147s) [upload=189623]: Got next segment. Inline: false Mar 18 21:37:13.408123553 (197h26m16.423793153s) [upload=189623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:37:13.408130704 (197h26m16.423800304s) [upload=189623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:37:21.655023605 (197h26m24.670693205s) [seg_pos={0 0}, upload=189623]: Begin upload segment... Mar 18 21:37:21.655094469 (197h26m24.670764059s) [seg_pos={0 0}, upload=189623]: Done begin upload segment. Mar 18 21:37:21.655129337 (197h26m24.67079894s) [noise=true , address=145.79.227.4:4189 , piece_id=NMJ2X5ZX7D3U7KRZZYC2MC4XAAJNPQAZAWHLLBY2HNXSZCKCQCUQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.656236260 (197h26m24.671905872s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.656865510 (197h26m24.672535113s) [noise=true , address=145.79.227.20:4439 , piece_id=3FWVMDZ7DLJV652FESYLUQS6CRTDL4ESGMHWEATRSFTBY2B6LTLA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.657306588 (197h26m24.672976189s) [noise=true , address=145.79.227.35:4486 , piece_id=RSLN2AZ2HSBPPU7XTUASMYU2Y2PFZN4YW6ZMPIOJNZTEWKWEQXJA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.657690602 (197h26m24.673360202s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.658444416 (197h26m24.674114022s) [noise=true , address=145.79.227.8:4335 , piece_id=U3ICUMTLPLHULI57IQ3V3H4VVEEAXSMQ4GKVFXZOUYWRYN6O2TYA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.659018578 (197h26m24.674688178s) [noise=true , address=145.79.227.15:4877 , piece_id=BQJN5RVMLYB7IIX3IFHYVFN3D2A5GYPV5L3Q6FWJ3VKVWH4NZWRA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.659604237 (197h26m24.675273838s) [noise=true , address=145.79.227.26:4657 , piece_id=O6UZJ7OLIA4BJQ6IRCI5ERXO3IURKV3GLJVG476KSPMRVQGY7LYA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.660277318 (197h26m24.675946921s) [noise=true , address=145.79.227.46:4601 , piece_id=3SWEFW5XZ3DZLPLKJZE3NBRRJCDZ7F7TDA5UXM7QBJUOWZTVPUPQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.660830355 (197h26m24.67649996s) [noise=true , address=45.91.116.7:4012 , piece_id=2JJXJQZO47ZTD7PFKABUENRESMEJMJNKAYCV6E5PKPAPO3IOKNOA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.661333927 (197h26m24.677003527s) [noise=true , address=145.79.227.3:4149 , piece_id=O4PG4M6HCKSRJZMO6KHNSFCRF5JRD2RWLNOMQGPGN2BS6TN2B2RQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.664022598 (197h26m24.679692201s) [noise=true , address=145.79.227.18:4011 , piece_id=NLH7AVODIX6ICBEYYN5JXGGLWLVOXZVYA3XWEOGUEQCW4HZTHEFQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.666068628 (197h26m24.681738234s) [noise=true , address=145.79.227.48:4306 , piece_id=FOUGOBCQYYPKA45WADXNNU4CRYNJDWVFVIFBF6VRKINXKAVXNBTQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.667030758 (197h26m24.682700369s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.671055812 (197h26m24.686725418s) [noise=true , address=145.79.227.39:4018 , piece_id=DBAKT7XQX5IZVLEXDY63HP4NXGZVF6RTFGTNSCYKNX6EFAWVJTKQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.677611108 (197h26m24.693280721s) [noise=true , address=145.79.227.33:4874 , piece_id=NLQ737OMRU53WHBROA6TWFT4JU2RWL6AFGAVFFNWLHHEHMHFU5LQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:22.277424525 (197h26m25.293094127s) [upload=189623]: Next returned nil segment Mar 18 21:37:22.277443392 (197h26m25.293113004s) [upload=189623]: Waiting for error group managing segments... Mar 18 21:37:22.414377107 (197h26m25.430046707s) [noise=true , address=145.79.227.48:4306 , piece_id=FOUGOBCQYYPKA45WADXNNU4CRYNJDWVFVIFBF6VRKINXKAVXNBTQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.418771029 (197h26m25.434440635s) [noise=true , address=145.79.227.39:4018 , piece_id=DBAKT7XQX5IZVLEXDY63HP4NXGZVF6RTFGTNSCYKNX6EFAWVJTKQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.420950984 (197h26m25.436620574s) [noise=true , address=145.79.227.8:4335 , piece_id=U3ICUMTLPLHULI57IQ3V3H4VVEEAXSMQ4GKVFXZOUYWRYN6O2TYA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.421003817 (197h26m25.436673414s) [noise=true , address=145.79.227.4:4189 , piece_id=NMJ2X5ZX7D3U7KRZZYC2MC4XAAJNPQAZAWHLLBY2HNXSZCKCQCUQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.421987949 (197h26m25.437657549s) [noise=true , address=145.79.227.18:4011 , piece_id=NLH7AVODIX6ICBEYYN5JXGGLWLVOXZVYA3XWEOGUEQCW4HZTHEFQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.422142469 (197h26m25.437812071s) [noise=true , address=145.79.227.26:4657 , piece_id=O6UZJ7OLIA4BJQ6IRCI5ERXO3IURKV3GLJVG476KSPMRVQGY7LYA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.425641685 (197h26m25.441311285s) [noise=true , address=145.79.227.20:4439 , piece_id=3FWVMDZ7DLJV652FESYLUQS6CRTDL4ESGMHWEATRSFTBY2B6LTLA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.426101126 (197h26m25.44177072s) [noise=true , address=145.79.227.3:4149 , piece_id=O4PG4M6HCKSRJZMO6KHNSFCRF5JRD2RWLNOMQGPGN2BS6TN2B2RQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.427039270 (197h26m25.44270887s) [noise=true , address=145.79.227.15:4877 , piece_id=BQJN5RVMLYB7IIX3IFHYVFN3D2A5GYPV5L3Q6FWJ3VKVWH4NZWRA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.427616837 (197h26m25.443286441s) [noise=true , address=145.79.227.46:4601 , piece_id=3SWEFW5XZ3DZLPLKJZE3NBRRJCDZ7F7TDA5UXM7QBJUOWZTVPUPQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.431130437 (197h26m25.446800038s) [noise=true , address=45.91.116.7:4012 , piece_id=2JJXJQZO47ZTD7PFKABUENRESMEJMJNKAYCV6E5PKPAPO3IOKNOA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.431183849 (197h26m25.446853442s) [noise=true , address=145.79.227.33:4874 , piece_id=NLQ737OMRU53WHBROA6TWFT4JU2RWL6AFGAVFFNWLHHEHMHFU5LQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.456250331 (197h26m25.471919937s) [noise=true , address=145.79.227.35:4486 , piece_id=RSLN2AZ2HSBPPU7XTUASMYU2Y2PFZN4YW6ZMPIOJNZTEWKWEQXJA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.456277546 (197h26m25.47194715s) [seg_pos={0 0}, upload=189623]: Satisfied cohort requirements after 13 pieces Mar 18 21:37:22.456632071 (197h26m25.472301676s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.456649964 (197h26m25.47231957s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.456701058 (197h26m25.472370669s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.456714788 (197h26m25.472384393s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.457052179 (197h26m25.47272179s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.457069637 (197h26m25.472739249s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.457128468 (197h26m25.472798071s) [seg_pos={0 0}, upload=189623]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:37:22.457274145 (197h26m25.472943746s) [upload=189623]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:37:22.457281197 (197h26m25.472950797s) [upload=189623]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:37:38.941520305 (197h26m41.957189918s) [upload=189623]: Done uploading segments... Mar 18 21:37:38.941544429 (197h26m41.957214028s) [upload=189623]: Upload finished. err: <nil> Duration: 2.497710011s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 3.906348623s trace: https://jaeger.dev.storj.io/trace/2e6fa05e97e090 FAIL (2026-03-18 21:38:29 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.906460303s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 2.447896471s trace: https://jaeger.dev.storj.io/trace/1fea1be3553256d8 FAIL (2026-03-18 21:38:14 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:37:13.407226914 (197h26m16.422896563s) [upload=189623]: Starting upload Mar 18 21:37:13.407304870 (197h26m16.422974473s) [upload=189623]: Done starting upload Mar 18 21:37:13.407540972 (197h26m16.423210564s) [upload=189623]: Uploading segments... Mar 18 21:37:13.408026524 (197h26m16.423696147s) [upload=189623]: Got next segment. Inline: false Mar 18 21:37:13.408123553 (197h26m16.423793153s) [upload=189623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:37:13.408130704 (197h26m16.423800304s) [upload=189623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:37:21.655023605 (197h26m24.670693205s) [seg_pos={0 0}, upload=189623]: Begin upload segment... Mar 18 21:37:21.655094469 (197h26m24.670764059s) [seg_pos={0 0}, upload=189623]: Done begin upload segment. Mar 18 21:37:21.655129337 (197h26m24.67079894s) [noise=true , address=145.79.227.4:4189 , piece_id=NMJ2X5ZX7D3U7KRZZYC2MC4XAAJNPQAZAWHLLBY2HNXSZCKCQCUQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.656236260 (197h26m24.671905872s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.656865510 (197h26m24.672535113s) [noise=true , address=145.79.227.20:4439 , piece_id=3FWVMDZ7DLJV652FESYLUQS6CRTDL4ESGMHWEATRSFTBY2B6LTLA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.657306588 (197h26m24.672976189s) [noise=true , address=145.79.227.35:4486 , piece_id=RSLN2AZ2HSBPPU7XTUASMYU2Y2PFZN4YW6ZMPIOJNZTEWKWEQXJA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.657690602 (197h26m24.673360202s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.658444416 (197h26m24.674114022s) [noise=true , address=145.79.227.8:4335 , piece_id=U3ICUMTLPLHULI57IQ3V3H4VVEEAXSMQ4GKVFXZOUYWRYN6O2TYA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.659018578 (197h26m24.674688178s) [noise=true , address=145.79.227.15:4877 , piece_id=BQJN5RVMLYB7IIX3IFHYVFN3D2A5GYPV5L3Q6FWJ3VKVWH4NZWRA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.659604237 (197h26m24.675273838s) [noise=true , address=145.79.227.26:4657 , piece_id=O6UZJ7OLIA4BJQ6IRCI5ERXO3IURKV3GLJVG476KSPMRVQGY7LYA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.660277318 (197h26m24.675946921s) [noise=true , address=145.79.227.46:4601 , piece_id=3SWEFW5XZ3DZLPLKJZE3NBRRJCDZ7F7TDA5UXM7QBJUOWZTVPUPQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.660830355 (197h26m24.67649996s) [noise=true , address=45.91.116.7:4012 , piece_id=2JJXJQZO47ZTD7PFKABUENRESMEJMJNKAYCV6E5PKPAPO3IOKNOA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.661333927 (197h26m24.677003527s) [noise=true , address=145.79.227.3:4149 , piece_id=O4PG4M6HCKSRJZMO6KHNSFCRF5JRD2RWLNOMQGPGN2BS6TN2B2RQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.664022598 (197h26m24.679692201s) [noise=true , address=145.79.227.18:4011 , piece_id=NLH7AVODIX6ICBEYYN5JXGGLWLVOXZVYA3XWEOGUEQCW4HZTHEFQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.666068628 (197h26m24.681738234s) [noise=true , address=145.79.227.48:4306 , piece_id=FOUGOBCQYYPKA45WADXNNU4CRYNJDWVFVIFBF6VRKINXKAVXNBTQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.667030758 (197h26m24.682700369s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.671055812 (197h26m24.686725418s) [noise=true , address=145.79.227.39:4018 , piece_id=DBAKT7XQX5IZVLEXDY63HP4NXGZVF6RTFGTNSCYKNX6EFAWVJTKQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.677611108 (197h26m24.693280721s) [noise=true , address=145.79.227.33:4874 , piece_id=NLQ737OMRU53WHBROA6TWFT4JU2RWL6AFGAVFFNWLHHEHMHFU5LQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:22.277424525 (197h26m25.293094127s) [upload=189623]: Next returned nil segment Mar 18 21:37:22.277443392 (197h26m25.293113004s) [upload=189623]: Waiting for error group managing segments... Mar 18 21:37:22.414377107 (197h26m25.430046707s) [noise=true , address=145.79.227.48:4306 , piece_id=FOUGOBCQYYPKA45WADXNNU4CRYNJDWVFVIFBF6VRKINXKAVXNBTQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.418771029 (197h26m25.434440635s) [noise=true , address=145.79.227.39:4018 , piece_id=DBAKT7XQX5IZVLEXDY63HP4NXGZVF6RTFGTNSCYKNX6EFAWVJTKQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.420950984 (197h26m25.436620574s) [noise=true , address=145.79.227.8:4335 , piece_id=U3ICUMTLPLHULI57IQ3V3H4VVEEAXSMQ4GKVFXZOUYWRYN6O2TYA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.421003817 (197h26m25.436673414s) [noise=true , address=145.79.227.4:4189 , piece_id=NMJ2X5ZX7D3U7KRZZYC2MC4XAAJNPQAZAWHLLBY2HNXSZCKCQCUQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.421987949 (197h26m25.437657549s) [noise=true , address=145.79.227.18:4011 , piece_id=NLH7AVODIX6ICBEYYN5JXGGLWLVOXZVYA3XWEOGUEQCW4HZTHEFQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.422142469 (197h26m25.437812071s) [noise=true , address=145.79.227.26:4657 , piece_id=O6UZJ7OLIA4BJQ6IRCI5ERXO3IURKV3GLJVG476KSPMRVQGY7LYA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.425641685 (197h26m25.441311285s) [noise=true , address=145.79.227.20:4439 , piece_id=3FWVMDZ7DLJV652FESYLUQS6CRTDL4ESGMHWEATRSFTBY2B6LTLA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.426101126 (197h26m25.44177072s) [noise=true , address=145.79.227.3:4149 , piece_id=O4PG4M6HCKSRJZMO6KHNSFCRF5JRD2RWLNOMQGPGN2BS6TN2B2RQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.427039270 (197h26m25.44270887s) [noise=true , address=145.79.227.15:4877 , piece_id=BQJN5RVMLYB7IIX3IFHYVFN3D2A5GYPV5L3Q6FWJ3VKVWH4NZWRA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.427616837 (197h26m25.443286441s) [noise=true , address=145.79.227.46:4601 , piece_id=3SWEFW5XZ3DZLPLKJZE3NBRRJCDZ7F7TDA5UXM7QBJUOWZTVPUPQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.431130437 (197h26m25.446800038s) [noise=true , address=45.91.116.7:4012 , piece_id=2JJXJQZO47ZTD7PFKABUENRESMEJMJNKAYCV6E5PKPAPO3IOKNOA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.431183849 (197h26m25.446853442s) [noise=true , address=145.79.227.33:4874 , piece_id=NLQ737OMRU53WHBROA6TWFT4JU2RWL6AFGAVFFNWLHHEHMHFU5LQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.456250331 (197h26m25.471919937s) [noise=true , address=145.79.227.35:4486 , piece_id=RSLN2AZ2HSBPPU7XTUASMYU2Y2PFZN4YW6ZMPIOJNZTEWKWEQXJA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.456277546 (197h26m25.47194715s) [seg_pos={0 0}, upload=189623]: Satisfied cohort requirements after 13 pieces Mar 18 21:37:22.456632071 (197h26m25.472301676s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.456649964 (197h26m25.47231957s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.456701058 (197h26m25.472370669s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.456714788 (197h26m25.472384393s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.457052179 (197h26m25.47272179s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.457069637 (197h26m25.472739249s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.457128468 (197h26m25.472798071s) [seg_pos={0 0}, upload=189623]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:37:22.457274145 (197h26m25.472943746s) [upload=189623]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:37:22.457281197 (197h26m25.472950797s) [upload=189623]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:37:38.941520305 (197h26m41.957189918s) [upload=189623]: Done uploading segments... Mar 18 21:37:38.941544429 (197h26m41.957214028s) [upload=189623]: Upload finished. err: <nil> Duration: 2.447938709s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 20.381875867s trace: https://jaeger.dev.storj.io/trace/1fea1be3553256d8 FAIL (2026-03-18 21:38:07 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:37:13.407226914 (197h26m16.422896563s) [upload=189623]: Starting upload Mar 18 21:37:13.407304870 (197h26m16.422974473s) [upload=189623]: Done starting upload Mar 18 21:37:13.407540972 (197h26m16.423210564s) [upload=189623]: Uploading segments... Mar 18 21:37:13.408026524 (197h26m16.423696147s) [upload=189623]: Got next segment. Inline: false Mar 18 21:37:13.408123553 (197h26m16.423793153s) [upload=189623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:37:13.408130704 (197h26m16.423800304s) [upload=189623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:37:21.655023605 (197h26m24.670693205s) [seg_pos={0 0}, upload=189623]: Begin upload segment... Mar 18 21:37:21.655094469 (197h26m24.670764059s) [seg_pos={0 0}, upload=189623]: Done begin upload segment. Mar 18 21:37:21.655129337 (197h26m24.67079894s) [noise=true , address=145.79.227.4:4189 , piece_id=NMJ2X5ZX7D3U7KRZZYC2MC4XAAJNPQAZAWHLLBY2HNXSZCKCQCUQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.656236260 (197h26m24.671905872s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.656865510 (197h26m24.672535113s) [noise=true , address=145.79.227.20:4439 , piece_id=3FWVMDZ7DLJV652FESYLUQS6CRTDL4ESGMHWEATRSFTBY2B6LTLA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.657306588 (197h26m24.672976189s) [noise=true , address=145.79.227.35:4486 , piece_id=RSLN2AZ2HSBPPU7XTUASMYU2Y2PFZN4YW6ZMPIOJNZTEWKWEQXJA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.657690602 (197h26m24.673360202s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.658444416 (197h26m24.674114022s) [noise=true , address=145.79.227.8:4335 , piece_id=U3ICUMTLPLHULI57IQ3V3H4VVEEAXSMQ4GKVFXZOUYWRYN6O2TYA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.659018578 (197h26m24.674688178s) [noise=true , address=145.79.227.15:4877 , piece_id=BQJN5RVMLYB7IIX3IFHYVFN3D2A5GYPV5L3Q6FWJ3VKVWH4NZWRA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.659604237 (197h26m24.675273838s) [noise=true , address=145.79.227.26:4657 , piece_id=O6UZJ7OLIA4BJQ6IRCI5ERXO3IURKV3GLJVG476KSPMRVQGY7LYA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.660277318 (197h26m24.675946921s) [noise=true , address=145.79.227.46:4601 , piece_id=3SWEFW5XZ3DZLPLKJZE3NBRRJCDZ7F7TDA5UXM7QBJUOWZTVPUPQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.660830355 (197h26m24.67649996s) [noise=true , address=45.91.116.7:4012 , piece_id=2JJXJQZO47ZTD7PFKABUENRESMEJMJNKAYCV6E5PKPAPO3IOKNOA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.661333927 (197h26m24.677003527s) [noise=true , address=145.79.227.3:4149 , piece_id=O4PG4M6HCKSRJZMO6KHNSFCRF5JRD2RWLNOMQGPGN2BS6TN2B2RQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.664022598 (197h26m24.679692201s) [noise=true , address=145.79.227.18:4011 , piece_id=NLH7AVODIX6ICBEYYN5JXGGLWLVOXZVYA3XWEOGUEQCW4HZTHEFQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.666068628 (197h26m24.681738234s) [noise=true , address=145.79.227.48:4306 , piece_id=FOUGOBCQYYPKA45WADXNNU4CRYNJDWVFVIFBF6VRKINXKAVXNBTQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.667030758 (197h26m24.682700369s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.671055812 (197h26m24.686725418s) [noise=true , address=145.79.227.39:4018 , piece_id=DBAKT7XQX5IZVLEXDY63HP4NXGZVF6RTFGTNSCYKNX6EFAWVJTKQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.677611108 (197h26m24.693280721s) [noise=true , address=145.79.227.33:4874 , piece_id=NLQ737OMRU53WHBROA6TWFT4JU2RWL6AFGAVFFNWLHHEHMHFU5LQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:22.277424525 (197h26m25.293094127s) [upload=189623]: Next returned nil segment Mar 18 21:37:22.277443392 (197h26m25.293113004s) [upload=189623]: Waiting for error group managing segments... Mar 18 21:37:22.414377107 (197h26m25.430046707s) [noise=true , address=145.79.227.48:4306 , piece_id=FOUGOBCQYYPKA45WADXNNU4CRYNJDWVFVIFBF6VRKINXKAVXNBTQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.418771029 (197h26m25.434440635s) [noise=true , address=145.79.227.39:4018 , piece_id=DBAKT7XQX5IZVLEXDY63HP4NXGZVF6RTFGTNSCYKNX6EFAWVJTKQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.420950984 (197h26m25.436620574s) [noise=true , address=145.79.227.8:4335 , piece_id=U3ICUMTLPLHULI57IQ3V3H4VVEEAXSMQ4GKVFXZOUYWRYN6O2TYA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.421003817 (197h26m25.436673414s) [noise=true , address=145.79.227.4:4189 , piece_id=NMJ2X5ZX7D3U7KRZZYC2MC4XAAJNPQAZAWHLLBY2HNXSZCKCQCUQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.421987949 (197h26m25.437657549s) [noise=true , address=145.79.227.18:4011 , piece_id=NLH7AVODIX6ICBEYYN5JXGGLWLVOXZVYA3XWEOGUEQCW4HZTHEFQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.422142469 (197h26m25.437812071s) [noise=true , address=145.79.227.26:4657 , piece_id=O6UZJ7OLIA4BJQ6IRCI5ERXO3IURKV3GLJVG476KSPMRVQGY7LYA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.425641685 (197h26m25.441311285s) [noise=true , address=145.79.227.20:4439 , piece_id=3FWVMDZ7DLJV652FESYLUQS6CRTDL4ESGMHWEATRSFTBY2B6LTLA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.426101126 (197h26m25.44177072s) [noise=true , address=145.79.227.3:4149 , piece_id=O4PG4M6HCKSRJZMO6KHNSFCRF5JRD2RWLNOMQGPGN2BS6TN2B2RQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.427039270 (197h26m25.44270887s) [noise=true , address=145.79.227.15:4877 , piece_id=BQJN5RVMLYB7IIX3IFHYVFN3D2A5GYPV5L3Q6FWJ3VKVWH4NZWRA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.427616837 (197h26m25.443286441s) [noise=true , address=145.79.227.46:4601 , piece_id=3SWEFW5XZ3DZLPLKJZE3NBRRJCDZ7F7TDA5UXM7QBJUOWZTVPUPQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.431130437 (197h26m25.446800038s) [noise=true , address=45.91.116.7:4012 , piece_id=2JJXJQZO47ZTD7PFKABUENRESMEJMJNKAYCV6E5PKPAPO3IOKNOA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.431183849 (197h26m25.446853442s) [noise=true , address=145.79.227.33:4874 , piece_id=NLQ737OMRU53WHBROA6TWFT4JU2RWL6AFGAVFFNWLHHEHMHFU5LQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.456250331 (197h26m25.471919937s) [noise=true , address=145.79.227.35:4486 , piece_id=RSLN2AZ2HSBPPU7XTUASMYU2Y2PFZN4YW6ZMPIOJNZTEWKWEQXJA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.456277546 (197h26m25.47194715s) [seg_pos={0 0}, upload=189623]: Satisfied cohort requirements after 13 pieces Mar 18 21:37:22.456632071 (197h26m25.472301676s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.456649964 (197h26m25.47231957s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.456701058 (197h26m25.472370669s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.456714788 (197h26m25.472384393s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.457052179 (197h26m25.47272179s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.457069637 (197h26m25.472739249s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.457128468 (197h26m25.472798071s) [seg_pos={0 0}, upload=189623]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:37:22.457274145 (197h26m25.472943746s) [upload=189623]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:37:22.457281197 (197h26m25.472950797s) [upload=189623]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:37:38.941520305 (197h26m41.957189918s) [upload=189623]: Done uploading segments... Mar 18 21:37:38.941544429 (197h26m41.957214028s) [upload=189623]: Upload finished. err: <nil> Duration: 20.381903775s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 29.459899286s trace: https://jaeger.dev.storj.io/trace/2e6fa05e97e090 FAIL (2026-03-18 21:38:07 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 29.4599173s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 17.551497613s trace: https://jaeger.dev.storj.io/trace/94bb3ca45ba55fa FAIL (2026-03-18 21:38:07 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 17.551539569s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 29.716997592s trace: https://jaeger.dev.storj.io/trace/94bb3ca45ba55fa FAIL (2026-03-18 21:37:48 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 29.717020809s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.003252209s trace: https://jaeger.dev.storj.io/trace/1fea1be3553256d8 FAIL (2026-03-18 21:37:43 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:37:13.407226914 (197h26m16.422896563s) [upload=189623]: Starting upload Mar 18 21:37:13.407304870 (197h26m16.422974473s) [upload=189623]: Done starting upload Mar 18 21:37:13.407540972 (197h26m16.423210564s) [upload=189623]: Uploading segments... Mar 18 21:37:13.408026524 (197h26m16.423696147s) [upload=189623]: Got next segment. Inline: false Mar 18 21:37:13.408123553 (197h26m16.423793153s) [upload=189623]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:37:13.408130704 (197h26m16.423800304s) [upload=189623]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:37:21.655023605 (197h26m24.670693205s) [seg_pos={0 0}, upload=189623]: Begin upload segment... Mar 18 21:37:21.655094469 (197h26m24.670764059s) [seg_pos={0 0}, upload=189623]: Done begin upload segment. Mar 18 21:37:21.655129337 (197h26m24.67079894s) [noise=true , address=145.79.227.4:4189 , piece_id=NMJ2X5ZX7D3U7KRZZYC2MC4XAAJNPQAZAWHLLBY2HNXSZCKCQCUQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.656236260 (197h26m24.671905872s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.656865510 (197h26m24.672535113s) [noise=true , address=145.79.227.20:4439 , piece_id=3FWVMDZ7DLJV652FESYLUQS6CRTDL4ESGMHWEATRSFTBY2B6LTLA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.657306588 (197h26m24.672976189s) [noise=true , address=145.79.227.35:4486 , piece_id=RSLN2AZ2HSBPPU7XTUASMYU2Y2PFZN4YW6ZMPIOJNZTEWKWEQXJA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.657690602 (197h26m24.673360202s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.658444416 (197h26m24.674114022s) [noise=true , address=145.79.227.8:4335 , piece_id=U3ICUMTLPLHULI57IQ3V3H4VVEEAXSMQ4GKVFXZOUYWRYN6O2TYA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.659018578 (197h26m24.674688178s) [noise=true , address=145.79.227.15:4877 , piece_id=BQJN5RVMLYB7IIX3IFHYVFN3D2A5GYPV5L3Q6FWJ3VKVWH4NZWRA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.659604237 (197h26m24.675273838s) [noise=true , address=145.79.227.26:4657 , piece_id=O6UZJ7OLIA4BJQ6IRCI5ERXO3IURKV3GLJVG476KSPMRVQGY7LYA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.660277318 (197h26m24.675946921s) [noise=true , address=145.79.227.46:4601 , piece_id=3SWEFW5XZ3DZLPLKJZE3NBRRJCDZ7F7TDA5UXM7QBJUOWZTVPUPQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.660830355 (197h26m24.67649996s) [noise=true , address=45.91.116.7:4012 , piece_id=2JJXJQZO47ZTD7PFKABUENRESMEJMJNKAYCV6E5PKPAPO3IOKNOA, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.661333927 (197h26m24.677003527s) [noise=true , address=145.79.227.3:4149 , piece_id=O4PG4M6HCKSRJZMO6KHNSFCRF5JRD2RWLNOMQGPGN2BS6TN2B2RQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.664022598 (197h26m24.679692201s) [noise=true , address=145.79.227.18:4011 , piece_id=NLH7AVODIX6ICBEYYN5JXGGLWLVOXZVYA3XWEOGUEQCW4HZTHEFQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.666068628 (197h26m24.681738234s) [noise=true , address=145.79.227.48:4306 , piece_id=FOUGOBCQYYPKA45WADXNNU4CRYNJDWVFVIFBF6VRKINXKAVXNBTQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.667030758 (197h26m24.682700369s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.671055812 (197h26m24.686725418s) [noise=true , address=145.79.227.39:4018 , piece_id=DBAKT7XQX5IZVLEXDY63HP4NXGZVF6RTFGTNSCYKNX6EFAWVJTKQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:21.677611108 (197h26m24.693280721s) [noise=true , address=145.79.227.33:4874 , piece_id=NLQ737OMRU53WHBROA6TWFT4JU2RWL6AFGAVFFNWLHHEHMHFU5LQ, seg_pos={0 0}, upload=189623]: Uploading piece... Mar 18 21:37:22.277424525 (197h26m25.293094127s) [upload=189623]: Next returned nil segment Mar 18 21:37:22.277443392 (197h26m25.293113004s) [upload=189623]: Waiting for error group managing segments... Mar 18 21:37:22.414377107 (197h26m25.430046707s) [noise=true , address=145.79.227.48:4306 , piece_id=FOUGOBCQYYPKA45WADXNNU4CRYNJDWVFVIFBF6VRKINXKAVXNBTQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.418771029 (197h26m25.434440635s) [noise=true , address=145.79.227.39:4018 , piece_id=DBAKT7XQX5IZVLEXDY63HP4NXGZVF6RTFGTNSCYKNX6EFAWVJTKQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.420950984 (197h26m25.436620574s) [noise=true , address=145.79.227.8:4335 , piece_id=U3ICUMTLPLHULI57IQ3V3H4VVEEAXSMQ4GKVFXZOUYWRYN6O2TYA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.421003817 (197h26m25.436673414s) [noise=true , address=145.79.227.4:4189 , piece_id=NMJ2X5ZX7D3U7KRZZYC2MC4XAAJNPQAZAWHLLBY2HNXSZCKCQCUQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.421987949 (197h26m25.437657549s) [noise=true , address=145.79.227.18:4011 , piece_id=NLH7AVODIX6ICBEYYN5JXGGLWLVOXZVYA3XWEOGUEQCW4HZTHEFQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.422142469 (197h26m25.437812071s) [noise=true , address=145.79.227.26:4657 , piece_id=O6UZJ7OLIA4BJQ6IRCI5ERXO3IURKV3GLJVG476KSPMRVQGY7LYA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.425641685 (197h26m25.441311285s) [noise=true , address=145.79.227.20:4439 , piece_id=3FWVMDZ7DLJV652FESYLUQS6CRTDL4ESGMHWEATRSFTBY2B6LTLA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.426101126 (197h26m25.44177072s) [noise=true , address=145.79.227.3:4149 , piece_id=O4PG4M6HCKSRJZMO6KHNSFCRF5JRD2RWLNOMQGPGN2BS6TN2B2RQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.427039270 (197h26m25.44270887s) [noise=true , address=145.79.227.15:4877 , piece_id=BQJN5RVMLYB7IIX3IFHYVFN3D2A5GYPV5L3Q6FWJ3VKVWH4NZWRA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.427616837 (197h26m25.443286441s) [noise=true , address=145.79.227.46:4601 , piece_id=3SWEFW5XZ3DZLPLKJZE3NBRRJCDZ7F7TDA5UXM7QBJUOWZTVPUPQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.431130437 (197h26m25.446800038s) [noise=true , address=45.91.116.7:4012 , piece_id=2JJXJQZO47ZTD7PFKABUENRESMEJMJNKAYCV6E5PKPAPO3IOKNOA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.431183849 (197h26m25.446853442s) [noise=true , address=145.79.227.33:4874 , piece_id=NLQ737OMRU53WHBROA6TWFT4JU2RWL6AFGAVFFNWLHHEHMHFU5LQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.456250331 (197h26m25.471919937s) [noise=true , address=145.79.227.35:4486 , piece_id=RSLN2AZ2HSBPPU7XTUASMYU2Y2PFZN4YW6ZMPIOJNZTEWKWEQXJA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: <nil> Mar 18 21:37:22.456277546 (197h26m25.47194715s) [seg_pos={0 0}, upload=189623]: Satisfied cohort requirements after 13 pieces Mar 18 21:37:22.456632071 (197h26m25.472301676s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.456649964 (197h26m25.47231957s) [noise=true , address=77.90.31.14:4663 , piece_id=IWCMYCJXN32SUB2CBFCV5TVGBL6WBISPBAZCLONZ5YDM4CJ46ZRQ, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.456701058 (197h26m25.472370669s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.456714788 (197h26m25.472384393s) [noise=true , address=145.79.227.37:4566 , piece_id=CDLXWWDZWSXAUILEWL34GDA6ZNMYDYCETMQ5DDA474H4K4JGA6NQ, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.457052179 (197h26m25.47272179s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Mar 18 21:37:22.457069637 (197h26m25.472739249s) [noise=true , address=77.90.31.19:4845 , piece_id=53YIOMVMGMQYUKUQBWWDWLUI22FSBQEKBIGZTCPADBO2XRONWTGA, seg_pos={0 0}, upload=189623]: Long tail cancellation due to optimal threshold Mar 18 21:37:22.457128468 (197h26m25.472798071s) [seg_pos={0 0}, upload=189623]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:37:22.457274145 (197h26m25.472943746s) [upload=189623]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:37:22.457281197 (197h26m25.472950797s) [upload=189623]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:37:38.941520305 (197h26m41.957189918s) [upload=189623]: Done uploading segments... Mar 18 21:37:38.941544429 (197h26m41.957214028s) [upload=189623]: Upload finished. err: <nil> Duration: 30.003300831s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [X.XX..........XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXXX] duration: 5.001423339s trace: https://jaeger.dev.storj.io/trace/468bbab930a6aba3 FAIL (2026-03-18 21:37:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001446806s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.005848934s trace: https://jaeger.dev.storj.io/trace/2e6fa05e97e090 FAIL (2026-03-18 21:37:37 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.00587904s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.001565979s trace: https://jaeger.dev.storj.io/trace/94bb3ca45ba55fa FAIL (2026-03-18 21:37:18 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.001591081s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.00198765s trace: https://jaeger.dev.storj.io/trace/5cc28c3f4ec2c38d FAIL (2026-03-18 21:36:55 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Mar 18 21:36:25.661178637 (197h25m28.676848241s) [upload=189613]: Starting upload Mar 18 21:36:25.661255531 (197h25m28.676925137s) [upload=189613]: Done starting upload Mar 18 21:36:25.661445021 (197h25m28.677114689s) [upload=189613]: Uploading segments... Mar 18 21:36:25.661757363 (197h25m28.677426953s) [upload=189613]: Got next segment. Inline: false Mar 18 21:36:25.661811713 (197h25m28.677481313s) [upload=189613]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 21:36:25.661817155 (197h25m28.677486755s) [upload=189613]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 21:36:29.871559002 (197h25m32.887228705s) [seg_pos={0 0}, upload=189613]: Begin upload segment... Mar 18 21:36:29.871630531 (197h25m32.88730014s) [seg_pos={0 0}, upload=189613]: Done begin upload segment. Mar 18 21:36:29.871666194 (197h25m32.887335808s) [noise=true , address=45.91.116.5:4297 , piece_id=GAU5LNOZRGGKCU5DH5NGU6LQKMFMLASDVISVJMBG3G35HL33NMJQ, seg_pos={0 0}, upload=189613]: Uploading piece... Mar 18 21:36:29.871693296 (197h25m32.887362907s) [noise=true , address=145.79.227.20:4443 , piece_id=65TZ3YHW2QHJMRHHBLO3XM3TYEDZWXC5VH77RNX3PVYM6RSEJZSA, seg_pos={0 0}, upload=189613]: Uploading piece... Mar 18 21:36:29.872222878 (197h25m32.887892486s) [noise=true , address=77.90.31.16:4736 , piece_id=TSMFXRIZGXB5FQOQJFAZDUIGO3NZPFBPGXVOE7D6XFOJZYDGE4LA, seg_pos={0 0}, upload=189613]: Uploading piece... Mar 18 21:36:29.872714975 (197h25m32.888384629s) [noise=true , address=145.79.227.32:4833 , piece_id=XLRHUG77ALLXFOSNPNMNABC7ZC5JHAFFVCMMHRZSWVKHKF6G7L6A, seg_pos={0 0}, upload=189613]: Uploading piece... Mar 18 21:36:29.873182998 (197h25m32.888852608s) [noise=true , address=145.79.227.34:4907 , piece_id=RSBGXG3U4NPXAHV6IMYNPNTJVKZHMR4MYTZAUW2SREHUPOLT2UHQ, seg_pos={0 0}, upload=189613]: Uploading piece... Mar 18 21:36:29.873652126 (197h25m32.889321736s) [noise=true , address=45.91.116.4:4937 , piece_id=7Y4U5CJKIWEYRPIGNSLVLCXS6XFVNXVJW3EIRGXJ3SH73KSK6BCA, seg_pos={0 0}, upload=189613]: Uploading piece... Mar 18 21:36:29.874266095 (197h25m32.889935705s) [noise=true , address=145.79.227.47:4265 , piece_id=CUQ4N7VKRTF27Y6YHHK6NIWYIBYYPT52WL6GWEA5L4Y4ZMO5YUWQ, seg_pos={0 0}, upload=189613]: Uploading piece... Mar 18 21:36:29.874849824 (197h25m32.890519478s) [noise=true , address=77.90.31.21:4880 , piece_id=6ADT3RZ23JWYNBJ7FNN5AXXTJP32ZILTA4ZQH4SZS6DZWNU4GLRA, seg_pos={0 0}, upload=189613]: Uploading piece... Mar 18 21:36:29.875602075 (197h25m32.891271693s) [noise=true , address=145.79.227.42:4121 , piece_id=AGVG6ZBOXDK4LHQP7IU7D547EB3CI5V3UI2AC44ZHJUPTPYPDY6Q, seg_pos={0 0}, upload=189613]: Uploading piece... Mar 18 21:36:29.877876705 (197h25m32.893546319s) [noise=true , address=145.79.227.14:4843 , piece_id=X7M5GIS5MXYKDVGTCZUORKYGWJAMB4S6EYMZTI63KBLBJKHHUDNQ, seg_pos={0 0}, upload=189613]: Uploading piece... Mar 18 21:36:29.879769693 (197h25m32.895439306s) [noise=true , address=145.79.227.33:4871 , piece_id=KJETW6RJXOFX45WF4YEGDYCLYUVN4GBB4BU63PWNA6HB434CARYA, seg_pos={0 0}, upload=189613]: Uploading piece... Mar 18 21:36:29.881648124 (197h25m32.897317736s) [noise=true , address=145.79.227.16:4913 , piece_id=O4CGAXFWBC535N4P7L5UG7M5YYT77SQKGAADUPUFMHRPGPQHXC4A, seg_pos={0 0}, upload=189613]: Uploading piece... Mar 18 21:36:29.883576657 (197h25m32.899246302s) [noise=true , address=145.79.227.45:4233 , piece_id=SK4FLO7E5XV2ISE6IFVXKUTU7EHV3YMREB6YZNRQ73RHWFV4ZGHA, seg_pos={0 0}, upload=189613]: Uploading piece... Mar 18 21:36:29.885570220 (197h25m32.901239845s) [noise=true , address=145.79.227.18:4011 , piece_id=KBUJ3DKSK4VDTN3IECHN2VHYBM7V33VI2H7I2KHBYDOYXAMFWGGA, seg_pos={0 0}, upload=189613]: Uploading piece... Mar 18 21:36:29.887425275 (197h25m32.903094887s) [noise=true , address=45.91.116.7:4026 , piece_id=42XRE4IR3POV66S3YRBNDP4NKMSCJZY7LILZYFTISXKOFHKOOYMA, seg_pos={0 0}, upload=189613]: Uploading piece... Mar 18 21:36:29.888009225 (197h25m32.903678838s) [noise=true , address=145.79.227.31:4800 , piece_id=YKRIB7MA6J2KP4D3CCL2XG63WIM7BPJJWMAO4OP2ZVKQE357A4VQ, seg_pos={0 0}, upload=189613]: Uploading piece... Mar 18 21:36:30.503714547 (197h25m33.519384161s) [upload=189613]: Next returned nil segment Mar 18 21:36:30.503733859 (197h25m33.519403473s) [upload=189613]: Waiting for error group managing segments... Mar 18 21:36:30.626836203 (197h25m33.642505808s) [noise=true , address=45.91.116.4:4937 , piece_id=7Y4U5CJKIWEYRPIGNSLVLCXS6XFVNXVJW3EIRGXJ3SH73KSK6BCA, seg_pos={0 0}, upload=189613]: Done uploading piece. err: <nil> Mar 18 21:36:30.628306218 (197h25m33.64397582s) [noise=true , address=145.79.227.34:4907 , piece_id=RSBGXG3U4NPXAHV6IMYNPNTJVKZHMR4MYTZAUW2SREHUPOLT2UHQ, seg_pos={0 0}, upload=189613]: Done uploading piece. err: <nil> Mar 18 21:36:30.643647922 (197h25m33.659317523s) [noise=true , address=145.79.227.16:4913 , piece_id=O4CGAXFWBC535N4P7L5UG7M5YYT77SQKGAADUPUFMHRPGPQHXC4A, seg_pos={0 0}, upload=189613]: Done uploading piece. err: <nil> Mar 18 21:36:30.644517053 (197h25m33.660186655s) [noise=true , address=145.79.227.45:4233 , piece_id=SK4FLO7E5XV2ISE6IFVXKUTU7EHV3YMREB6YZNRQ73RHWFV4ZGHA, seg_pos={0 0}, upload=189613]: Done uploading piece. err: <nil> Mar 18 21:36:30.646400794 (197h25m33.662070397s) [noise=true , address=145.79.227.14:4843 , piece_id=X7M5GIS5MXYKDVGTCZUORKYGWJAMB4S6EYMZTI63KBLBJKHHUDNQ, seg_pos={0 0}, upload=189613]: Done uploading piece. err: <nil> Mar 18 21:36:30.647607134 (197h25m33.663276734s) [noise=true , address=145.79.227.18:4011 , piece_id=KBUJ3DKSK4VDTN3IECHN2VHYBM7V33VI2H7I2KHBYDOYXAMFWGGA, seg_pos={0 0}, upload=189613]: Done uploading piece. err: <nil> Mar 18 21:36:30.648701585 (197h25m33.664371186s) [noise=true , address=145.79.227.32:4833 , piece_id=XLRHUG77ALLXFOSNPNMNABC7ZC5JHAFFVCMMHRZSWVKHKF6G7L6A, seg_pos={0 0}, upload=189613]: Done uploading piece. err: <nil> Mar 18 21:36:30.649891528 (197h25m33.66556113s) [noise=true , address=145.79.227.20:4443 , piece_id=65TZ3YHW2QHJMRHHBLO3XM3TYEDZWXC5VH77RNX3PVYM6RSEJZSA, seg_pos={0 0}, upload=189613]: Done uploading piece. err: <nil> Mar 18 21:36:30.651098798 (197h25m33.666768403s) [noise=true , address=145.79.227.42:4121 , piece_id=AGVG6ZBOXDK4LHQP7IU7D547EB3CI5V3UI2AC44ZHJUPTPYPDY6Q, seg_pos={0 0}, upload=189613]: Done uploading piece. err: <nil> Mar 18 21:36:30.657398228 (197h25m33.673067834s) [noise=true , address=145.79.227.31:4800 , piece_id=YKRIB7MA6J2KP4D3CCL2XG63WIM7BPJJWMAO4OP2ZVKQE357A4VQ, seg_pos={0 0}, upload=189613]: Done uploading piece. err: <nil> Mar 18 21:36:30.658619670 (197h25m33.674289274s) [noise=true , address=145.79.227.33:4871 , piece_id=KJETW6RJXOFX45WF4YEGDYCLYUVN4GBB4BU63PWNA6HB434CARYA, seg_pos={0 0}, upload=189613]: Done uploading piece. err: <nil> Mar 18 21:36:30.659393966 (197h25m33.675063569s) [noise=true , address=45.91.116.7:4026 , piece_id=42XRE4IR3POV66S3YRBNDP4NKMSCJZY7LILZYFTISXKOFHKOOYMA, seg_pos={0 0}, upload=189613]: Done uploading piece. err: <nil> Mar 18 21:36:30.659990900 (197h25m33.6756605s) [noise=true , address=45.91.116.5:4297 , piece_id=GAU5LNOZRGGKCU5DH5NGU6LQKMFMLASDVISVJMBG3G35HL33NMJQ, seg_pos={0 0}, upload=189613]: Done uploading piece. err: <nil> Mar 18 21:36:30.660003523 (197h25m33.675673126s) [seg_pos={0 0}, upload=189613]: Satisfied cohort requirements after 13 pieces Mar 18 21:36:30.660438227 (197h25m33.676107836s) [noise=true , address=77.90.31.21:4880 , piece_id=6ADT3RZ23JWYNBJ7FNN5AXXTJP32ZILTA4ZQH4SZS6DZWNU4GLRA, seg_pos={0 0}, upload=189613]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Mar 18 21:36:30.660448805 (197h25m33.676118408s) [noise=true , address=77.90.31.21:4880 , piece_id=6ADT3RZ23JWYNBJ7FNN5AXXTJP32ZILTA4ZQH4SZS6DZWNU4GLRA, seg_pos={0 0}, upload=189613]: Long tail cancellation due to optimal threshold Mar 18 21:36:30.668179972 (197h25m33.683849572s) [noise=true , address=77.90.31.16:4736 , piece_id=TSMFXRIZGXB5FQOQJFAZDUIGO3NZPFBPGXVOE7D6XFOJZYDGE4LA, seg_pos={0 0}, upload=189613]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Mar 18 21:36:30.668199123 (197h25m33.683868724s) [noise=true , address=77.90.31.16:4736 , piece_id=TSMFXRIZGXB5FQOQJFAZDUIGO3NZPFBPGXVOE7D6XFOJZYDGE4LA, seg_pos={0 0}, upload=189613]: Long tail cancellation due to optimal threshold Mar 18 21:36:30.684837292 (197h25m33.700506897s) [noise=true , address=145.79.227.47:4265 , piece_id=CUQ4N7VKRTF27Y6YHHK6NIWYIBYYPT52WL6GWEA5L4Y4ZMO5YUWQ, seg_pos={0 0}, upload=189613]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: expected piece hash; context canceled; EOF Mar 18 21:36:30.684857338 (197h25m33.700526939s) [noise=true , address=145.79.227.47:4265 , piece_id=CUQ4N7VKRTF27Y6YHHK6NIWYIBYYPT52WL6GWEA5L4Y4ZMO5YUWQ, seg_pos={0 0}, upload=189613]: Long tail cancellation due to optimal threshold Mar 18 21:36:30.684940607 (197h25m33.700610197s) [seg_pos={0 0}, upload=189613]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 21:36:30.685030754 (197h25m33.700700368s) [upload=189613]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 21:36:30.685041091 (197h25m33.700710703s) [upload=189613]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 21:36:43.052045158 (197h25m46.067714768s) [upload=189613]: Done uploading segments... Mar 18 21:36:43.052070094 (197h25m46.067739696s) [upload=189613]: Upload finished. err: <nil> Duration: 30.00201917s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XX.XX..........XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXXX] duration: 5.016916553s trace: https://jaeger.dev.storj.io/trace/232862972182f00b FAIL (2026-03-18 21:36:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.016948855s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002135961s trace: https://jaeger.dev.storj.io/trace/210c81557c2b408d FAIL (2026-03-18 21:36:11 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 30.002166398s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XXX.XX..........XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXXX] duration: 5.001552061s trace: https://jaeger.dev.storj.io/trace/2b195108edee4a5f FAIL (2026-03-18 21:35:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001584371s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XXXX.XX..........XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXXX] duration: 5.009018497s trace: https://jaeger.dev.storj.io/trace/45bc8d934deed9bc FAIL (2026-03-18 21:34:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.009048154s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XXXXX.XX..........XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXXX] duration: 5.001280111s trace: https://jaeger.dev.storj.io/trace/7284b77697a9a933 FAIL (2026-03-18 21:33:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001306292s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XXXXXX.XX..........XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXXX] duration: 5.001320913s trace: https://jaeger.dev.storj.io/trace/2194dfdd7d3fdf33 FAIL (2026-03-18 21:32:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001343221s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XXXXXXX.XX..........XXXXXXXXXXXXXXXXXXXXXXXXXX......XXXXXXXX] duration: 5.001760938s trace: https://jaeger.dev.storj.io/trace/265e09e28f53aca4 FAIL (2026-03-18 21:32:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001794838s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)