Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.36244417s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:23:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.362469945s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.430295828s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:22:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.430325135s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.364437217s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:21:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.364462298s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.358161577s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:20:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.358207063s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.326199002s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:20:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.32622828s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.574499568s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:20:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.574534516s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.323833416s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:19:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.323927726s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.356671372s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:18:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.35675379s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.348286109s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:17:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.348326986s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.471707904s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:16:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.471807964s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.373043615s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:15:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.373075376s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.382897724s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:14:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.382928788s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.38084222s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:14:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.380873999s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.478073614s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:13:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.478180657s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.344428799s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:12:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.344452691s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.378022588s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:12:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.378047777s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.348586167s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:11:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.348616608s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.443906378s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:11:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.44393401s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.392534689s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:10:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.392558618s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.532953716s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:10:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.533332263s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.387577492s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:09:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.387604827s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.341789341s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:08:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.341815316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.438578284s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:07:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.43861381s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.398483519s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:06:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.398513704s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.437204733s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:05:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.43723728s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.389465172s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:04:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.389494283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.364795207s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:03:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.364821568s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.457601981s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:02:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.457618815s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.410943895s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:01:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.410975317s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.4050772s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:01:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.405105388s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.30808962s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 19:00:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.30812068s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.526664995s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 18:59:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.526692428s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.332554125s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 18:58:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.33258036s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.418275146s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 18:57:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 2.41831649s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 3.285695389s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 18:56:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 3.285713734s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 28.292635421s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 18:56:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 28.292658813s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 27.109856233s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 18:55:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 27.109878999s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 20.214271877s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 18:55:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 20.214321525s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.003515886s trace: https://jaeger.dev.storj.io/trace/95e62fe014f59b8 FAIL (2026-01-04 18:53:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded ======LOGS====== Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments... Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment... Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment. Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece... Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments... Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil> Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces. Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments... Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil> Duration: 30.003559171s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 1.948457909s trace: https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf FAIL (2026-01-04 06:53:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.948502367s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 1.711139114s trace: https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf FAIL (2026-01-04 06:52:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.71116607s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 1.663819912s trace: https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf FAIL (2026-01-04 06:52:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.663856737s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 2.213769051s trace: https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf FAIL (2026-01-04 06:51:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.213797486s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 1.580999023s trace: https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf FAIL (2026-01-04 06:50:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.581028876s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 1.645957929s trace: https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf FAIL (2026-01-04 06:50:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.645991295s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 1.481541849s trace: https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf FAIL (2026-01-04 06:49:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.481647592s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 1.583025237s trace: https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf FAIL (2026-01-04 06:49:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.583054283s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 1.497923333s trace: https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf FAIL (2026-01-04 06:48:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.497952308s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 1.526677169s trace: https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf FAIL (2026-01-04 06:47:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.526703776s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 1.485317059s trace: https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf FAIL (2026-01-04 06:46:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.485362744s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)