Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
2.36244417s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:23:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.362469945s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
2.430295828s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:22:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.430325135s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
2.364437217s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:21:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.364462298s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
2.358161577s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:20:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.358207063s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
2.326199002s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:20:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.32622828s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
2.574499568s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:20:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.574534516s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
2.323833416s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:19:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.323927726s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
2.356671372s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:18:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.35675379s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
2.348286109s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:17:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.348326986s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
2.471707904s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:16:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.471807964s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
2.373043615s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:15:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.373075376s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
2.382897724s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:14:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.382928788s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
2.38084222s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:14:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.380873999s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
2.478073614s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:13:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.478180657s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
2.344428799s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:12:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.344452691s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
2.378022588s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:12:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.378047777s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
2.348586167s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:11:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.348616608s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
2.443906378s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:11:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.44393401s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
2.392534689s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:10:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.392558618s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
2.532953716s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:10:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.533332263s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
2.387577492s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:09:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.387604827s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
2.341789341s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:08:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.341815316s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
2.438578284s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:07:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.43861381s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
2.398483519s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:06:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.398513704s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
2.437204733s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:05:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.43723728s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
2.389465172s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:04:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.389494283s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
2.364795207s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:03:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.364821568s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
2.457601981s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:02:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.457618815s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
2.410943895s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:01:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.410975317s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
2.4050772s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:01:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.405105388s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
2.30808962s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 19:00:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.30812068s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
2.526664995s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 18:59:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.526692428s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
2.332554125s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 18:58:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.33258036s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
2.418275146s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 18:57:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 2.41831649s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
3.285695389s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 18:56:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 3.285713734s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
28.292635421s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 18:56:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 28.292658813s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
27.109856233s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 18:55:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 27.109878999s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
20.214271877s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 18:55:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 20.214321525s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.003515886s trace:
https://jaeger.dev.storj.io/trace/95e62fe014f59b8
FAIL (2026-01-04 18:53:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ) context deadline exceeded
======LOGS======
Jan 4 18:53:29.941240173 (723h25m9.921592822s) [upload=636675]: Starting upload
Jan 4 18:53:29.941330163 (723h25m9.921682803s) [upload=636675]: Done starting upload
Jan 4 18:53:29.941402632 (723h25m9.921755258s) [upload=636675]: Uploading segments...
Jan 4 18:53:29.941698115 (723h25m9.922050745s) [upload=636675]: Got next segment. Inline: false
Jan 4 18:53:29.941755178 (723h25m9.922107828s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 4 18:53:29.941764111 (723h25m9.922116758s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 4 18:53:30.091229729 (723h25m10.07158239s) [seg_pos={0 0}, upload=636675]: Begin upload segment...
Jan 4 18:53:30.091376906 (723h25m10.071729555s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092755120 (723h25m10.07310777s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.092791953 (723h25m10.073144594s) [seg_pos={0 0}, upload=636675]: Done begin upload segment.
Jan 4 18:53:30.092909376 (723h25m10.073262031s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093375979 (723h25m10.073728625s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093636250 (723h25m10.0739889s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.093842246 (723h25m10.074194891s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094318919 (723h25m10.074671559s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094373682 (723h25m10.07472634s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.094746945 (723h25m10.075099599s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095068347 (723h25m10.075421006s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095231398 (723h25m10.075584047s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.095774560 (723h25m10.076127223s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.096676161 (723h25m10.077028813s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.097469917 (723h25m10.077822568s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098125917 (723h25m10.078478583s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.098947355 (723h25m10.079300005s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Uploading piece...
Jan 4 18:53:30.914099868 (723h25m10.894452506s) [upload=636675]: Next returned nil segment
Jan 4 18:53:30.914115302 (723h25m10.894467939s) [upload=636675]: Waiting for error group managing segments...
Jan 4 18:53:31.103847422 (723h25m11.08420006s) [noise=true , address=145.79.227.41:4086 , piece_id=LPFYRLD4MPX225JVH4YPJ2QCMZWRZOBKFKCD7XYVOD2XGR4VO5AA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.105531475 (723h25m11.085884112s) [noise=true , address=145.79.227.9:4366 , piece_id=MZQ6AMKBEFLRV3MZYUMDAD6PTZKSI4H6SJQSRXNCYE3LUDFOCMUQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.107047250 (723h25m11.087399889s) [noise=true , address=145.79.227.16:4911 , piece_id=PKQFIYJMOBL5FWEFTLFLJ4TSUM4YD4WMKLFA2HXZQ6FN5VHOVKZQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.108360246 (723h25m11.088712885s) [noise=true , address=145.79.227.15:4876 , piece_id=3NBVJHL5YXQG4T3GSTHERTJOPC7HSOXK2CR26NKQOOKISLFIJM2Q, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.122365598 (723h25m11.102718234s) [noise=true , address=145.79.227.20:4437 , piece_id=2UD7IVO34FOUGHRNOL7F3FPYFC5HR4J3R3GOWJXD37OES5MK3ADA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.124966518 (723h25m11.10531916s) [noise=true , address=145.79.227.51:4410 , piece_id=RBYHZRTLFSZIRMGSJQWJJ5B66LX3T7LUHQAYI4XTQZKDVM22MCNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.127084535 (723h25m11.107437179s) [noise=true , address=145.79.227.48:4303 , piece_id=BJRB72XZEYD6DL3A4MKMGDQYAYHFSLDPW5MFZU5CIDLNLJLTYXPQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.131295904 (723h25m11.111648543s) [noise=true , address=145.79.227.27:4690 , piece_id=KVTUGAX4TBM3I4O4OHFCU32ECRJYRHGZWSV4OV357NHPDQ3CUKTA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.281454425 (723h25m11.261807053s) [noise=true , address=145.79.227.11:4731 , piece_id=7TZXE53NCZIOGSBNMUHLI3LNHWJ2ZPV73TBLUROFSNWB4SP7CZMA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.565845406 (723h25m11.546198041s) [noise=true , address=45.91.116.7:4009 , piece_id=CCMH22EEUH4SLKGGQQPQBMITRHQGYEOXWWCKNXGYMR3CD52A5ROA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.613517635 (723h25m11.593870276s) [noise=true , address=45.91.116.14:4190 , piece_id=SAFLCICAWJLRYWIHWSD6NLQ6JRJ3VPKGN5WUTZ6LDYWXOJPGWKHA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.700710462 (723h25m11.681063101s) [noise=true , address=77.90.31.19:4844 , piece_id=MRCTM7GYNDCQWVO7EK77SQM6H72NBB3PGAUAOOL22PEMHPX6MAEQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744241498 (723h25m11.724594135s) [noise=true , address=77.90.31.13:4626 , piece_id=UNK3QRXFJCWWM2KQDMIOMGSG6WAYY4ODUAZDE3X33XA7A6SVWFJA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: <nil>
Jan 4 18:53:31.744265179 (723h25m11.72461782s) [seg_pos={0 0}, upload=636675]: Segment reached optimal threshold of 13 pieces.
Jan 4 18:53:31.744659538 (723h25m11.725012183s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Jan 4 18:53:31.744672090 (723h25m11.725024735s) [noise=true , address=145.79.227.34:4907 , piece_id=VM4UPMY7LJDVTPF5L54EBLJD2JRRIFJ6T6E2QT4PGUVWXLZFPMNQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745277513 (723h25m11.725630155s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: rpc: dial tcp 45.91.116.12:4225: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745289738 (723h25m11.725642377s) [noise=true , address=45.91.116.12:4225 , piece_id=ESMDL73K4KCUAETGC3XVO5DTBQWPB2J45FH6PLIML7BFS7LZ7VLA, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745473404 (723h25m11.725826047s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3): protocol: read tcp 139.59.144.22:37350->145.79.227.28:4727: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 4 18:53:31.745481225 (723h25m11.725833861s) [noise=true , address=145.79.227.28:4727 , piece_id=I6RDKKQ6QM3NVIUGK3SEKKDTSYR5CPBE4LEWD2HTENOOF3FBZ5FQ, seg_pos={0 0}, upload=636675]: Long tail cancellation due to optimal threshold
Jan 4 18:53:31.745498207 (723h25m11.725850834s) [seg_pos={0 0}, upload=636675]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 4 18:53:31.745591236 (723h25m11.725943872s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 4 18:53:31.745596326 (723h25m11.725948953s) [upload=636675]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 4 18:53:31.930454158 (723h25m11.910806803s) [upload=636675]: Done uploading segments...
Jan 4 18:53:31.930480173 (723h25m11.91083281s) [upload=636675]: Upload finished. err: <nil>
Duration: 30.003559171s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [X...........................................................] duration:
1.948457909s trace:
https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf
FAIL (2026-01-04 06:53:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.948502367s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.X..........................................................] duration:
1.711139114s trace:
https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf
FAIL (2026-01-04 06:52:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.71116607s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..X.........................................................] duration:
1.663819912s trace:
https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf
FAIL (2026-01-04 06:52:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.663856737s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...X........................................................] duration:
2.213769051s trace:
https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf
FAIL (2026-01-04 06:51:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.213797486s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....X.......................................................] duration:
1.580999023s trace:
https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf
FAIL (2026-01-04 06:50:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.581028876s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....X......................................................] duration:
1.645957929s trace:
https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf
FAIL (2026-01-04 06:50:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.645991295s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......X.....................................................] duration:
1.481541849s trace:
https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf
FAIL (2026-01-04 06:49:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.481647592s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......X....................................................] duration:
1.583025237s trace:
https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf
FAIL (2026-01-04 06:49:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.583054283s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........X...................................................] duration:
1.497923333s trace:
https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf
FAIL (2026-01-04 06:48:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.497952308s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........X..................................................] duration:
1.526677169s trace:
https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf
FAIL (2026-01-04 06:47:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.526703776s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........X.................................................] duration:
1.485317059s trace:
https://jaeger.dev.storj.io/trace/49ca8d6d10364cbf
FAIL (2026-01-04 06:46:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.485362744s
core
uplink
us-select-1
us1-placement-20
|