Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [......XX.X.X......X.X.XXXXX.......XX....................X..X] duration:
30.001245564s
FAIL (2025-07-09 01:31:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001375311s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
2.272149423s
FAIL (2025-07-09 01:31:34 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
cessful: 13 optimal: 13 errs: <nil>
Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments...
Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil>
Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload
Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload
Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments...
Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false
Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment...
Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment.
Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment
Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments...
Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection
Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection
Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection
Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments...
Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.272177585s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.......XX.X.X......X.X.XXXXX.......XX....................X..] duration:
23.323356512s
FAIL (2025-07-09 01:31:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 23.323378512s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
24 out of 60 is failed (threshold=3) [X..XX.X...........XXXXX...X.........X..X...XXX..XXXX.XXXXX..] duration:
49.768570271s
FAIL (2025-07-09 01:31:13 UTC)
orbiter warning: context deadline exceeded
Duration: 49.768594006s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [........XX.X.X......X.X.XXXXX.......XX....................X.] duration:
24.969095449s
FAIL (2025-07-09 01:30:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 24.96912314s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
2.41797237s
FAIL (2025-07-09 01:30:32 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
cessful: 13 optimal: 13 errs: <nil>
Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments...
Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil>
Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload
Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload
Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments...
Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false
Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment...
Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment.
Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment
Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments...
Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection
Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection
Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection
Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments...
Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.418000834s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
24 out of 60 is failed (threshold=3) [.X..XX.X...........XXXXX...X.........X..X...XXX..XXXX.XXXXX.] duration:
44.36297404s
FAIL (2025-07-09 01:30:23 UTC)
orbiter warning: context deadline exceeded
Duration: 44.362988877s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.........XX.X.X......X.X.XXXXX.......XX....................X] duration:
30.000657752s
FAIL (2025-07-09 01:30:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000703444s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
2.464090689s
FAIL (2025-07-09 01:30:12 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
cessful: 13 optimal: 13 errs: <nil>
Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments...
Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil>
Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload
Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload
Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments...
Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false
Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment...
Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment.
Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment
Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments...
Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection
Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection
Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection
Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments...
Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.464120553s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
24 out of 60 is failed (threshold=3) [..X..XX.X...........XXXXX...X.........X..X...XXX..XXXX.XXXXX] duration:
1m0.000751728s
FAIL (2025-07-09 01:29:38 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.000826289s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
9.953107224s
FAIL (2025-07-09 01:29:18 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
cessful: 13 optimal: 13 errs: <nil>
Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments...
Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil>
Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload
Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload
Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments...
Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false
Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment...
Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment.
Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment
Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments...
Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection
Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection
Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection
Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments...
Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 9.953139441s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [..........XX.X.X......X.X.XXXXX.......XX....................] duration:
2.618774036s
FAIL (2025-07-09 01:28:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.61879544s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [...........XX.X.X......X.X.XXXXX.......XX...................] duration:
29.639296953s
FAIL (2025-07-09 01:28:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 29.639319863s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
24 out of 60 is failed (threshold=3) [X..X..XX.X...........XXXXX...X.........X..X...XXX..XXXX.XXXX] duration:
1m0.001357893s
FAIL (2025-07-09 01:28:38 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001374292s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
2.38037232s
FAIL (2025-07-09 01:28:01 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
cessful: 13 optimal: 13 errs: <nil>
Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments...
Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil>
Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload
Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload
Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments...
Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false
Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment...
Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment.
Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment
Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments...
Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection
Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection
Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection
Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments...
Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.380410269s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
23 out of 60 is failed (threshold=3) [.X..X..XX.X...........XXXXX...X.........X..X...XXX..XXXX.XXX] duration:
1m0.003677144s
FAIL (2025-07-09 01:27:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.003688965s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [............XX.X.X......X.X.XXXXX.......XX..................] duration:
20.694827247s
FAIL (2025-07-09 01:27:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 20.694844662s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
2.751661633s
FAIL (2025-07-09 01:27:14 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
cessful: 13 optimal: 13 errs: <nil>
Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments...
Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil>
Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload
Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload
Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments...
Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false
Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment...
Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment.
Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment
Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments...
Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection
Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection
Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection
Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments...
Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.751694017s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.............XX.X.X......X.X.XXXXX.......XX.................] duration:
25.149540716s
FAIL (2025-07-09 01:27:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 25.14971145s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
22 out of 60 is failed (threshold=3) [..X..X..XX.X...........XXXXX...X.........X..X...XXX..XXXX.XX] duration:
1m0.000873375s
FAIL (2025-07-09 01:26:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000894904s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
2.272486946s
FAIL (2025-07-09 01:26:22 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
cessful: 13 optimal: 13 errs: <nil>
Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments...
Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil>
Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload
Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload
Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments...
Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false
Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment...
Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment.
Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment
Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments...
Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection
Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection
Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection
Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments...
Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.272515581s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
2.349112656s
FAIL (2025-07-09 01:26:06 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
cessful: 13 optimal: 13 errs: <nil>
Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments...
Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil>
Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload
Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload
Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments...
Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false
Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment...
Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment.
Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment
Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments...
Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection
Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection
Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection
Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments...
Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.349198298s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [X.............XX.X.X......X.X.XXXXX.......XX................] duration:
2.751783307s
FAIL (2025-07-09 01:25:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.751807654s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
22 out of 60 is failed (threshold=3) [X..X..X..XX.X...........XXXXX...X.........X..X...XXX..XXXX.X] duration:
1m0.001160708s
FAIL (2025-07-09 01:25:35 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001198775s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.X.............XX.X.X......X.X.XXXXX.......XX...............] duration:
2.804845785s
FAIL (2025-07-09 01:25:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.804866637s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [..X.............XX.X.X......X.X.XXXXX.......XX..............] duration:
22.493246063s
FAIL (2025-07-09 01:25:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 22.493278193s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
2.321456291s
FAIL (2025-07-09 01:25:12 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
cessful: 13 optimal: 13 errs: <nil>
Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments...
Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil>
Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload
Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload
Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments...
Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false
Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment...
Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment.
Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment
Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments...
Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection
Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection
Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection
Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments...
Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.32148551s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
21 out of 60 is failed (threshold=3) [.X..X..X..XX.X...........XXXXX...X.........X..X...XXX..XXXX.] duration:
1m0.001459032s
FAIL (2025-07-09 01:24:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001482962s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
2.298367912s
FAIL (2025-07-09 01:24:20 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
cessful: 13 optimal: 13 errs: <nil>
Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments...
Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil>
Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload
Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload
Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments...
Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false
Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment...
Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment.
Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment
Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments...
Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection
Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection
Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection
Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments...
Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.298401676s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [...X.............XX.X.X......X.X.XXXXX.......XX.............] duration:
23.081564946s
FAIL (2025-07-09 01:24:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 23.081590676s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
2.322287153s
FAIL (2025-07-09 01:24:05 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
cessful: 13 optimal: 13 errs: <nil>
Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments...
Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil>
Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload
Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload
Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments...
Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false
Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment...
Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment.
Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment
Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments...
Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection
Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection
Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection
Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments...
Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.322337009s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
22 out of 60 is failed (threshold=3) [X.X..X..X..XX.X...........XXXXX...X.........X..X...XXX..XXXX] duration:
1m0.001520784s
FAIL (2025-07-09 01:23:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001563158s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [....X.............XX.X.X......X.X.XXXXX.......XX............] duration:
21.560092102s
FAIL (2025-07-09 01:23:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 21.56012241s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
2.265157941s
FAIL (2025-07-09 01:22:56 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
cessful: 13 optimal: 13 errs: <nil>
Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments...
Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil>
Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload
Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload
Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments...
Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false
Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment...
Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment.
Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment
Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments...
Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection
Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection
Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection
Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments...
Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.265205495s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
22 out of 60 is failed (threshold=3) [XX.X..X..X..XX.X...........XXXXX...X.........X..X...XXX..XXX] duration:
1m0.000655259s
FAIL (2025-07-09 01:22:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000667666s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [.....X.............XX.X.X......X.X.XXXXX.......XX...........] duration:
25.56676873s
FAIL (2025-07-09 01:22:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 25.566797443s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
35.317503311s
FAIL (2025-07-09 01:21:57 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
3980 (375h26m3.9562776s) [upload=300223]: Starting upload
Jul 9 00:37:04.072968784 (375h26m3.956382412s) [upload=300223]: Done starting upload
Jul 9 00:37:04.073087735 (375h26m3.956501365s) [upload=300223]: Uploading segments...
Jul 9 00:37:04.073377778 (375h26m3.956791396s) [upload=300223]: Got next segment. Inline: false
Jul 9 00:37:04.073430757 (375h26m3.956844388s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:37:04.073437151 (375h26m3.956850769s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:37:04.304692457 (375h26m4.18810608s) [seg_pos={0 0}, upload=300223]: Begin upload segment...
Jul 9 00:37:04.304761975 (375h26m4.188175587s) [seg_pos={0 0}, upload=300223]: Done begin upload segment.
Jul 9 00:37:04.304833908 (375h26m4.18824754s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305365539 (375h26m4.18877917s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305738725 (375h26m4.189152359s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305907885 (375h26m4.189321515s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306301693 (375h26m4.189715324s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306679055 (375h26m4.190092689s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307048455 (375h26m4.190462086s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307412956 (375h26m4.19082659s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.308062976 (375h26m4.191476609s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.309126547 (375h26m4.192540184s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.310676166 (375h26m4.194089804s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311146138 (375h26m4.19455977s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311543981 (375h26m4.194957595s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311932868 (375h26m4.195346498s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312324028 (375h26m4.195737661s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312707561 (375h26m4.196121192s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313124992 (375h26m4.19653862s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313485189 (375h26m4.196898821s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.893020764 (375h26m4.776434382s) [upload=300223]: Next returned nil segment
Jul 9 00:37:04.893957299 (375h26m4.777370932s) [upload=300223]: Waiting for error group managing segments...
Jul 9 00:37:05.037354849 (375h26m4.920768466s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.038444983 (375h26m4.921858597s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.039356590 (375h26m4.922770207s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.047559125 (375h26m4.930972742s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.053248590 (375h26m4.93666221s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.054452827 (375h26m4.937866447s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.889326616 (375h26m5.772740232s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:29.628225928 (375h26m29.511639541s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:30.256578837 (375h26m30.139992453s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:32.331498448 (375h26m32.214912065s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:39.883025841 (375h26m39.766439475s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:51.499841545 (375h26m51.383255165s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:55.982236621 (375h26m55.86565024s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607812826 (375h27m3.49122645s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607853389 (375h27m3.491267022s) [seg_pos={0 0}, upload=300223]: Segment reached optimal threshold of 14 pieces.
Jul 9 00:38:03.608006922 (375h27m3.491420552s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608045600 (375h27m3.491459233s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608087309 (375h27m3.491500937s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RY6XGL5MJYc2idL6Np7DS3U9pCQGDhuEEqguY8AgjasjE4hiZ): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608191697 (375h27m3.491605324s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608213412 (375h27m3.49162703s) [seg_pos={0 0}, upload=300223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 00:38:03.608310462 (375h27m3.49172408s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 00:38:03.608316810 (375h27m3.491730438s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:03.761052568 (375h27m3.6444662s) [upload=300223]: Done uploading segments...
Jul 9 00:38:03.761076757 (375h27m3.644490378s) [upload=300223]: Upload finished. err: <nil>
Duration: 35.317541799s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
2.332197281s
FAIL (2025-07-09 01:21:54 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
cessful: 13 optimal: 13 errs: <nil>
Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments...
Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil>
Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload
Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload
Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments...
Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false
Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment...
Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment.
Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment
Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments...
Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection
Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection
Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection
Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments...
Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.332233226s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [......X.............XX.X.X......X.X.XXXXX.......XX..........] duration:
5.492012424s
FAIL (2025-07-09 01:21:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.492036481s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
21 out of 60 is failed (threshold=3) [.XX.X..X..X..XX.X...........XXXXX...X.........X..X...XXX..XX] duration:
1m0.001068933s
FAIL (2025-07-09 01:21:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001098568s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [X...............................................X...........] duration:
2.625476234s
FAIL (2025-07-09 01:21:19 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
cessful: 13 optimal: 13 errs: <nil>
Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments...
Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil>
Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload
Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload
Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments...
Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false
Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment...
Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment.
Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment
Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments...
Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection
Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection
Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection
Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments...
Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.62551466s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
20.015440841s
FAIL (2025-07-09 01:21:11 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
3980 (375h26m3.9562776s) [upload=300223]: Starting upload
Jul 9 00:37:04.072968784 (375h26m3.956382412s) [upload=300223]: Done starting upload
Jul 9 00:37:04.073087735 (375h26m3.956501365s) [upload=300223]: Uploading segments...
Jul 9 00:37:04.073377778 (375h26m3.956791396s) [upload=300223]: Got next segment. Inline: false
Jul 9 00:37:04.073430757 (375h26m3.956844388s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:37:04.073437151 (375h26m3.956850769s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:37:04.304692457 (375h26m4.18810608s) [seg_pos={0 0}, upload=300223]: Begin upload segment...
Jul 9 00:37:04.304761975 (375h26m4.188175587s) [seg_pos={0 0}, upload=300223]: Done begin upload segment.
Jul 9 00:37:04.304833908 (375h26m4.18824754s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305365539 (375h26m4.18877917s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305738725 (375h26m4.189152359s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305907885 (375h26m4.189321515s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306301693 (375h26m4.189715324s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306679055 (375h26m4.190092689s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307048455 (375h26m4.190462086s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307412956 (375h26m4.19082659s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.308062976 (375h26m4.191476609s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.309126547 (375h26m4.192540184s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.310676166 (375h26m4.194089804s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311146138 (375h26m4.19455977s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311543981 (375h26m4.194957595s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311932868 (375h26m4.195346498s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312324028 (375h26m4.195737661s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312707561 (375h26m4.196121192s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313124992 (375h26m4.19653862s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313485189 (375h26m4.196898821s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.893020764 (375h26m4.776434382s) [upload=300223]: Next returned nil segment
Jul 9 00:37:04.893957299 (375h26m4.777370932s) [upload=300223]: Waiting for error group managing segments...
Jul 9 00:37:05.037354849 (375h26m4.920768466s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.038444983 (375h26m4.921858597s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.039356590 (375h26m4.922770207s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.047559125 (375h26m4.930972742s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.053248590 (375h26m4.93666221s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.054452827 (375h26m4.937866447s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.889326616 (375h26m5.772740232s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:29.628225928 (375h26m29.511639541s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:30.256578837 (375h26m30.139992453s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:32.331498448 (375h26m32.214912065s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:39.883025841 (375h26m39.766439475s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:51.499841545 (375h26m51.383255165s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:55.982236621 (375h26m55.86565024s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607812826 (375h27m3.49122645s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607853389 (375h27m3.491267022s) [seg_pos={0 0}, upload=300223]: Segment reached optimal threshold of 14 pieces.
Jul 9 00:38:03.608006922 (375h27m3.491420552s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608045600 (375h27m3.491459233s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608087309 (375h27m3.491500937s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RY6XGL5MJYc2idL6Np7DS3U9pCQGDhuEEqguY8AgjasjE4hiZ): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608191697 (375h27m3.491605324s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608213412 (375h27m3.49162703s) [seg_pos={0 0}, upload=300223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 00:38:03.608310462 (375h27m3.49172408s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 00:38:03.608316810 (375h27m3.491730438s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:03.761052568 (375h27m3.6444662s) [upload=300223]: Done uploading segments...
Jul 9 00:38:03.761076757 (375h27m3.644490378s) [upload=300223]: Upload finished. err: <nil>
Duration: 20.015460621s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X...............................................X..........] duration:
2.327605724s
FAIL (2025-07-09 01:20:59 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
cessful: 13 optimal: 13 errs: <nil>
Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments...
Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil>
Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload
Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload
Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments...
Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false
Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment...
Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment.
Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment
Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments...
Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection
Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection
Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection
Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments...
Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.32763772s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [X......X.............XX.X.X......X.X.XXXXX.......XX.........] duration:
27.174325233s
FAIL (2025-07-09 01:20:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 27.174350873s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
36.795432942s
FAIL (2025-07-09 01:20:51 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
3980 (375h26m3.9562776s) [upload=300223]: Starting upload
Jul 9 00:37:04.072968784 (375h26m3.956382412s) [upload=300223]: Done starting upload
Jul 9 00:37:04.073087735 (375h26m3.956501365s) [upload=300223]: Uploading segments...
Jul 9 00:37:04.073377778 (375h26m3.956791396s) [upload=300223]: Got next segment. Inline: false
Jul 9 00:37:04.073430757 (375h26m3.956844388s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:37:04.073437151 (375h26m3.956850769s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:37:04.304692457 (375h26m4.18810608s) [seg_pos={0 0}, upload=300223]: Begin upload segment...
Jul 9 00:37:04.304761975 (375h26m4.188175587s) [seg_pos={0 0}, upload=300223]: Done begin upload segment.
Jul 9 00:37:04.304833908 (375h26m4.18824754s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305365539 (375h26m4.18877917s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305738725 (375h26m4.189152359s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305907885 (375h26m4.189321515s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306301693 (375h26m4.189715324s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306679055 (375h26m4.190092689s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307048455 (375h26m4.190462086s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307412956 (375h26m4.19082659s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.308062976 (375h26m4.191476609s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.309126547 (375h26m4.192540184s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.310676166 (375h26m4.194089804s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311146138 (375h26m4.19455977s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311543981 (375h26m4.194957595s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311932868 (375h26m4.195346498s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312324028 (375h26m4.195737661s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312707561 (375h26m4.196121192s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313124992 (375h26m4.19653862s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313485189 (375h26m4.196898821s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.893020764 (375h26m4.776434382s) [upload=300223]: Next returned nil segment
Jul 9 00:37:04.893957299 (375h26m4.777370932s) [upload=300223]: Waiting for error group managing segments...
Jul 9 00:37:05.037354849 (375h26m4.920768466s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.038444983 (375h26m4.921858597s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.039356590 (375h26m4.922770207s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.047559125 (375h26m4.930972742s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.053248590 (375h26m4.93666221s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.054452827 (375h26m4.937866447s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.889326616 (375h26m5.772740232s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:29.628225928 (375h26m29.511639541s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:30.256578837 (375h26m30.139992453s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:32.331498448 (375h26m32.214912065s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:39.883025841 (375h26m39.766439475s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:51.499841545 (375h26m51.383255165s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:55.982236621 (375h26m55.86565024s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607812826 (375h27m3.49122645s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607853389 (375h27m3.491267022s) [seg_pos={0 0}, upload=300223]: Segment reached optimal threshold of 14 pieces.
Jul 9 00:38:03.608006922 (375h27m3.491420552s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608045600 (375h27m3.491459233s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608087309 (375h27m3.491500937s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RY6XGL5MJYc2idL6Np7DS3U9pCQGDhuEEqguY8AgjasjE4hiZ): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608191697 (375h27m3.491605324s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608213412 (375h27m3.49162703s) [seg_pos={0 0}, upload=300223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 00:38:03.608310462 (375h27m3.49172408s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 00:38:03.608316810 (375h27m3.491730438s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:03.761052568 (375h27m3.6444662s) [upload=300223]: Done uploading segments...
Jul 9 00:38:03.761076757 (375h27m3.644490378s) [upload=300223]: Upload finished. err: <nil>
Duration: 36.795473967s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
20 out of 60 is failed (threshold=3) [..XX.X..X..X..XX.X...........XXXXX...X.........X..X...XXX..X] duration:
1m0.001051141s
FAIL (2025-07-09 01:20:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001081276s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
16.043664082s
FAIL (2025-07-09 01:20:12 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
3980 (375h26m3.9562776s) [upload=300223]: Starting upload
Jul 9 00:37:04.072968784 (375h26m3.956382412s) [upload=300223]: Done starting upload
Jul 9 00:37:04.073087735 (375h26m3.956501365s) [upload=300223]: Uploading segments...
Jul 9 00:37:04.073377778 (375h26m3.956791396s) [upload=300223]: Got next segment. Inline: false
Jul 9 00:37:04.073430757 (375h26m3.956844388s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:37:04.073437151 (375h26m3.956850769s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:37:04.304692457 (375h26m4.18810608s) [seg_pos={0 0}, upload=300223]: Begin upload segment...
Jul 9 00:37:04.304761975 (375h26m4.188175587s) [seg_pos={0 0}, upload=300223]: Done begin upload segment.
Jul 9 00:37:04.304833908 (375h26m4.18824754s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305365539 (375h26m4.18877917s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305738725 (375h26m4.189152359s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305907885 (375h26m4.189321515s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306301693 (375h26m4.189715324s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306679055 (375h26m4.190092689s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307048455 (375h26m4.190462086s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307412956 (375h26m4.19082659s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.308062976 (375h26m4.191476609s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.309126547 (375h26m4.192540184s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.310676166 (375h26m4.194089804s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311146138 (375h26m4.19455977s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311543981 (375h26m4.194957595s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311932868 (375h26m4.195346498s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312324028 (375h26m4.195737661s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312707561 (375h26m4.196121192s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313124992 (375h26m4.19653862s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313485189 (375h26m4.196898821s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.893020764 (375h26m4.776434382s) [upload=300223]: Next returned nil segment
Jul 9 00:37:04.893957299 (375h26m4.777370932s) [upload=300223]: Waiting for error group managing segments...
Jul 9 00:37:05.037354849 (375h26m4.920768466s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.038444983 (375h26m4.921858597s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.039356590 (375h26m4.922770207s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.047559125 (375h26m4.930972742s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.053248590 (375h26m4.93666221s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.054452827 (375h26m4.937866447s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.889326616 (375h26m5.772740232s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:29.628225928 (375h26m29.511639541s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:30.256578837 (375h26m30.139992453s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:32.331498448 (375h26m32.214912065s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:39.883025841 (375h26m39.766439475s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:51.499841545 (375h26m51.383255165s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:55.982236621 (375h26m55.86565024s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607812826 (375h27m3.49122645s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607853389 (375h27m3.491267022s) [seg_pos={0 0}, upload=300223]: Segment reached optimal threshold of 14 pieces.
Jul 9 00:38:03.608006922 (375h27m3.491420552s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608045600 (375h27m3.491459233s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608087309 (375h27m3.491500937s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RY6XGL5MJYc2idL6Np7DS3U9pCQGDhuEEqguY8AgjasjE4hiZ): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608191697 (375h27m3.491605324s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608213412 (375h27m3.49162703s) [seg_pos={0 0}, upload=300223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 00:38:03.608310462 (375h27m3.49172408s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 00:38:03.608316810 (375h27m3.491730438s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:03.761052568 (375h27m3.6444662s) [upload=300223]: Done uploading segments...
Jul 9 00:38:03.761076757 (375h27m3.644490378s) [upload=300223]: Upload finished. err: <nil>
Duration: 16.043691598s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X...............................................X.........] duration:
2.282743153s
FAIL (2025-07-09 01:20:06 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
cessful: 13 optimal: 13 errs: <nil>
Jul 9 01:12:51.706672230 (376h1m51.590085861s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 01:12:51.706677984 (376h1m51.590091617s) [upload=300705]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 01:12:51.883113268 (376h1m51.76652692s) [upload=300705]: Done uploading segments...
Jul 9 01:12:51.883135264 (376h1m51.766548909s) [upload=300705]: Upload finished. err: <nil>
Jul 9 01:13:20.653341358 (376h2m20.536754991s) [upload=300711]: Starting upload
Jul 9 01:13:20.653504915 (376h2m20.536918556s) [upload=300711]: Done starting upload
Jul 9 01:13:20.653690252 (376h2m20.537103868s) [upload=300711]: Uploading segments...
Jul 9 01:13:20.653990712 (376h2m20.537404332s) [upload=300711]: Got next segment. Inline: false
Jul 9 01:13:20.654059122 (376h2m20.537472753s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 01:13:20.654065941 (376h2m20.537479574s) [upload=300711]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 01:13:20.769797764 (376h2m20.653211413s) [seg_pos={0 0}, upload=300711]: Begin upload segment...
Jul 9 01:13:20.769876747 (376h2m20.653290386s) [seg_pos={0 0}, upload=300711]: Done begin upload segment.
Jul 9 01:13:20.769954018 (376h2m20.653367648s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770052187 (376h2m20.653465935s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770659155 (376h2m20.6540728s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.770677751 (376h2m20.654091394s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771126425 (376h2m20.654540056s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771277890 (376h2m20.654691526s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771476972 (376h2m20.654890603s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771804253 (376h2m20.655217892s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771829179 (376h2m20.655242816s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.771909583 (376h2m20.655323221s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.772476215 (376h2m20.655889855s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.773678060 (376h2m20.657091699s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774188246 (376h2m20.657601884s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.774573193 (376h2m20.657986824s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775340796 (376h2m20.658754434s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:20.775786189 (376h2m20.659199809s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Uploading piece...
Jul 9 01:13:21.378914824 (376h2m21.262328447s) [upload=300711]: Next returned nil segment
Jul 9 01:13:21.378938224 (376h2m21.262351854s) [upload=300711]: Waiting for error group managing segments...
Jul 9 01:13:21.517933925 (376h2m21.401347538s) [noise=true , address=45.91.116.14:4191 , piece_id=VEYEZNNPLRQSBSE62QVBT2Q4WPKXG4PGDXH5HJKTPVHQQDB4MKPA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.518663136 (376h2m21.402076749s) [noise=true , address=45.91.116.5:4297 , piece_id=R2Z5M6C32BGS2MCKBKMSNTIDLYG5RHCF5CTRNWPQTOWI4PY4B2MQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.522226581 (376h2m21.40564021s) [noise=true , address=45.91.116.8:4047 , piece_id=Z6ZRHEM6QQV4TW4ZHK6NL4Z47NI6IWG7TUH27QSS2L6PTW32YHFA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.535639134 (376h2m21.419052751s) [noise=true , address=45.91.116.8:4048 , piece_id=DOCCMTZC4IEFCVSBXIPSA4F6EIBQ7ZJKGSRLNW5VFE4VODCIWDYA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.539204781 (376h2m21.422618407s) [noise=true , address=45.91.116.7:4011 , piece_id=UY4HNXEUHJDV2UGJZFWEW6EN2MRMSMY3RHOVBEKGK57GFHRXF2LQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.545311795 (376h2m21.428725407s) [noise=true , address=77.90.31.11:4554 , piece_id=GOGCT6WPTDUBGCQ6R7MP7F4SRHMUMKX4DVBLTUBHVI6AHMF25M6A, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548650688 (376h2m21.432064308s) [noise=true , address=45.91.116.6:4975 , piece_id=5UBUFVKBEZA25VPY7XGITKY3THJPJ3V5JWYENF2QUNF4EC7NK7UQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.548766114 (376h2m21.432179732s) [noise=true , address=45.91.116.12:4226 , piece_id=HGFAMZDUY3ML6KVAR4JYAIRQNO5N6Y2ZYH3MLO5NZ5H3J7LBDWUA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.552618914 (376h2m21.436032531s) [noise=true , address=77.90.31.16:4733 , piece_id=IR6MHNY3MOXTYIAFLQAGXS2TU2OPDYJHKAMGN5QY7WIZP5UGCW4Q, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556234201 (376h2m21.439647819s) [noise=true , address=77.90.31.17:4771 , piece_id=XXZHZKKCF6N7TEJXKYQWKFWYJQCCXPDVJQKJ6IG7GYQKMFM4CLSA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.556714631 (376h2m21.440128256s) [noise=true , address=77.90.31.10:4520 , piece_id=4G4JPBIA3BKXM7CTKDF3KMSDX5KEDDDXCNCMMEPWJW2JRZUHUARA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:21.584297058 (376h2m21.467710677s) [noise=true , address=45.91.116.9:4261 , piece_id=34XG5S3MF77RT6W7VRFI5SKX67OLWSFQ3VXT5ECQSQRWQH7LWPHQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: <nil>
Jul 9 01:13:50.371498503 (376h2m50.254912119s) [noise=true , address=104.36.82.11:4113 , piece_id=ANKTMIW4IJ4A3F7ALG3FF4NKBQTSPOZ77NLZYE3OFXMZMODPIJCQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 01:13:50.371544262 (376h2m50.254957891s) [noise=true , address=104.36.82.7:4909 , piece_id=ONV4Q6GJHTEX4N6GHLAYDVXPSYJFRW7ZRECICSZGRRFQUMJOQ7SQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37278->104.36.82.7:4909: use of closed network connection
Jul 9 01:13:50.371667019 (376h2m50.255080637s) [noise=true , address=104.36.82.15:4257 , piece_id=W7UXNP7P4FHL7QTGQTHSKLHUER4Y7W7X3MA7CWJAEYUGTEGY56NA, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33616->104.36.82.15:4257: use of closed network connection
Jul 9 01:13:50.371684908 (376h2m50.255098535s) [noise=true , address=104.36.82.10:4078 , piece_id=3M6PXVRYRN4GZJVHDMIHLZZ6LM53B6NNLX6NJDOGFX7VSZ246CZQ, seg_pos={0 0}, upload=300711]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57754->104.36.82.10:4078: use of closed network connection
Jul 9 01:13:50.371707098 (376h2m50.255120766s) [seg_pos={0 0}, upload=300711]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Jul 9 01:13:50.484313152 (376h2m50.367726771s) [upload=300711]: Done uploading segments...
Jul 9 01:13:50.484347418 (376h2m50.367761038s) [upload=300711]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.28278488s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
23.231349049s
FAIL (2025-07-09 01:19:56 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
3980 (375h26m3.9562776s) [upload=300223]: Starting upload
Jul 9 00:37:04.072968784 (375h26m3.956382412s) [upload=300223]: Done starting upload
Jul 9 00:37:04.073087735 (375h26m3.956501365s) [upload=300223]: Uploading segments...
Jul 9 00:37:04.073377778 (375h26m3.956791396s) [upload=300223]: Got next segment. Inline: false
Jul 9 00:37:04.073430757 (375h26m3.956844388s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:37:04.073437151 (375h26m3.956850769s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:37:04.304692457 (375h26m4.18810608s) [seg_pos={0 0}, upload=300223]: Begin upload segment...
Jul 9 00:37:04.304761975 (375h26m4.188175587s) [seg_pos={0 0}, upload=300223]: Done begin upload segment.
Jul 9 00:37:04.304833908 (375h26m4.18824754s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305365539 (375h26m4.18877917s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305738725 (375h26m4.189152359s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305907885 (375h26m4.189321515s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306301693 (375h26m4.189715324s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306679055 (375h26m4.190092689s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307048455 (375h26m4.190462086s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307412956 (375h26m4.19082659s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.308062976 (375h26m4.191476609s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.309126547 (375h26m4.192540184s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.310676166 (375h26m4.194089804s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311146138 (375h26m4.19455977s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311543981 (375h26m4.194957595s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311932868 (375h26m4.195346498s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312324028 (375h26m4.195737661s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312707561 (375h26m4.196121192s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313124992 (375h26m4.19653862s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313485189 (375h26m4.196898821s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.893020764 (375h26m4.776434382s) [upload=300223]: Next returned nil segment
Jul 9 00:37:04.893957299 (375h26m4.777370932s) [upload=300223]: Waiting for error group managing segments...
Jul 9 00:37:05.037354849 (375h26m4.920768466s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.038444983 (375h26m4.921858597s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.039356590 (375h26m4.922770207s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.047559125 (375h26m4.930972742s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.053248590 (375h26m4.93666221s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.054452827 (375h26m4.937866447s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.889326616 (375h26m5.772740232s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:29.628225928 (375h26m29.511639541s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:30.256578837 (375h26m30.139992453s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:32.331498448 (375h26m32.214912065s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:39.883025841 (375h26m39.766439475s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:51.499841545 (375h26m51.383255165s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:55.982236621 (375h26m55.86565024s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607812826 (375h27m3.49122645s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607853389 (375h27m3.491267022s) [seg_pos={0 0}, upload=300223]: Segment reached optimal threshold of 14 pieces.
Jul 9 00:38:03.608006922 (375h27m3.491420552s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608045600 (375h27m3.491459233s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608087309 (375h27m3.491500937s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RY6XGL5MJYc2idL6Np7DS3U9pCQGDhuEEqguY8AgjasjE4hiZ): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608191697 (375h27m3.491605324s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608213412 (375h27m3.49162703s) [seg_pos={0 0}, upload=300223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 00:38:03.608310462 (375h27m3.49172408s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 00:38:03.608316810 (375h27m3.491730438s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:03.761052568 (375h27m3.6444662s) [upload=300223]: Done uploading segments...
Jul 9 00:38:03.761076757 (375h27m3.644490378s) [upload=300223]: Upload finished. err: <nil>
Duration: 23.231381779s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [.X......X.............XX.X.X......X.X.XXXXX.......XX........] duration:
12.230770287s
FAIL (2025-07-09 01:19:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 12.230806012s
edge
gateway
us-select-1
|