Sending file data within JSON

[Seymour Clufley]

X have changed their API for uploading media files. Their endpoint for receiving a file in chunks now requires everything to be inside a JSON block. The process is to call the initialize endpoint (stating the file's total size), then to send each chunk to the append endpoint, then to call the finalize endpoint. Each of these is an HTTP request with a JSON body.

Since it has to be inside JSON, I thought it would be necessary to convert the file to Base64 and use Mid() to send a snippet of the Base64 string with each "append" call. This seems to work but the finalize stage fails, saying that the chunks don't add up to the total file size stated in the "initialize" call. This happens whether I state the raw file size or the Base64 memory size.

I get the feeling that I am making some obvious mistake here, but I can't work out what it is. If anyone could advise, I'd be very grateful.

[kenmo]

How are you splitting up the Base64 string? I hope you're splitting on some multiple of 4... eg. 1024 would be OK but 1023 or 1025 would not.

The way Base64 works, if you split within a group of four characters, you've just corrupted the end of the first chunk and the beginning of the second chunk.

[Seymour Clufley]

Yes, I am doing that.

The issue is definitely in the "total file size". The Base64 segments never add up to that - at least, the way the X server is handling them.

But I can't see a way to include data within JSON except Base64...

[kenmo]

That was my only guess. Surely they have some example API request-response JSONs somewhere to compare against? I would hope so.

[infratec]

If this works,

https://github.com/xdevplatform/large-v ... -upload.py

then the size to send is the real binary file size.

[Seymour Clufley]

Infratec,

Yes, that's good to know. However, that Python script is 9 years old and the X API has been changed (recently).

I still can't work out how to send raw data inside JSON. In Python, it would look like this:

Code: Select all

chunk = file.read(4*1024*1024)

json = {
    'segment_index': segment_id
    'media': chunk
}

The variable "chunk" is simply read from the file then inserted into the JSON as "media". Have you any idea how to do this with PB?

[kenmo]

If this works,

https://github.com/xdevplatform/large-v ... -upload.py

then the size to send is the real binary file size.

Looks like that calls their API v1.1 and now they're on API version 2. The real binary size is probably correct to use, though.


Try "media_data" instead of "media" ? In some v1.1 docs I found this:
https://developer.x.com/en/docs/x-api/v ... dia-upload

media The raw binary file content being uploaded. Cannot be used with media_data.

media_data The base64-encoded file content being uploaded. Cannot be used with media.

[kenmo]

Actually, are you sure you can upload media by JSON requests in their v2 API ?

These links only mention uploading via HTTP POST, with some curl examples, and getting JSON as a response.
https://devcommunity.x.com/t/announcing ... -v2/234175
https://docs.x.com/x-api/media/quicksta ... ad-chunked

You could use PB's HTTPRequest() with method POST but you'll have to set the headers correctly and pre-format the POST data into a string... (maybe as base64? maybe urlencoded? I'm not sure)

Sorry, I don't have much else to suggest!

[Seymour Clufley]

The "media_data" field was in v1.1 but deprecated with v2. This tutorial is for v1.1. This is the v2 doc.

When I try to call "APPEND" using a multi-part form data request instead of using JSON, I get this error response:

{"errors":[{"message":"Request body is not valid JSON."}],"title":"Invalid Request","detail":"One or more parameters to your request was invalid.","type":"https://api.twitter.com/2/problems/invalid-request"}*

So the request body has to be JSON.

[Seymour Clufley]

Apparently you would use Base64 and that's how it is integrated into the JSON. But I've found a different way - multi-part form-data - which doesn't use JSON at all.