Implement Message Chunking for Content Exceeding NIP-44 Size Limit #2

New issue

Open

opened 2025-03-22 19:13:32 +00:00 by dannym · 0 comments

dannym commented

2025-03-22 19:13:32 +00:00

Owner

Description

NIP-44 has a limitation where it can only encode content up to 65535 bytes in length. This creates problems when users need to transmit larger content, as messages exceeding this limit will fail to be properly encrypted and processed.

Proposed Solution

Split messages into 32768-byte chunks
Add metadata to each chunk (total chunks, chunk index, message ID)
Only process/reassemble when all chunks are received
Implement error correction for reliability

Error Handling

Add Forward Error Correction (FEC) to recover from missing chunks
Include checksums for validation
Allow reconstruction even with ~15% chunk loss
Implement timeout and retry mechanisms
Cache chunks temporarily to facilitate retransmission

Implementation Details

Messages under the size limit continue to be processed normally
For oversized content, implement splitting logic before encryption
Add chunk metadata to event properties
Develop logic to buffer partial messages until all chunks arrive
Implement reassembly functionality to reconstruct the original message

Acceptance Criteria

Content larger than 65535 bytes can be successfully transmitted
Chunked messages are correctly reassembled at the destination
Partial message chunks are not stored in the relay
Recover from partial chunk loss via error correction
System handles missing chunks gracefully (timeouts, error messages)
Performance impact on the relay is acceptable

Technical Considerations

Need to ensure that chunk ordering is preserved
Consider how to handle failed or missing chunks
Ensure backward compatibility with clients not supporting chunking
Error correction adds overhead but improves reliability
Balance correction capability vs. increased chunk size
Ensure backward compatibility

## Description NIP-44 has a limitation where it can only encode content up to 65535 bytes in length. This creates problems when users need to transmit larger content, as messages exceeding this limit will fail to be properly encrypted and processed. ## Proposed Solution - Split messages into 32768-byte chunks - Add metadata to each chunk (total chunks, chunk index, message ID) - Only process/reassemble when all chunks are received - Implement error correction for reliability ### Error Handling - Add Forward Error Correction (FEC) to recover from missing chunks - Include checksums for validation - Allow reconstruction even with ~15% chunk loss - Implement timeout and retry mechanisms - Cache chunks temporarily to facilitate retransmission ## Implementation Details - Messages under the size limit continue to be processed normally - For oversized content, implement splitting logic before encryption - Add chunk metadata to event properties - Develop logic to buffer partial messages until all chunks arrive - Implement reassembly functionality to reconstruct the original message ## Acceptance Criteria - [ ] Content larger than 65535 bytes can be successfully transmitted - [ ] Chunked messages are correctly reassembled at the destination - [ ] Partial message chunks are not stored in the relay - [ ] Recover from partial chunk loss via error correction - [ ] System handles missing chunks gracefully (timeouts, error messages) - [ ] Performance impact on the relay is acceptable ## Technical Considerations - Need to ensure that chunk ordering is preserved - Consider how to handle failed or missing chunks - Ensure backward compatibility with clients not supporting chunking - Error correction adds overhead but improves reliability - Balance correction capability vs. increased chunk size - Ensure backward compatibility