dmesg --follow
[ 70396140.000 ] posts.x: "Can I have the code that solved the CAPTCHAs?" Since my talk at @aiDotEngineer World's Fair, that's the question I get most. I've got something…  |   [ 70396200.000 ] posts.x: The talk this comes from, The Dark Arts of Web Automation, at AI Engineer World's Fair:  |   [ 70390020.000 ] posts.x: ↻ @htmx_org: some big news! after a long & intensive search process i am very happy to announce @JensenHuang as the next CEO of htmx ve…  |   [ 70241940.000 ] posts.x: ↻ @JayNagy: I’ve worked that heatshield my entire adult life. Peak career moment today To go from little 7 tile patches to a full reusabl…  |   [ 69798180.000 ] posts.x: ↻ @atlantajug: Maintainers are drowning in AI-generated slop. How do you make sure your PR isn't part of the problem? 🤖 @CoreyGallon shar…  |   [ 69798120.000 ] posts.x: ↻ @devnexus: Maintainers are drowning in AI-generated slop. How do you make sure your PR isn't part of the problem? 🤖 At #Devnexus2027, @…  |   [ 69626580.000 ] posts.x: @sentinelwayne Not following … ticketing systems are typically async, nay?  |   [ 69483540.000 ] posts.x: @sentinelwayne The chat dropped a couple of times, due to timeout. It’s just the same as if I was distracted on another screen and didn’t respond in…  |   [ 69435300.000 ] posts.x: Happy V for Vendetta Day!  |   [ 69435120.000 ] posts.x: Happy V for Vendetta Day!  |   [ 69346140.000 ] posts.x: This is a *chef’s kiss* salutation from the app in the morning. Nicely done, Claude team! 👌  |   [ 69303240.000 ] posts.x: @markfenner Nyope! It all went off perfectly, as though I had engaged in the chat myself.  |   [ 69303180.000 ] posts.x: @noelkimx Happy to share more — what are you after?  |   [ 69285720.000 ] posts.x: @noelkimx It doesn't have the password -- I logged in myself, then handed the browser session over. It's dodgy practice to ask for a customer…  |  
corey@gallon.me:~/til$

Using the BytesIO Class in Python

FIGURE 1 ⋅ Using the BytesIO Class in Python

The io.BytesIO class in Python is an in-memory stream for binary data. It provides a file-like interface that lets you read and write bytes just like you would with a file, but all the data is kept in memory rather than on disk. This can be extremely useful when you need a temporary buffer, when you're processing data that's generated on the fly, or when you want to simulate a file without touching the filesystem.


Key Concepts

1. Binary Data

  • Binary Data vs. Text Data: Binary data is any data that is not necessarily human-readable (e.g., images, audio files, compiled programs). It consists of bytes, whereas text data is typically encoded in formats like UTF-8.
  • Why Binary?: When dealing with non-textual content (like images or executables), you need to handle data at the byte level to preserve its exact structure.

2. Files in Binary Mode

  • Opening Files in Binary Mode: When reading or writing binary data to a file, you typically open the file in binary mode ('rb' for reading and 'wb' for writing). This ensures that no encoding/decoding happens automatically, preserving the raw bytes.

```python # Writing binary data to a file: with open('output.bin', 'wb') as f: f.write(b'\x00\x01\x02')

# Reading binary data from a file: with open('output.bin', 'rb') as f: data = f.read() print(data) # Output: b'\x00\x01\x02' ```

3. io.BytesIO in Practice

  • In-memory File-like Object: BytesIO acts like a file that exists in memory. This is particularly handy for testing, manipulating binary data without writing to disk, or when performance matters (reducing I/O overhead).
  • Interface Similarity: It supports many of the same methods as regular file objects (like .read(), .write(), .seek(), etc.).

Applied Examples

Example 1: Manipulating Image Data

Imagine you’re working with an image processing library that expects a file-like object, but your image is coming from a web request as bytes. You can wrap the bytes in a BytesIO to provide that interface:

import io
from PIL import Image  # Pillow library for image processing

# Simulated image bytes (normally you'd get this from a request)
image_bytes = b'...'  # Replace with actual image bytes

# Wrap the bytes in a BytesIO object
image_stream = io.BytesIO(image_bytes)

# Open the image using PIL, which expects a file-like object
image = Image.open(image_stream)
image.show()  # Display the image

Example 2: Temporary Data Buffer

If you need a temporary buffer to collect binary data before writing it to disk, BytesIO is ideal:

import io

# Create an in-memory binary stream
buffer = io.BytesIO()

# Write some binary data to it
buffer.write(b'Hello, ')
buffer.write(b'world!')

# Move to the beginning of the buffer to read the data
buffer.seek(0)
data = buffer.read()
print(data)  # Output: b'Hello, world!'

Example 3: Testing Without File I/O

When writing tests, you might want to avoid creating actual files. BytesIO allows you to simulate file operations entirely in memory:

import io

def process_binary_data(file_obj):
    # Example function that reads binary data and processes it
    data = file_obj.read()
    return data[::-1]  # Return reversed data

# Create a BytesIO object with some binary data
fake_file = io.BytesIO(b'abcdef')

# Pass the BytesIO object to your function
result = process_binary_data(fake_file)
print(result)  # Output: b'fedcba'

When to Use io.BytesIO

  • Unit Testing: Simulate file objects without creating real files.
  • Web Applications: Handle file uploads or downloads in memory.
  • Data Processing Pipelines: Process streams of binary data (e.g., for image manipulation, compression, encryption) without intermediate files.
  • Performance Sensitive Applications: Reduce I/O overhead by using memory-based buffers.

Summary

  • io.BytesIO is an in-memory binary stream that mimics file operations.
  • It's useful when you need a temporary file-like object for binary data.
  • Common use cases include image processing, temporary buffers, and testing.
  • Understanding binary data is crucial when working with non-text files to avoid encoding issues.

This flexibility makes io.BytesIO a powerful tool in Python for efficiently handling binary data without the need for disk I/O.

corey@gallon.me:~$ tail -f /writing Attach to the stream. An email when I have something worth sending. Replies encouraged!
corey@gallon.me:~$ ls -lt /til ↑2025-02-22 AI Engineer Summit 2025: Agents at Work
▸2025-02-14 Using the BytesIO Class in Python ⋅ you are here
↓2024-12-16 Linking to Posts in Python Pelican