dmesg --follow
[ 66151680.000 ] posts.x: Full talk on training a search sub-agent with RL, and the speed and cost gains that come with it:  |   [ 66153540.000 ] posts.x: @hypeapps This kind of time tracking is quite important. Until you’ve logged how long projects (and, if you’re as easily distracted as I am…  |   [ 66153720.000 ] posts.x: @Mappletons I’ve been redesigning my personal site this past week and I’ve studied your site extensively as a great example. I really appreciate your…  |   [ 66153780.000 ] posts.x: @Mappletons The redesign isn’t live yet, BTW …  |   [ 66159420.000 ] posts.x: Full talk on why there's no single right chunk size and how multiscale indexing with RRF closes the gap:  |   [ 66159420.000 ] posts.x: A fixed chunk size is a bet on queries you haven't seen yet. @yuvalinthedeep, Sr. Developer Advocate at AI21, tests that bet in "Stop Chunking Like…  |   [ 66166140.000 ] posts.x: @hypeapps @toggl Is this because you’re straddling multiple agent sessions for multiple projects simultaneously?  |   [ 66166560.000 ] posts.x: @hypeapps @toggl You could still get pretty close. Agent sessions are stored on disk. Subtract the long gaps between your inputs to the session…  |   [ 66168660.000 ] posts.x: Full talk on why llms.txt isn't enough and what actually makes a website agent-ready:  |   [ 66168660.000 ] posts.x: Almost half of the websites in one study already publish an llms.txt file for agents to read, but almost none of the agents actually use it…  |   [ 66169620.000 ] posts.x: Full talk on why AI cluster networks need a receiver-driven, message-based protocol instead of TCP:  |   [ 66169620.000 ] posts.x: Most AI clusters still tune their networks for giant weight transfers, but the workloads pushing performance limits now are tiny messages: a KV cache…  |   [ 66179400.000 ] posts.x: Full talk on the harness layers, the files-vs-databases tradeoff, and context rot:  |   [ 66179400.000 ] posts.x: Most of what makes an AI agent reliable has nothing to do with the model itself. In "Total Recall: Agent Memory and Harness Engineering,"…  |  
corey@gallon.me:~/til$

Copying Files in Linux With a Progress Bar

FIGURE 1 ⋅ Copying Files in Linux With a Progress Bar

I’ve long used rsync for high fidelity, detailed copies of large swathes of files in Linux. For example:

rsync -av --progress /source/directory /destination/directory

This gives you really nice, file-by-file progress indications — which is especially useful when moving large files across the network between machines or even locally across physical devices.

Today I was moving data from old hard drives and just wanted overall progress on the total job to be done. rsync still FTW!

rsync -a --info=progress2 /source/directory /destination/directory/

NB — do not use a trailling / in the source directory if you want the copied files to be retained inside of that directory name in the destination!

This provides lovely summary output like this:

ubuntu@ubuntu:/media/ubuntu$ rsync -a --info=progress2 /source/directory /destination/directory
  2,675,525,167  71%   13.49MB/s    0:03:09 (xfr#1281, to-chk=0/1712)  

A Better, More Robust Way to Do This

Running this command gets some extra good stuffs.

rsync -a --info=progress2 --checksum --partial /source/directory /destination/directory/

Explanation of the Additions:

  1. -checksum:
    • Ensures that files are compared using checksums rather than just timestamps and file sizes.
    • Guarantees that files are transferred only if their content differs, even if metadata like modification times are identical.
  2. -partial:
    • Retains partially transferred files if the transfer is interrupted, allowing resumption without starting over.
corey@gallon.me:~$ tail -f /writing Attach to the stream. An email when I have something worth sending. Replies encouraged!
corey@gallon.me:~$ ls -lt /til ↑2024-12-16 Linking to Posts in Python Pelican
▸2024-11-28 Copying Files in Linux With a Progress Bar ⋅ you are here
↓2024-11-28 Why An 8TB Drive Isn’t 8 Usable TB